SAN FRANCISCO – Open-weight AI models, the kind anyone can download, run and modify, trail the best proprietary systems by roughly three to nine months. Akshay of benchmarking firm Artificial Analysis gave that estimate on an SF Tech Week panel on Oct. 7.
He said the gap shifts with each release cycle. On some features it has already closed. Image and video input were long limited to closed models, and recent open releases now support both natively.
The gap matters because developers are already moving. Open-weight models handled 56% of the tokens routed through Vercel's AI Gateway in August, the first month they made up a majority, according to the company's September AI Gateway Production Index. They accounted for only 14% of spending, because open models cost far less to run than closed ones.
The panel, on the open-source AI stack, was hosted by AI cloud provider Novita AI at the ASW Builder Loft and moderated by Novita's Johnny Chin. Panelists:
- Sam Herring, machine-learning engineering lead at Nous Research
- Mao Cheng of AI infrastructure startup RadixArk
- Walter Korman, an engineering lead at Vercel
- Akshay of Artificial Analysis
See also: AI lab Nous Research ranks models by how they run Hermes
Inference is where the costs are
Inference is the work of running a trained model to produce answers. Panelists said it now drives much of the innovation and cost.
Cheng's team builds the open-source SGLang inference engine and Miles, a framework for post-training models with reinforcement learning, a method that improves a model by rewarding good outputs. He called the relationship between training and inference "co-design."
Faster inference speeds up the practice runs reinforcement learning depends on, which leaves more compute for training and produces better models.
Herring said open weights let researchers adjust a model's behavior while it runs, using techniques such as steering vectors. Closed models don't allow that.
Akshay said labs increasingly compete on cost per task, not raw intelligence. Caching is one of the biggest factors. It lets a provider reuse work on repeated parts of a prompt instead of computing them again.
Artificial Analysis, which tracks caching across providers, has found a 9 to 14 times difference in cost between third-party providers serving the same open model. The gap comes down to how much each provider caches and how deep a discount it offers.
Akshay said the difference grows on long, multi-step agent tasks.
Korman said Vercel's gateway lets developers send each request to a cheaper or pricier model depending on the job, with automatic fallback when a model fails. In his view, most companies already use many models, and the choice comes down to return on investment.
The harness counts
A harness is the software wrapped around a model that lets it act as an agent, calling tools and running multi-step tasks. Akshay said the model sets the ceiling, and the harness decides how much of that capacity gets used.
He added that labs often train models with a particular harness in the loop, which can tilt results toward that harness. Herring said Nous trains with its own Hermes Agent in the loop for the same reason.
Treat agents as untrusted software
Asked about AI agents reaching places they shouldn't, Cheng said he treats an agent in daily use as untrusted software. He gives it the fewest permissions it needs, and never his Social Security number, bank account or credit card.
During training, he said, agents belong in a sandbox, an isolated environment they can't escape.
Korman pointed to sandboxes, budget caps to stop runaway spending, and rules that keep inference in a chosen region. He said developers also need one view of what every app is spending and how each model performs.
What founders get wrong
Each panelist named one blind spot:
- Korman warned against premature optimization of "software factories," automated pipelines for producing code, while the models change month to month.
- Akshay said founders underrate integration: finding the mix of model, harness and inference provider that does their job at the lowest cost.
- Cheng named evaluation and simplicity.
- Herring said inference-time improvements remain largely untapped.
The 2027 view
Akshay predicted agents embedded in workspaces that run on their own, with people checking in every few days. He also expects capable small open models, in the 8 billion to 27 billion parameter range, to run on laptops and phones.
Cheng predicted swarms of agents working for weeks on hard research problems.
Herring said he expects the open-closed gap to shrink, and private models running on phones to handle agent tasks. Korman said users will get more value per token from a widening range of models and prices.
An audience member pressed the panel to push Apple to let iPhones and iPads run terminals and local models. Herring noted that Apple has its own MLX framework for running models on device and said users should pressure the company.
Korman, who said he once worked on Android and Google Glass, backed the push. He noted the trade-off: the most useful local agents also need the most personal data.
HOW AI WAS USED IN THE PRODUCTION OF THIS PIECE: We drafted this story with Claude from the Riverside transcript of the SF Tech Week panel, following the Distro house-style rules and the structure of Distro's report-writeup skill. Background on Vercel's gateway data and Artificial Analysis' caching research came from the linked sources. We filed the story to DeAI News through the Distro Publisher connector.