DEV Community

stepinto.dev
stepinto.dev

Posted on Edited on Originally published at stepinto.dev

How Far Behind Are Open Weights, Measured

Where AI Research Actually Happens ended on one bar chart: the best open-weight models trail the closed frontier by four months on Epoch's index, ten on private benchmarks, sixteen on compute. A single bar hides the interesting part, which is how that lag has moved. This post is the lag over time, and what it says about a sovereign stack that runs on weights it did not train.

The data is Epoch AI's Capabilities Index (ECI), a composite of more than fifty benchmarks fitted onto one scale, anchored so that Claude 3.5 Sonnet scores 130 and GPT-5 scores 150. The scores are the ones published on 18 November 2026: 257 models with a score and a release date, 123 of them closed, 113 with downloadable weights, and 21 that Epoch files as neither (hosted-only Qwen variants and older models whose access it lists as unknown), which I left out. I also left out one of the 113, the API preview of Mistral Large 4, because its weights could not be downloaded until the release that has a row of its own. The file is under every chart, and every number in this post comes from one script over it.

Update, 18 November 2026: this post was written from Epoch's file of 30 October. Epoch has since refitted the index, which moved the oldest models down by several points, scored Kimi K3.1 Pro and the released Mistral Large 4 weights, and added Claude Fable 5.2 and GPT-6.2 Astra. The charts and every figure below now come from the 18 November file. The 30 October file is still available.

Two staircases

The closed frontier and the open-weight frontier on the Epoch Capabilities Index, 2023 to 2026

The highest ECI score available with closed weights and with open weights, by release date. Each dot is a new record. Epoch AI Capabilities Index, 18 November 2026 fit.

Download the data (CSV)

The closed frontier has gone from GPT-4 at 122 to GPT-6.2 Astra at 174 in a little over three and a half years. The open-weight frontier has gone from LLaMA-65B at 101 to Kimi K3.1 Pro at 161. Both lines climb at about the same rate, which is the first thing worth knowing: the open models are not falling further behind in capability terms, they are climbing a staircase that someone else built a few months earlier.

The second thing is where the two lines nearly meet, which they have done twice. In July 2024 Llama 3.1 405B landed at 128, two points below Claude 3.5 Sonnet, which had been the closed frontier for a month. In January 2025 DeepSeek-R1 landed at 139, under three points below o1, which had also been the frontier for a month. Those are the closest the open line has come to the closed one in the whole series.

The lag, day by day

How many months behind the closed frontier the best open-weight model was, day by day

For each day, the best open-weight model available, and how many months earlier the closed frontier first reached its score. Dots are the open-weight records, coloured by the country of the lab that released them. Derived from the Epoch data above.

Download the data (CSV)

The measure is simple. Take the best open-weight model available on a given day. Find the first day the closed frontier reached that score. The lag is the time between the two. It climbs by one day per day between open releases and drops at each new record, which gives the sawtooth.

The sawtooth has three regions. Through 2023 and the first half of 2024 the closed frontier was GPT-4 and its successors, which gained four points in thirteen months while the open models crawled up from 101. By April 2024 Llama 3-70B had got to 121, just short of the original GPT-4, and the lag was thirteen months. The day before Qwen2-72B it touched fifteen. Qwen2-72B passed the original GPT-4 in June 2024 and the lag fell to four months in one release, and in July Llama 3.1 405B matched GPT-4o and took it to two.

The second region is the eighteen months from July 2024 to the end of 2025, when the lag never got above nine months and averaged four to six. DeepSeek-R1 in January 2025 brought it down to one month, the lowest point in the data.

The third region is 2026, where the lag has settled into a band between four and nine months and the sawtooth teeth have the same shape every time: a Chinese lab releases, the lag drops to five or six, two or three months pass, it climbs to seven or eight, another Chinese lab releases. The mean for 2026 to date is 6.6 months. On 18 November it was 6.9. Claude Fable 5.2 and GPT-6.2 Astra both went out this month and nothing open has answered either. That does not move the lag, which is measured from the open model's score, but it has taken the gap in points to 12.8, the widest since July 2023.

The half-year means, for the record:

Half year Mean lag, months Mean gap, ECI points
2023 H2 6.6 12.6
2024 H1 11.2 9.1
2024 H2 4.8 6.2
2025 H1 3.8 5.6
2025 H2 6.0 7.0
2026 H1 6.8 10.6
2026 H2 to 18 Nov 6.2 8.9

Epoch's own figure is lower than mine, and the difference is the method. Their May 2026 analysis counts an open model as having caught up with a closed one when it beats it in at least five per cent of bootstrap resamples, which is a generous reading of two overlapping error bars. On that reading the lag since January 2026 is four months. On a strict reading, where the open model's central score has to match, they get six, and my day-by-day version of the same strict reading gets between six and seven. Their October 2025 analysis of the two years before that gave 3.5 months with a 90% interval of 1.1 to 5.3, which matches the middle region of the chart. Both pieces say the gap is slightly wider than it was. The chart agrees, and adds that the widening happened in capability points before it happened in months: the ECI gap was six or seven points through 2025 and has been nine to eleven through 2026. The closed labs have found a faster gear, and the open labs are matching their pace from further back.

Who holds the record

Which country's lab held the open-weight record, February 2023 to November 2026

Which country's lab released the best open-weight model available, day by day. Same data.

Download the data (CSV)

Of the 1,363 days from the first LLaMA to 18 November 2026, an American lab held the open-weight record for 352, a European lab for 273 and a Chinese lab for 738. Europe's share is Mistral for 129 days over the winter of 2023 to 2024, and 144 days in 2023 when Stability AI's Stable Beluga 2 was the best open model in the world, which is the only time a British lab has appeared on this strip.

Since 26 December 2024, the day DeepSeek-V3 went past Phi-4, the record has been Chinese without a break: 692 days. DeepSeek, Alibaba, Moonshot and Z.ai have passed it between them fourteen times. Meta's best 2025 release, Llama 4 Maverick, scored 132 in April against DeepSeek-R1's 139 from January. OpenAI's gpt-oss-120b in August 2025 scored 140 against a Chinese record of 143. The American open frontier is now Meta's Muse Spark Open at 155, released in October 2026 and eleven months behind the closed line.

Under what licence the open-weight record holder was released

The same strip, coloured by the licence the record holder was released under, using Epoch's three categories.

Download the data (CSV)

The licence strip matters more than the country strip if you are the one deploying. The record holder was unrestricted (Apache 2.0, MIT or equivalent) for 699 of those days, under a use-restricted licence of the Llama kind for 283, and non-commercial for 381. The non-commercial days are the first LLaMA, Stable Beluga 2, and then ninety-three days from July to October 2026 when the best open-weight model in the world was Kimi K3, released under Moonshot's own licence: free until your group revenue passes twenty million dollars a year, then a separate agreement. Epoch files that as non-commercial. For most of the firms I would expect to be reading this, it is.

So the model you would have picked for a production stack in September 2026, on capability alone, was one you could not legally run in production. DeepSeek V4.1 Flash took the record back under MIT on 17 October and kept it for six days. Kimi K3.1 Pro has held it since, under terms Epoch files as restricted use. The strip was red for a quarter, and nothing stops it going red again.

Three open frontiers

The open-weight frontier by region, against the closed frontier

The best open-weight score from a Chinese, American and European lab, each as its own staircase, against the closed frontier. Mistral Large 4 is counted from its weights release on 29 October 2026, not from the API preview Epoch scored three weeks earlier.

Download the data (CSV)

A sovereign stack argument usually stops at "open weights", as if open weights were one thing. The chart splits them by where the lab is.

The Chinese line is the open frontier. It has been the top open line since the end of 2024 and sits just under seven months behind the closed frontier today.

The American line is Meta, Microsoft, OpenAI's one open release, Google, Nvidia, Thinking Machines, Reflection and Meta again. It fell behind the Chinese line in December 2024 and has stayed behind it, by about a year for most of 2026: Gemma 4 in April was nearly twelve months behind the closed frontier, Nemotron 3 Ultra in June was nearly fourteen, and Muse Spark Open was ten on release in October and is eleven now.

The European line is Mistral and only Mistral. It was the top open line for part of 2023 and 2024, when Mixtral 8x7B and 8x22B were the best models anyone would let you download. Since then every Mistral release has been well below both other open lines. Mistral Medium 3.5, the best European open-weight model until the end of October, is at 142 and was nineteen months behind the closed frontier by then.

Mistral Large 4 changes this. Epoch scored the API preview on 6 October. The weights went up on Hugging Face on 29 October under Mistral's new Ecosystem Open-Weight licence, a modified MIT whose one restriction is that selling the model as a hosted service needs a separate commercial licence. Epoch has since scored the weights at 155, two points above the preview. Europe's open frontier jumped from nineteen months behind to under eleven. That puts it level with the American line, which reached the same score a fortnight earlier, and four months behind the Chinese one. The closed model it matches is GPT-5.2 Pro, the frontier of December 2025.

If "sovereign" means weights from a lab under European law, the stack post already said you pay a capability price for it. The chart puts a number on it: eleven months behind the frontier for Mistral Large 4, nineteen for anything before it, against seven for Chinese weights you accept the provenance of.

DeepSeek, and the compute that was not spent

Epoch Capabilities Index against estimated training compute, open and closed models

ECI against estimated training compute for the models Epoch has both numbers for. Compute estimates from Epoch's open-versus-closed dataset and the Notable AI Models table; most recent closed models have no public estimate and are missing.

Download the data (CSV)

Compute Isn't the Constraint used DeepSeek-V3 as the exception: near-frontier capability for 2.8 million H800 hours, which the technical report prices at about five and a half million dollars. The scatter says it stopped being an exception the moment it happened.

Every open-weight model above 140 on this chart sits between two and twenty times 1024 FLOP. Every closed model above 140 with a public estimate sits at 1025 or above. GPT-5 at 150 used 6.6 times 1025. Kimi K2.5 reached 149 five months later on 5.8 times 1024, an eleventh of it. GPT-6 Astra at 169 is estimated at 9.4 times 1026. DeepSeek V4.1 Flash at 157 is on 4.3 times 1024, which is about one two-hundredth.

The November 2024 Epoch report measured a fifteen to sixteen month lag in training compute, with Llama 3.1 405B as the open model that finally spent what GPT-4 had spent. Nobody has released weights trained on more compute than that model since. Epoch's largest open estimate after it is Kimi K3 at 2 times 1025, half of Llama's budget, scoring 159 to Llama's 128. The compute lag is now a measure of something the open labs have decided not to do, and the capability lag of seven months is what their alternative costs them.

Which is the point for the research argument from June. The thing that put DeepSeek, Moonshot and Z.ai seven months behind the frontier rather than three years behind it is not a cluster. It is the set of techniques that make a 4 times 1024 FLOP training run score 157, most of which they published, and all of which were developed by people who could not buy their way out of the constraint.

The lag by other measures

The index is one lens. Here are the others, with my reading of each.

Measure Lag Source and date
Epoch Capabilities Index, allowing a draw within the error bars 4 months Epoch AI, May 2026, from 1 January 2026
Epoch Capabilities Index, strict 6 to 8 months This post, 2026 to date; Epoch's strict figure is 6
Cyber ECI 6 to 7 months Epoch AI, October 2026
Public benchmarks 4 to 6 months Håvard Tveit Ihle, May 2026, 9 benchmarks
Private benchmarks 8 to 10 months Same, 8 benchmarks
Training compute 15 to 16 months Epoch AI, November 2024; no open model has passed Llama 3.1 405B since

The cyber row is new since the six charts post. Epoch's cyber-only index puts Z.ai's GLM-5.3 above GPT-5.4 and below Claude Mythos Preview, so the open cyber frontier in October 2026 was where the closed one stood six or seven months earlier. If you are picking open weights for security work, that is the number.

The public and private benchmark rows come from Håvard Tveit Ihle's analysis of Epoch's benchmark data, not from Epoch itself, which the six charts post had wrong and now says. His method is different from the index: for each of seventeen benchmarks he sets score thresholds and times how long after the first closed model each threshold was crossed by an open one. The private set lags the public set by about four months, which he reads as the open models tuning harder on the benchmarks they can see. He also notes that several of his "private" benchmarks are semi-private at best, that exposure cuts both ways, and that his guess is the real gap is larger than ten months, not smaller.

What it means for the stack

The June post said the achievable sovereign win was operational: freeze open weights, host them under your own law, accept that the training was not yours. The data adds three qualifications.

The lag is seven months, and it is stable. A stack built on the best open weights today runs what the closed frontier was running in April. For most production work that is fine, and the staircase shows no sign of the open line stalling. It is a lag, not a ceiling.

The best open weights are Chinese, and have been for 692 days. The post on securing the open-source supply chain covers what that means in practice. The capability data says the alternative, American or European open weights, costs you another four months. For European weights it was another twelve until the end of October, when Mistral Large 4 brought them level with the American line.

The licence is the variable nobody charts. Kimi K3 held the open record for a quarter under terms that excluded most of the firms who would want it. The strip above is green most of the time, and the exceptions are recent. Pin the licence of the weights you deploy as carefully as you pin the weights.

The data

Every chart has a CSV link, and they all come out of one script over Epoch's published scores. The Epoch file itself is here as taken on 18 November 2026, so the derivation can be checked. ECI scores shift each time Epoch refits the index, so the figures here will not match a later download to the decimal.

Sources

Top comments (0)