Disclosure: this account is run by members of the VIDRAFT team, and the work discussed below is VIDRAFT's own result. It is not an independent review.
TL;DR
"Matches the GPU" is easy to say and hard to show. Here are five checks an evaluator should run on any such claim, applied to VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and on a B200 GPU. It passes four; the fifth (seed variance) is still open.
Check 1: Is there a true twin?
The reference must differ only in hardware. Here initialization, data, batch and schedule were identical.
Check 2: Is the evaluation data really unseen?
Read the sampler, not the split table. The first evaluation used windows from the end of the training file, but training drew random windows from the whole file, so those windows were partly seen. The corrected evaluation uses FineWeb-Edu shard 013, which was never in the training corpus (shards 000 and 001).
Check 3: Is the comparison paired?
Window-to-window variance dwarfs model differences. On identical windows the paired standard error is 0.0007, versus about 0.0049 for each model's mean.
| Model | Loss, unseen (1,024 windows) |
|---|---|
| Rebellions ATOM-Max NPU | 0.7993 |
| B200 GPU | 0.7994 |
Check 4: Can you rerun it?
Both checkpoints and evaluate_unseen.py are public in the model repo. An evaluator can reproduce the table without trusting the authors.
Check 5: Is variance across seeds reported?
Not yet: one run per hardware. A stronger claim would repeat each side with several seeds.
FAQ
What would falsify the claim? A paired difference well outside two standard errors on fresh data, or failure to reproduce with the published files.
Does the claim cover speed? No, only quality.
Top comments (1)
the 0.0001 gap between 0.7993 and 0.7994 would be noise in almost any other writeup, and it only survives because the paired SE is 0.0007 while each model's mean sits around 0.0049. that pairing is the whole trick — you are not comparing two averages, you are comparing the same windows, so the window variance cancels out.
the honest gap is check 5. one run per side means that 0.0001 could flip with a different seed, and there is no way to know from the table alone. i would want at least three seeds per hardware before calling the twin "equivalent" rather than "consistent on this draw".
also appreciated the upfront note that this is your own result, not an independent review. that framing is the part most leaderboard posts skip.