Day 2 of MLH Global Hack Week: Builders Week. Yesterday my tool Build Buddy fixed 6 of 6 simple C# errors with a local model. Today I wanted to compare models on something harder, and to do it properly.
The setup
One C# file with three planted errors:
-
CS0029:
GetStringAsyncreturnsTask<string>, assigned tostring -
CS1061:
body.Lenghtinstead ofLength -
CS0161:
Half()doesn't return on every path
Before asking any model, I committed a protocol: the exact prompt, the input file, the scoring rules, and the compiler's real output for the broken file. Then each answer was built in a clean copy of the project.
The baseline caught my mistake first
My first version used a missing semicolon instead of the typo. The baseline showed the compiler stopped at a syntax error (CS1003) and never reported the other two errors. A model "fixing three errors" would really have fixed one. I swapped it for the typo before running anything.
Results
| Grok 4.7 (Cursor) | Gemma 3 4B (local, Ollama) | |
|---|---|---|
| Builds | ✅ | ✅ |
| Lines touched | 3 | 6 |
| Error codes named correctly | 3/3 | 0/3 |
| Time | 33 s | 17 s |
Only Grok 4.7 was unlocked on my Cursor plan, so I added my local Gemma 3 4B as a comparison point. It's not a Cursor model.
Both compiled. Only one was right.
Grok 4.7 used await, fixed the typo, added return 0;, and changed nothing else. It named all three error codes correctly.
Gemma 3 4B was twice as fast, and its code compiled, but:
- it used
.Result, which blocks the thread - it rewrote
Half(), soHalf(5)now returns 3 instead of 2 - it added an unneeded
using System; - its explanation invented a "System.Linq was not used" error and never mentioned the
Task/awaitproblem
If I had only checked "does it build?", I'd have scored these the same.
What I'm taking away
- Compiling is necessary, not sufficient. Behavior and explanation need checking too.
- Record a baseline before testing fixes. My own test case was broken.
- Start every check from a clean folder. One of my manual checks failed with nine errors because I copied into an old folder, and none of the errors were the model's fault.
Protocol, raw answers, and results: benchmarks/cursor-models
Next
Loading Build Buddy's results into a Tiger Data hypertable, so I can query fix rates over time.
How it was built: protocol and analysis worked through with Claude; Build Buddy features built in Cursor; every result comes from a real build on my laptop.
Update (Nov 12): a reader suggested checking behaviour, not just builds. Build Buddy now runs each fixed program and compares its output. Live: 6/6 compiled, 5/6 behaved. One extra sentence in the prompt took it to 6/6. Details in issue #1.
Top comments (6)
the Half(5) returning 3 instead of 2 is the scary one because it builds and looks fine. would a couple of behavior tests per planted error catch that automatically
Yes, that's exactly the one that worried me too. It passes every check the compiler can do.
A couple of behaviour tests per planted error would catch it:
Half(4) == 2andHalf(5) == 2pass on the original and fail on Gemma's version. I'd only test behaviour the original code actually defined (n > 0), since it never said what should happen for 0 or negatives. That keeps the test fair to Grok'sreturn 0;too.The catch with this sample is that
Halfis a local function inside a top-level program, and the program makes a network call, so it's awkward to test from outside. My plan is to make each sample deterministic (no network) with anexpected.txt, have Build Buddy run the patched program, and only count a fix if it builds and the output matches.Thanks, this is going straight into the next version.
running the patched program and comparing against expected.txt is cleaner than writing tests per sample. curious how many of the compiled fixes survive that once its in
It's in! Live rerun: 6/6 compiled, 5/6 behaved. The one that still fails is CS0161: the fix compiles but prints 3 lines instead of 1. At temperature 0 Gemma gives that same answer every run, so it's a consistent weak spot, not bad luck.
Thanks for the nudge.
5 of 6 behaving is a much more honest number than 6 of 6 compiled. if CS0161 gives the same wrong answer every run at temp 0 you can keep it as a known failure case and see if a prompt change ever moves it
One sentence (“keep the program’s output exactly the same”) took it from 5/6 to 6/6: CS0161 now prints just one line, and the other five still pass. Caveat: tuned and tested on the same six samples, so it could be overfitting. Thanks for the idea!