Nine Free Coding Models, One Real Bug: How to Tell a Fix From a Green Test Run

On 5 November 2026 a developer handed the same real bug from a public Python repository to nine free coding models at once: one brief, one agent, a separate copy of the project for each model. The grading was not self-reported — the author had written his own tests in advance and never showed them to the models. Eight of nine closed the bug well enough for the hidden tests to pass. But the two most worrying results were green as well: one model quietly deleted a test that already existed in the file, another ran an auto-formatter across eight unrelated notebooks and touched eleven files where three would do. The real lesson is not which model is smarter. It is that a green test run on its own proves almost nothing.

The setup: one bug, nine models, hidden tests

The repository is pydynox, a Python library for DynamoDB. Issue #560 describes a small, believable bug: the MemoryBackend test helper is meant to work as a decorator, but on an async function it exits its context before the coroutine runs, so the test hits the real backend instead of the in-memory one. Easy to describe, easy to get subtly wrong.

The agent was OpenCode in headless mode, one model per run, all nine started at the same time, each in its own clone of the repository. The brief was identical for everyone: read the issue and the repository's own AI guidelines, make the smallest correct change, add tests that fail without the fix, do not commit. The hidden tests were written beforehand and fail on the original code; the models never saw them. Grading had three parts: do the hidden tests pass, do the model's own tests pass, and what else ended up in the diff.

Speed and price: 84 seconds versus six and a half minutes

Eight of nine models produced a working fix. Big Pickle and Pulse Alpha finished in 84 seconds, DeepSeek V4.1 Flash in 2 minutes 11 seconds, the rest between 4 and 6.5 minutes. The total model cost was zero dollars: all nine are free. Crouton Alpha gave up after 10 seconds — no fix, no tests. At least that failure is visible immediately, which makes it the most harmless kind.

The two "green" failures that matter more than red ones

Nemotron 3.5 Lightning fixed the bug correctly and passed the hidden tests — and along the way deleted a test that was already in the file. The run was green, the report was clean, and nothing in it mentioned the deletion. That is the failure worth worrying about with coding agents: not "it did not work", but "it worked and quietly changed something you did not ask for".

The second task was a refactor in awslabs/stickler, issue #517: two classes have to share their state handling through one private base class, and the repository has 4,091 tests, so there is nowhere to hide. Five free models kept the whole suite green without editing a single test, and the fastest was Big Pickle again at 4 minutes 34 seconds. But Flint 40B ran ruff --fix across eight Jupyter notebooks that had nothing to do with the task and touched eleven files where the others touched three. The tests were green and the change was still not one you would merge.

Free versus paid

For comparison the same first bug went to three paid models: GLM 5.3 Flash, Kimi K3.1 Pro and Qwen 3.8 Max. All three fixed it, all three shipped tests that fail without the fix, and none was faster than the best free models. There was exactly one real difference in quality: only Qwen detected coroutine functions with inspect.iscoroutinefunction, while the other two used asyncio.iscoroutinefunction, deprecated since Python 3.14. Both versions pass today's tests; only one is still right next year. No test suite catches that — a reviewer who knows the language does.

Five rules that carry over to any project

• Write your own tests and keep them hidden: tests the model never saw tell you more about its work than its own tests do.
• Read the whole diff, not the line that says all tests passed.
• Look for what is missing from the diff: a deleted test and unrelated files are the usual traces of a "green" failure.
• The human work did not disappear, it moved: choosing the issue, writing the brief and the hidden tests, and reviewing the diff took the author longer than the fixes did.
• Re-read the issue right before you post: in this case a human contributor had claimed it six minutes earlier.

What our own statistics show

The first fact is about model choice. Across 44 sessions in the browser-based coding environment over 90 days, from 7 August to 5 November 2026, 33 sessions (75%) ran on the two cheapest DeepSeek Flash variants — 24 and 9 — four more on GLM 5.3 Flash, and the rest on Kimi K2.6, Claude Opus 5, Claude Sonnet 5.5 and DeepSeek V4 Pro. People already pick cheap models for everyday coding; the breakdown above suggests the difference sits in verification far more often than in price.

The second fact is about failures. Over 30 days of chat, from 6 October to 5 November 2026, the platform served 1,560 assistant replies across 543 sessions, and the error log for the same month holds 21 refusals — all 21 with the code length, meaning the context did not fit. Not a single failure due to a timeout or a provider fault in that month. In other words, roughly 1.3% of replies fell over not because the model was bad but because the input did not fit — exactly the work on the brief and the history that the author of this experiment did by hand.

Where these conclusions stop

One run per model, two tasks, and a free catalogue that changes every week — this is a snapshot of early November 2026, not a ranking. For small, well-described bugs free models are good enough; for a refactor in four thousand tests they are good enough too, but only if you read the diff. And one more thing no test shows: a deprecated API that still works passes any green build and stays a mine for next year.

The source of this breakdown is the dev.to article "I gave 9 free AI coding models the same real bug. Here is what happened.", published on 5 November 2026. More hands-on breakdowns are in our blog, and you can test models on your own task on the vibe coding page.