18 AI models entered a fully autonomous research race — open-weight Kimi K3 nearly caught Claude

Prime Intellect just ran an unusual experiment: 18 top language models — from Fable 5 and GPT-5.6 Sol to open-weight Kimi K3 — were set loose on autonomous research runs where humans never stepped in. The ending is loud: an open Chinese model wired to a cheap agent harness came within touching distance of Anthropic's most expensive flagship. Here is what happened, and why it matters for anyone tracking AI progress.

The race: train a model in as few steps as possible

The task comes from Andrej Karpathy's well-known nanoGPT speedrun project. A small 124-million-parameter model has to reach a validation loss of 3.28 — using as few training steps as possible. The starter recipe gets there in 3,290 steps. The best humans have managed is 2,600.

The agent receives a code repository, a short rulebook and one goal. After that it is on its own: tweaking the optimizer, running experiments, telling real improvement from random noise, deciding what to test next. Hardware: eight H200 GPUs, everything locked inside an offline sandbox so the model cannot look up ready answers. In total there were 153 fully autonomous runs; the longest stretched past eight days.

Who finished where

Anthropic's Fable 5 went furthest at 2,726 steps, closing about 82% of the gap between the baseline recipe and the human record. The flagship Claude Opus 5 crossed the line at 2,920.

Then came the surprise. Open-weight Kimi K3 from Moonshot AI, running through the multi-agent Prime Agent Harness, stopped at 2,930 steps. Ten steps behind a flagship that costs noticeably more per token. GPT-5.6 Sol finished at 3,042 — behind the open model.

The funniest part: nobody invented anything new

Not a single run produced a genuinely novel method. Everything that actually worked — clever normalizations, learning-rate schedules, weight averaging — was described in papers years ago. Every model converged on roughly the same bag of ideas.

The difference was execution. Stronger models do not bury a hypothesis after one bad roll: they change the random seed, repeat measurements, revisit old ideas once conditions shift. They separate real progress from noise more carefully. And they build their own tools: Kimi K3 wrote custom functions for comparing loss curves and assembled a mini numerical lab, validating the Newton–Schulz method on toy cases before moving it into real training.

Why this dents the "AI improving AI" script

The classic recursive self-improvement story goes like this: the smartest closed model starts upgrading itself and pulls away irreversibly while everyone else eats dust. This experiment chips away at that picture. When ideas run out fast, the winner is not the smartest player but the one whose loop of "propose — test — discard" costs less and spins faster. Open models driven by a good harness run more attempts per dollar — and that turns out to matter more than another benchmark point.

The catch

To be fair: this is one very narrow task. A single training script, one clear metric, an unambiguous success criterion — real science looks far messier. No new methods emerged either, so this is still about automating a researcher's routine rather than replacing the researcher. And results hinge heavily on the harness itself: the same Kimi K3 without the agent layer runs noticeably worse.

FAQ

What is an agent harness?

A layer around the model: tools, a code-execution environment, memory of previous steps, a task scheduler. The model stays the same, but working becomes easier — like a person getting a proper desk with proper tools.

Can regular users try Kimi K3?

Yes, the weights are open. You can chat with it in NeuralSpace Chat and try agent-style workflows in Code.

So will AI write scientific papers on its own soon?

Not that fast. Autonomous agents do well where there is a crisp metric and quick feedback. Hypotheses, taste and asking the right questions are still human territory. But the infrastructure around models is being upgraded faster than anything else right now — worth keeping an eye on.