Scaffolding helps the smallest Gemma 4 agent, not the larger ones
MOBIUS.LLCA pre-registered experiment on local Gemma 4 software-engineering agents found the opposite of its own prediction: a richer scaffold lifts the 4B-class model by about five points and does nothing for 12B and 26B-A4B. We submitted the writeup to the Gemma 4 Developer Agent Paper Track on Kaggle on 31 October 2026.
The question
Does a richer agent scaffold (rules, code-graph tools, an analyzer sub-agent) help a local Gemma 4 agent fix real Python bugs, and does that depend on model size?
What we registered, and what happened
The registered prediction was: the scaffold helps 12B and 26B-A4B, not E4B. We ran 3 model sizes × 2 scaffold levels × 5 seeds on 104 competition tasks, scored by the official verifier after repairing the shipped local grader. Both predicted directions failed.
| Model | Change from the richer scaffold (max − min) | Reading |
|---|---|---|
| Gemma 4 E4B | +5.4 points (95% CI +2.3 to +8.8) | helped |
| Gemma 4 12B | −2.1 points | equivalent to zero within ±5 (TOST) |
| Gemma 4 26B-A4B | −1.0 point | equivalent within ±5, pooled estimate |
Why
E4B quits early, so rules that say "locate, check, then submit" turn early exits into tested patches. The larger models already spend their whole budget; the heavier scaffold pushes more of them onto the 50-tool-call cap before they submit. Under a fixed budget a scaffold is a lever for a model that under-uses the budget and a tax on one that exhausts it. This account is post hoc; the next experiment is to relax the tool-call cap.
For people running agents on consumer GPUs
- Repeat runs before comparing configurations: two identical runs disagree on 7.5–16% of tasks.
- At 12B and above, do not expect gains from prompt rules or extra tools under tight budgets.
- On E4B-class hardware this scaffold was worth about five points, but it does not substitute for capacity: E4B with the full scaffold (15.4%) stays below 12B with the minimal one (25.2%).
- Validate the local grader with gold and empty patches before trusting a local score.
Everything is released
4,056 run records, the pre-registration and freeze manifests, the deviation log D1–D11, the grader repair, and a CPU notebook that recomputes every number (Apache-2.0).
- Writeup: Kaggle Paper Track
- Code and data: github.com/mobius-style/gemma4-scaffold-capacity
- Archive: doi:10.5281/zenodo.23813679
- Reproduce on CPU: Kaggle Notebook
Disclosure: the design, code, analysis and text were drafted with an AI assistant (Claude); the illustrations were rendered with ChatGPT; the author reviewed each step.
＼ Get the latest news ／
