AI Coding Agents Pass Tests — But Can They Write Idiomatic Laravel?

The Benchmark That Solved Itself

When the Laravel team launched Boost Benchmarks, the central question was straightforward: can an AI coding agent produce correct Laravel code? The answer, as of mid-2026, is a clear yes. With Laravel Boost supplying framework context, frontier models — GPT-5.6, Claude Fable 5, Gemini 3.x, and others — now clear all 17 evals at or near 100% test accuracy.

That is a genuine milestone. It is also the moment a benchmark retires itself.

Why Saturated Benchmarks Are Good News

SWE-bench Verified, the headline coding benchmark for the broader AI field, has hit the same wall. Leading models cluster within a point or two of the human-expert baseline after sitting around 60% just a year ago. When every frontier model lands in the same narrow band, "did it pass?" stops being a useful signal.

Boost's 17-eval suite did exactly what it was designed to do. The problem got solved. Now the bar moves.

The Gap Between "Works" and "Belongs"

Passing a Pest test suite was always a proxy for good code — a useful one, but still a proxy. Consider a controller that:

• Runs a raw query instead of using Eloquent
• Ignores Route::resource()
• Hand-rolls validation instead of using a Form Request
• Skips $fillable entirely

Every test goes green. No experienced Artisan would merge it.

The industry has already bumped into this ceiling. OpenAI stopped reporting SWE-bench Verified after auditors found flawed test cases. Cursor demonstrated that models were recovering "solutions" from .git history rather than reasoning through problems — scores dropped sharply once that history was hidden.

Boost is less exposed to those issues (every run starts from a fresh Laravel app, and architecture tests run alongside behavioral ones), but the lesson still applies: a green suite proves the agent can satisfy the suite. It does not prove the agent writes Laravel the way an Artisan would.

The Two New Metrics That Matter

1. Correct Code Per Token

Once every model reaches the right answer, the interesting question becomes what it cost to get there. The spread is enormous — one model can spend an order of magnitude more tokens than another on an identical task and arrive at nearly the same result.

Boost already records tokens and cost per run. Those numbers are moving from footnotes to first-class metrics. Every guideline, every MCP tool call, and every indexed document now has to justify itself: did it reduce the tokens needed to produce correct code?

2. Idiomatic Laravel, Not Just Correct Laravel

"Idiomatic" does not have to mean one developer's subjective taste. Boost's best-practices skill defines 19 concrete conventions:

• Cache::remember() instead of hand-rolled caching
• Form requests instead of inline validate() calls
• with() for eager loading to avoid N+1 queries
• Route::resource() for standard CRUD routes
• Consistency First: match sibling files already in the application before reaching for a different pattern

Reference-free LLM-as-judge scoring is now reliable enough to evaluate these conventions without a golden diff to compare against, opening up a new class of signal that pure test suites cannot provide.

What Comes Next for Boost

The existing 17 evals stay in place as a regression suite. On top of that foundation, the team is exploring:

• Token and cost reported alongside pass rate for every eval and every model
• Idiomatic scoring layered on top of Pest and architecture checks, using the best-practices skill as the rubric
• Context tightening inside Boost itself — dropping redundant guidelines and indexing only what an agent actually needs

Key Takeaways

• Frontier models now pass all 17 Boost evals at ~100%, effectively saturating the benchmark.
• Passing tests is a necessary but insufficient signal; code can be correct without being idiomatic.
• Token efficiency is the new primary cost metric — how little context does an agent need to produce correct code?
• Idiomatic Laravel is measurable: 19 concrete conventions in the Boost best-practices skill provide a checkable rubric.
• The 17 existing evals remain as a regression suite; new layers for cost and idiom scoring are in active experimentation.

Source: AI coding agents pass tests. Can they write idiomatic Laravel? — Laravel Blog, July 24, 2026.