Reviewing the app, not just the build
The Java failure came down to packaging: the jar's manifest lacked Main-Class, which tells Java where to begin execution. Although the build was reproducible and the tests covered application logic, none tested whether the packaged app would start. An AI critic acting as a desktop application engineer gave packaging a 4 out of 10. One repair round added the missing entry and expanded the test suite from 38 to 94; the next packaging score was 6.
Favur added the review process to its harness in September and refined it throughout that month. The loop now closes every phase of a run. During October, it was used in 74 public runs, with the critics' scores, explanations and supporting evidence published on the Favur Evals board.
Other reviewers exposed gaps between working code and the requested user experience. In an October 4 Gemini 3.8 Flash run, the application worked, but users had to try keys until they discovered how to reverse an orbit. A first-time-user critic gave time to first success a score of 4. Displaying the controls on screen raised that score to 8.
An October 9 Kimi K2.6 run in Go missed a different part of the specification: it was supposed to show a small circle orbiting a larger, stationary one. Instead, both circles moved, were the same size and lacked a label. The first-time-user critic scored comprehensibility 2 out of 10, reporting "no stationary circle, no size differentiation, and no direction label." After repairs made the circles distinct, the score reached 6. In each of these cases, the team's own tests had not covered the problem the critic found.
Evidence is required, and the lowest score decides
Favur does not accept a critic's score without supporting evidence. Reviewers must supply a file and line reference, an image of the running application or output from a command they executed. Unsupported scores are rejected before storage. If a reviewer cannot obtain the evidence because an app will not launch or its window cannot be captured, the affected part is reported as unscorable rather than assessed from source code alone.
Each specialist works against criteria established before inspecting the code. Critics begin without the build plan or the team's description of its accomplishments. The panel's lowest score determines whether a phase passes, so favorable assessments from other specialists cannot override a problem one reviewer identifies.
Repairs trigger a fresh panel. Those critics have not seen the build, earlier scores or the repair list, and therefore must establish for themselves whether it meets their criteria.
Favur adapted the Gauntlet Loop described by Matt Shumer in his July 27, 2026 post, "How to Run a Gauntlet Loop." That write-up followed a viral Call of Duty-style game built by Claude Opus 5 from one prompt: roughly 55,000 lines of code, with all textures, meshes, animations and sounds generated from scratch. Shumer's method calls for a concrete reference an agent can inspect, such as actual Call of Duty screenshots, rather than an open-ended request to make something impressive.
Six comparisons show gains with substantial overhead
To examine the loop beyond individual examples, Favur compared October runs using it with runs that shared the same specification, language and models but did not use it. Six sets met those matching conditions. Completeness was higher with the loop in four sets, with changes ranging from a 26.1-point gain for Kimi K2.6 in Python to an 8.3-point decline for Gemini 3.7 Flash in Rust.
The resource costs were more consistent than the quality gains. Five sets took longer with the loop, and the median set required about 3.4 times as much wall-clock time. Token use also increased in five sets, with a median of about 1.9 times the usage without the loop.
The time penalty varied considerably. Muse Spark 1.3 in C# took approximately the same time either way, while the Kimi K2.6 set took more than six times as long with review. One comparison included an unusually short, 19-minute run without the loop. Excluding that outlier reduced the median time multiple only slightly, to 3.3.
These are small comparisons, generally with just one or two runs on each side. Almost all runs without the loop also preceded October's harness, so the differences cannot be attributed solely to review. Favur presents the results as an indication of direction, not a reliable estimate of effect size.
The record of repeat reviews offers a separate view of changes within the loop. Across 93 October reviews conducted after fixes, the lowest panel score increased 48 times, stayed unchanged 18 times and decreased 27 times. Favur notes that a decline can reflect a new critic discovering a problem the previous panel did not examine.
Why completeness and the overall score can diverge
Favur's aggregate score combines eight subjects, but does not weight them equally. Speed and effort receive 13 points each, compared with 6 for completeness. The extra work required by the loop can therefore reduce the total even when the delivered project is more complete. Reviewed runs scored lower overall in five of the six matched sets.
For readers prioritizing a complete application, Favur recommends examining the completeness result and the critics' evidence rather than ranking runs by the aggregate score alone.
Critic sessions are only part of the cost
A critic consumes a full agent session to launch the application, execute commands and capture its window. Three reviewers working across three rounds add nine sessions. Favur says the larger time cost, however, comes from responding to findings: the team must repair the application, rebuild it and wait for another panel.
Favur identifies panel size and review timing as the main controls on that overhead. A small panel at a phase boundary can detect a launch failure without reviewing every individual change. It also suggests ending a phase early when the lowest score stops changing.
Broader multi-agent research provides context, though it does not measure Favur's review loop. Anthropic's published guidance says equivalent tasks typically require 3 to 10 times as many tokens in multi-agent implementations as in single-agent approaches. It also says the additional computation often increases total elapsed time, describing thoroughness, rather than speed, as the principal benefit of parallel agents.
Vals AI's Vibe Code Bench examined coordinated agent teams rather than review loops. Its findings, reported in October 2026, put teams' cost per completed application at 1.8 to 5.1 times that of single-agent counterparts. Only one of four tested model-and-effort conditions showed a statistically significant quality gain: GPT-6 Sol at medium reasoning effort improved by 7.3 points. At maximum reasoning effort, neither model showed a meaningful benefit from teams.
Model reviewers also need scrutiny
Evidence requirements do not remove the broader question of how reliably models judge code. The 2026 study "Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering" found that presentation changes could systematically shift judgments even when the underlying code stayed the same.
The study identified sensitivity to position, verbosity, authority and "refined-version" cues. In some settings, the shifts were large enough to alter task-level conclusions and relative model rankings. Its recommended controls included swapping A/B order and testing controlled prompt variations. Those findings concern LLM judges generally, not an evaluation of Favur's critics specifically.
The public record
The Favur Evals dataset generated on October 20, 2026 contained 262 scored runs spanning 66 distinct models from 21 vendors. It covered four Statements of Work: circles, solarSystem, the2048 and arenaEvolution. Publisher Awesoft Solutions describes the benchmark as fully self-funded, without vendor sponsorship, free credits or API grants.
The public board contains the scores, explanations and evidence for the Java launch failure, the Kimi K2.6 circle mismatch and the Gemini 3.8 Flash hidden-control case. Favur itself remains closed-source and invite-only.
