When 20 independent AI agents receive the exact same requirements and an AGENTS.md instruction file, they do not build the same software. According to an experiment published by vegegoku on DEV.to titled "The Results Were SCARY: One Brief, 20 AI-Built Payment Apps", letting autonomous agents translate a shared brief into working code without human steering reveals deep inconsistencies in runtime behavior and design choices.
To test consistency, the author built a small payment service accepting JSON and XML messages. An orchestration agent created 20 isolated folders. Each folder received the identical prompt and an independent agent built the implementation. No cross-agent feedback or human code reviews occurred during the build phase.
Dissecting the Test Setup and Execution
The evaluation used a fixed fixture of ten accounts and a test corpus of 250 JSON and 250 XML payment messages, covering successful transfers, validation errors, and boundary cases. The test runner sent these 500 requests sequentially to all 20 implementations. Balances were not reset between messages, meaning earlier outcomes directly affected later requests.
Across the 10,000 total request attempts, the 20 applications produced visibly different runtime results. Successful 2xx responses ranged from a low of 244 out of 500 for implementation 17 up to 260 out of 500 for implementation 08. Ending balances for identical starting accounts also diverged. For instance, an account starting at 500.000 JOD ended between 3.576 and 4.816 JOD across different runs. Transport failures also occurred, with implementations 12 and 13 each logging two failures on sample 073, while implementations 06 and 14 introduced an extra development-seeded account outside the original ten-account fixture.
The Real Bottleneck Is Control
Human developers also interpret requirements differently and produce varied codebases over time. The fundamental change with AI agentic coding is velocity. A team of developers might take days to build 20 variants, interspersed with code reviews, QA cycles, and bug reports. AI agents can compress that entire implementation phase into a matter of hours.
When code arrives at that speed, the review process becomes the ultimate constraint. If an agent writes its own code, runs its own tests, and confirms success, both the code and the tests can share the exact same blind spots. Passing an internal test suite does not prove the system matches the actual intent of the initial brief.
For front-end and full-stack developers shipping production features with AI tools, the takeaway is clear. Faster implementation requires tighter, independent verification. Precise acceptance criteria and automated checks are non-negotiable when agents can spin up entire application architectures in minutes.
How do you currently audit the output of your AI coding tools before merging to main?