What's the largest software project AI can complete on its own?

That being said, I am still guiding it on a regular basis and only recently let it run open ended trying fixing small to medium size bugs. The next stage will be try let it run in a loop trying to fix the 100 issues it has identified as well as getting most of the bash test suite to pass.

This solution should look for more problems.

- Project portfolio management software. Good results so far. This is unguided experimentation: Fable researches feature ideas and plans them, then Opus 5 implements them, without me in the loop. The features include things such as an auth server, full-text search, audit logging, resource leveling, HA/DR, IaC, and more.

- FHIR & SNOMED medical software experimentation in demo-only environments. This has mixed results so far. This is guided experimentation: the FHIR specifications and SNOMED specifications are thorough and I'm in the loop. Fable plans OK, but Opus 5 is thrashing a lot, going down many rabbit holes.

It includes a custom WAL, structure aware semantic chunking, ingestion and indexing pipelines, hybrid (rrf), text and vector retrieval, ACL aware filtering at query level, query classifier, citation tracking, and a CrateDB backend that performs fused retrieval over a single table. total LOC: 180 K. Status: evaluation by State agencies for investigative search on legacy documents. tool used - Codex, Luna, and earlier codex 5.3, and gpt 4 mini.

I don't think i wrote more than 10 lines of code in the largest project i'm working on. Lines of code: Java: 900_635, typescript: 725_418, C++: 180_445, Dart: 96_181.

It's been obvious from the start that no model, as good as it is, can do large(-ish) amounts of work by its own without supervision, control, criticism, etc ... If left unsupervised, models usually do half the work, leaving stubs and todos everywhere.

Quality comes from applying software engineering principles as much as possible, just like you would do with teams of junior devs: planning sessions and implementation sessions with adversarial critiques, specifying as much as possible upfront, planning unit/smoke/integration tests, etc ...

Many systems rely on swarm of agents to build software but i've found it very difficult to get good results without lots of overhead/token waste because of inter agent communications mostly.

So instead i built what is mostly a workflow engine to structure / organize processes into workflows with different agents assigned different roles. I've setup a basic landing page here https://kodfactory.com if anyone wants to follow along.

It’s an interesting eval but this is a distinct limitation — most applications that consumers use directly are not purely text-based.

All models in the leaderboard probably have had access to the original source code in their training data.

The interesting question in my view is what is the largest system an AI can maintain over a long time horizon -- while keeping a coherent architecture that adapts and responds well to change. All the while remaining performant, scalable, and understandable to humans. I am unaware of what is the currently longest-lived vibe coded project but it cannot be more than a few years old.

So to my mind, we don't have very much real data yet.

The space is too big and the code too extensive for a 1M context window and the capabilities of the current SOTA models.

I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. I wonder what kind of structure AI-only code would have.