If AI coding is lowering your code quality, you're not managing quality right

I think this is a bit of a simplistic mental approach. I've certainly seen a lot of "The engineer owns the outcome, AI is just a tool, don't release anything you don't vouch for."

However, I just don't think that's realistic. It's asking an author to suddenly become an editor. It's asking somebody who writes code to now read and debug others code.

It can actually be harder to find the the bug in a tricky piece of code than it can be to write your own correct code from scratch. I see AI introduce all sorts of bugs all the time in my personal projects that I would never introduce, and would never think to test for, especially around anything graphical.

I agree that agents can produce decent code. In general, I don’t find agentic code beautiful but neither is most of the code I write. The code for ingesting CSV files into my ETL pipeline doesn’t have to be beautiful, it just has to work.

I think the bigger issue (like many things in software engineering) is a management issue. Once upon a time, I could take a look at the final output of a project and if it looked like a Ferrari on the outside, I could have some confidence that there was a good engine under the hood. OF COURSE THIS WASNT ALWAYS TRUE, but something that looked good, or was performant, or whatever, was a decent proxy for the code underneath being good. And with a smart human, there were ancillary things. Having spent 20 hours coding something, they probably thought through the edge cases that their manager, or product team hadn’t considered.

With AI, everyone’s output looks like a Ferrari, so it is hard to know what the internals are like.

A lot of people will probably look at this and say “well you need better management”, but better management has always been elusive in software engineering. Furthermore, reviewing AI generated code is soul crushing work and I don’t know who wants to do it.

In my guesstimate the number of good engineering managers out there is actually very very small and in practice, the best managers that I’ve seen are the ones who don’t think they are good managers, so they just set a very high hiring bar and hire people who don’t need much management.

Ah, the "skill issue" argument again. Same crap aswhen everyonewas worshiping Musk 5-6 years ago, this time it's dario and altman with a claude/chatgpt mask. Crash can't come soon enough.

I think there's a lot of setup and context required for an AI agent to consistently write good code. Once the agent has these guard rails in place I usually get great quality- far better than what I would write in most cases.

I think where things get dicey is being able to write in any language. I write and review code in many languages and frameworks I'm not fluent in, so it's hard for me to distinguish between working code and great code. I can spot when the fundamental logic is wrong, but when it comes to "best fit" choices I'm clueless.

I am getting really good results from claude. We have a 22-year old legacy system. The system is stable, but had issues as all legacy systems do. Claude has been great for modernizing the codebase, updating dependencies, auditing security, and rapidly adding new features. It has worked well with existing code style and patterns. Sometimes it is a little off-track, but overall it is pretty amazing.

When implementing new features or making large refactoring changes; I use the superpowers:brainstorming skill. That has consistent process which has worked really well. I alway review the code before merging, but most of the time there are few issues to correct.

I don't do 95% coverage, but I have increased it from 65% to about +80% and that is sufficient.

I think it’s much more simple than that. It comes down to caring.

I’ve had a long discussion with a coworker on a long drive.

What we came to realize is the difference in our attitude towards writing code.

I approach it as craft. Even when I’m doing 100% of my coding with an agent these days. I still care about the result to be of high quality and maintainability. I still use my system design knowledge to guide the agent to produce scalable systems.

He treats it like just a job. If it’s good enough he ships. The edge cases and bugs don’t matter. Can be fixed later.

But in my mind that’s a fallacy. We all know things don’t get fixed later unless they are obvious defects and users complain.

Instead we get slow degradation of overall quality. All those small issues compound overtime to create a brittle systems that is difficult to debug and maintain.

My mental model of software engineering is like this. Each commit/PR is a small LEGO block. If you make them well they’ll snap well and create a stable structure that can withstand forces. If every LEGO block you make is just slightly off here and there. Your structure becomes unstable and will always have faults and will always have failures under unpredictable environmental pressures.

I'd say step 0 is know your audience.

I'm happily vibing my own toy projects, but would prefer if the tech in hospitals is not vibe coded.

And I don't think it's plausible that the gap between those two is "well you just need to use it right".

If AI coding isn’t lowering your code quality, you have a low starting point.

A sibling comment talks about needing a lot of setup and context for agents to produce good code. That’s both true and bizarre.

If the compiler that I write produces lousy code, I get bugs that I fix until it doesn’t.

And that is the most annoying thing about this revolution. It’s obviously powerful and transformative and I use in my job all the time.

But many, perhaps even most, purveyors seem intent on blaming their users when they have issues, rather than fixing their own bugs.

General model improvement is going a long way here, but basic things like “ensure you use good style and programming practices” really shouldn’t be a thing users need to put in any .md file.

If AI is not lowering your code quality, you weren't very good to begin with. The point of AI is to increase your productivity tenfold while maintaining acceptable (but not great) code quality.

I'm quite happy with the process I've stumbled into:

1) Plan the hell out of everything. Aggressively have multiple agents weigh-in on that plan, in sequential waves. Don't skimp here.

2) Have subagents review every code commit.

3) Create tests for EVERYTHING. If something breaks you want it discovered immediately. Not just unit tests - use golden masters to ensure your UI doesn't break, etc, etc.

Nothing magical, but it gets me to a very stable dev system. And all I have to do is paste those three rules into my agent, and he does it all for me. It's not difficult.

This entire discussion can be summarized as:

LLMs have speed development up so much, the difference between engineers and programmers is becoming too obvious to ignore.

You are holding it wrong!

If AI coding isn't lowering your code quality, you're not using it enough

Of course, it's your fault, not LLMs not being able to write good code and destroying whole codebases in a matter of weeks.

The issue that I have with this "skill issue" argument is that it is essentially "screw you, got mine". Whether it is teaching programming or going up to making software, it has always been an intractable problem to bring experience, heuristics and intuition to words, something teachable, transferrable. Ok, senior engineers with 30+ years of experience say they are using it right and I'm using it wrong, what do I do with that information? Back to sink or swim, just at a massively faster pace than before.

For the part, I do believe there is a way to gain the "eyes of experience" without spending the years, just not sure exactly how.

A recent HN article (below) concluded that asking agents to do TDD wasn't particularly helpful. I hope there is more research on this because TDD will be slower, use more tokens and results in more code to review.

https://news.ycombinator.com/item?id=49605246

"You're holding it wrong."

Since I am retired, my uses of agentic coding harnesses include:

1. update my old open source projects by searching for and fixing defects, adding tests and documentation

2. working on my own agentic coding harnesses, using the coding harness I am modifying to update itself. I am tightly in the loop

Sure, not highly practical use of AI, but I am retired!

> Unit tests at >95% coverage

Eh. I wouldn't focus on unit test coverage.

I think it's true that good, well tested code will have higher code coverage than crappy code.

But, above a certain point (which will vary from codebase to codebase), unit tests aren't meaningfully increasing confidence that the code is working.

I'd recommend focusing instead on the code being written in a pure 'functional core, imperative shell' to the extent that's possible. For that pure/functional part, 100% code coverage is attainable (& so not worth remarking on). For the impure parts, unit tests are probably using "mocks" just to get the code to compile anyway.

AI writes unmaintainable code - you can see that many projects don't accept it.

I have a suspect that the people who thing AI code is high quality are the same people that never cared about quality in the first place and now are advocating to stop even having code reviews

That makes no sense. Garbage article.

If an LLM makes a good codebase bad, you can't in good faith blame the coders. You blame the LLM.

When I went down the path that the article advocates, I found that code quality improved but design quality suffered. Everything may have been implemented to spec, but that spec was Byzantine and the implementation was bloated.

Which perhaps isn’t a complete surprise in retrospect because it represents something of a return to the waterfall-y, micro-managed enterprisey style of software development that the agile movement was originally responding to.

This is really quality as in "Quality Management System" rather than good code

If you're a ordinary or bad programmer, AI will puke 10x what you do bad

I am starting to think that AI fails most when used in a recursive loop, which is e.g. the case for software projects, research or long-form writing (books, papers): You start with a given state, give the AI a prompt to modify it, get a new state, then repeat. Each step introduces more AI generated data into the state of the system, which then again goes into the context for producing the next state. AIs pick up context probabilistically and they do not distinguish if data they operate on was produced by an AI or a human. I think how successful people are with AI depends on how much human steering they inject into the system at each step and how well represented their workflow was in the training data of the AI.

As a simple experiment, try giving AI a high level goal for your software and let it iterate on it by just repeatedly prompting it to continue, it will happily churn forever on the goal, turning the codebase into a useless spaghetti mess with very high probability, and growing it more and more without ever cutting anything back. That's what happens without human intervention regarding system state and manipulation. The main issues here are most prompts that are extremely underspecified ("fix the issue with the buttons on the main page") so AI will ingest context data it likely generated itself in a previous step and assumptions from its own training data, then act on that to produce a new state. Think of it like a random walk, the AI makes a small step in one random direction to achieve a goal, that brings the system to a new state which is now the basis for the next step, and so on. If there's no (or not enough) corrective action that pulls the system back to a known good reference state it will keep wandering in random directions.

That's the main issue, people have a hard time steering recursive, probabilistic systems, especially when they never look at the output of the system after each step and correct it. And let's be real, if you examine AI generated output in great detail after each iteration you're often better off writing the code yourself, so I would argue that the promised speed up of agentic development can only be realized if you stop inspecting every output of the system. And it seems we still haven't figured out how to specify the steering instructions that keep a system close to a given ideal state that allow unsupervised, recursive work on most codebases. I think some codebases are by themselves better suited for this as they provide a more rigid harness for AI development and exist in the training data (e.g. CRUD apps using RoR), whereas complex software that doesn't use rigid frameworks is at much higher risk of destruction by AI as there's no reference point in the training data that would hold the AI back from randomly walking to a garbage state.

And that's why people have such different views on agentic software development, some work on codebases that are better represented in the training data and so have great success using agentic tools on them, others work on software that isn't represented so well so AI does poorly on it. I don't think it's an issue with quality management, from my own experiments no amount of hand-written rules or system prompts will keep AI from destroying a codebase for which it doesn't have a strong idea how the code is supposed to look from its own training data in the first place. As another experiment, try giving AI strict rules about how to change code or introduce new features, it will always find a way around them or appropriate them in a maliciously funny way that you haven't anticipated. That's also an artefact of the training process, these systems aren't designed to say no or do nothing, they produce outputs to achieve goals and they will bend your rules to the greatest amount possible if it helps with goal fulfilment.

I completely agree. It 100% matches my experience. The C++ code I maintain now is higher quality, more maintainable, higher performance, less buggy, and faster to modify now using Claude Code.

However it doesn't happen automatically. I spent a lot of time experimenting with Claude Code to figure out the right way to use it.

It's a tool. Learn how to use it well.

basic harness knowledge will improve your code quality significantly

Lmao this is just software best practices. It has fuck all to do with AI

This is a variant of: if [tool] isn't giving you good results, then it's your fault.

For some values of [tool], this is right. Question, is it true for this particular value?

Maybe it’s addressed here, but LLMS will not produce better quality new code/systems/products than the persons prompting are capable of. Either by specing out in detail up front, or by a lot of interactive back and forth steering as it's built, or by having it copy some reference system.

I don't mind this, but this is not how this is being sold at all, and many folks use these tools to be lazy.

Wow what an opener comment thread. One thing is for sure, this is a very contentious topic lol. I have had this opinion since way before this ai boom; someone who pushes code to prod is responsible for what happens in prod with that code. This blogpost is very relatable

So, I'm holding the AI wrong is what you're telling me

You can have all the measures in place that are described in that post, and your code can still be bad. High unit test coverage tells you exactly zero about the solution itself.

And technical quality gates do not help if the human side lacks defense against slop code. If you don't have the right managers in place, the 2 years of experience vibecoder who ships a feature in 4 hours will always win against the 20+ year senior who actually looks at the code he is about to ship.

Folks need to look outside their box when evaluating these kinds of issues.

Software quality has been a solved issue in many realms of the digital industry - for decades. There are countless examples of high quality software producing the certainty and safety required to properly ship products.

The way you do it properly: review, review, review. Not just once, not just twice - but on a continual basis.

Take for example, the issue with safety systems engineering, SIL-4. You identify your requirements through analysis, you write your specs, you then write the tests that will prove the specs, and then you write the code. You apply the tests to the code to confirm that the code delivers on the specs.

But, you know what else you do? You do code coverage testing - meaning you don’t ship a single damn line of code that hasn’t been tested. This doesn’t guarantee that the code is correct, or ‘high quality’ - it does however prevent you from shipping untested code.

Then, you pass a review. Code quality reviews usually involve multiple-eyes-on-the-codebase sessions, where a diverse set of engineers read the code, line by line. It is evaluated on the basis of conformance to stringent, well defined coding rules and standards. Anything that doesn’t pass - goes back for analysis, specs, tests, coding, and then again .. the exact same review.

Then, you ship the code. But for safety systems you also have portions of the system that are there to do online tests - to ensure that the code is functioning on the hardware it is running on, as intended. In some cases these online tests run within a boundary of 10 milliseconds, or even less, shutting everything down within that time frame if something is unexpected - cosmic rays happen, bits get flipped, etc.

That’s a loose, generalization of the situation - but it describes the review, review, review process. Review is a constant, it is not a fixed frame - it is done on multiple frames.

To do code quality, one must be willing to check oneself before one wrecks oneself. Always. Constantly. Without fail, without hubris (there is an enormous amount of hubris in the software world), with humility and responsibility.

AI must be taught the same workflow by humans, enforcing it. If you vibe code some junk code and ship it - you failed to review it. Yes, that’s a lot of code to review that you just produce in an hour and a few tens of thousands of tokens. So? Fucking review it, kids.

There will be models that take this seriously. Use them to do the review. Review the review.

The human attention span must be applied to this review with as much rigor and autonomy - and, very important: agency - as possible. Human attention spans must, in a cyclic fashion, come as close to the actual clock cycles driving the software as possible.

Where you have a code quality issue in an AI-driven project, it is because the cycle of human attention to review and the cycles of the software system itself, are out of sync, not in harmony, and indeed in conflict with each other. Managers must learn to identify when that happens, and immediately add more review.

Too many times, arrogance and hubris ship faulty, buggy code - “it works on my machine!” - but there are countless examples in the pre-AI timeline which demonstrate how human arrogance and hubris are managed, cyclically, in a process designed specifically to erase it from the equation.

You are responsible for the code your AI generates for you. No, the cyclomatic complexity is not an excuse to ignore that responsibility. It is a duty - and the developers who will survive the AI onslaught are the ones who understand that responsibility. Same as it ever was.

I hate that AI makes this kind of vacuous article appear, on the surface, to be credible enough that it makes it in front of my eyeballs.

[dead]