Ask HN: How do you review gen-AI created code?

Posted by captainkrtek 7 hours ago

Counter4Comment4OpenOriginal

I've posed this in a couple comments, but want to get a bigger thread going.

There are some opinions that using LLMs to write code is just a new high level language we are dealing in as engineers. However, this leads to a disconnect come code-review time, in that the reviewed code is an artifact of the process that created it. If we are now expressing ourselves via natural language, (prompting, planning, writing, as the new "programming language"), but only putting the generated artifact (the actual code) up for review, how do we review it completely?

I struggle with what feels like a missing piece these days of lacking the context around how the change was produced, the plans, the prompting, to understand how an engineer came to this specific code change as a result. Did they one-shot this? did they still spend hours prompting/iterating/etc.? something in-between?

The summary in the PR often says what the change is, but doesn't contain the full dialog or how we arrived at this specific change (tradeoffs, alternatives, etc.)

How do you review PRs in your organization given this? Any rules/automation/etc. you institute?

Comments

Comment by lazypl82 2 hours ago

The artifact review approach makes sense to me. How the code was produced doesn't change what the reviewer needs to answer: does this do what it's supposed to, and does it do it cleanly? If anything, I'd rather have a short design note in the PR – intent, constraints, alternatives considered – than a full prompt history. The prompt history is noise; the intent is signal.

That said, one thing review can't fully cover is runtime behavior under real traffic. Not saying that's a review problem – it's just a separate layer that still needs attention after the merge.

Comment by christophilus 6 hours ago

> lacking the context around how the change was produced, the plans, the prompting, to understand how an engineer came to this specific code change as a result. Did they one-shot this? did they still spend hours prompting/iterating/etc.? something in-between?

In my opinion, you have to review it the way you always review code. Does the code do what it's supposed to do? Does it do it in a clean, modular way? Does it have a lot of boilerplate that should be reduced via some helper functions, etc.

It doesn't matter how it was produced. A code review is supposed to be: "Here's this feature {description}" and then, you look at the code and see if it does the thing and does it well.

Comment by al_borland 6 hours ago

At the end of the day, the code is the only thing that matters. That is what defines, in concrete terms, what the program is doing. I don't view an LLM as a high level language, as we can run a prompt 10 different times and get 10 different results. This isn't valuable when it comes to programming.

Even without LLMs, there was a thought process that led to the engineer coming to a specific outcome for the code, maybe some conversations with other team members, discussions and thoughts about trade offs, alternatives, etc... all of this existed before.

Was that all included in the PR in the past? If so, the engineer would have to add it in, so they should still do so. If not, why do you all the sudden need it because of an AI involved?

I don't see why things would fundamentally change.

Comment by JamesLangford 6 hours ago

I mostly ignore how the code was made and review the artifact the same way I always have: correctness, tests, readability, and whether it matches the system design. The prompt history is fine, but it’s not something I want to rely on during review because it’s not stable or reproducible.

What does help is requiring a short design note in the PR explaining the intent, constraints, and alternatives considered. That gives the context reviewers actually need without turning the review into reading a chat transcript.