Show HN: Sphinx – A hook that quizzes you on your AI's code before shipping

FOSS, Basically it plugs into your existing coding agent workflow and every time you push to prod it'll throw up a questionnaire that checks if you understand how the code you're pushing works.

Comments

Comment by theParadox42 4 days ago

I certainly feel like certain AI coding practices I have leave me with nothing learned in a frustrating way. Sometimes the most effective way to getting what I want is just describing a goal really clearly, and then bashing the AI until it gets there. That bashing process can take a lot of energy and yet I don’t gain useful skills or a “better way” that I would if I was still coding. I generate more training data for BigAI corp, but that’s a slow return cycle.

Not sure if this is the right solution, but I would like to learn/understand/grow more as I pursue reckless efficiency.

Comment by liendolucas 4 days ago

Reason #1 to keep writing code by hand is that you don't ever lose that connection between you and the code.

There was an excellent post here on HN that unfortunately went unnoticed. I highly recommend reading it as it exactly points out this problem: understanding.

Here is the link: https://news.ycombinator.com/item?id=50128468

Comment by liendolucas 4 days ago

Note: The post has now been flagged. Why? I have no idea but it seems incorrect.

Comment by 0c3ca83 4 days ago

Because it's not glazing LLMs.

Comment by layer8 4 days ago

Flying monkeys, probably. You can email hn@ycombinator.com about unwarranted flagging.

Comment by fn-mote 3 days ago

The post generated a mass of low quality comments (show dead). Plus after about a few paragraphs the site posted degenerates into just calling all AI companies liars with no (in-blog) justification.

I thought the flagging was appropriate after I read it.

Comment by 0c3ca83 4 days ago

Learning is the bottleneck for shipping code. Building the model in your head, feeling out the edge cases and system interactions, and understanding the intricacies. Typing out the code is a small part of the process.

So, if you want to speed things up with LLMs, the bulk of the speed comes from reducing learning.

Comment by rootnod3 4 days ago

Code by hand? The good old fashioned way? Might be "less efficient", but you will understand what you did and build experience.

Comment by bootsmann 4 days ago

Imo if you’re reprompting you’re losing. If the agent oneshots the task its good but if it doesn’t manage it on the first try you will get there faster by doing it yourself.

Comment by LelouBil 3 days ago

I feel the same a bit. I'm learning how to use agent harnesses and how to get them to work on problems I have with projects.

But it very often feels like brute forcing the solution, and I'm not a big fan of it.

But maybe I'm using it wrong.

Comment by StilesCrisis 4 days ago

Isn't this what /goal was meant for? Claude will bash _itself_ into reaching the goal.

Comment by ceejayoz 3 days ago

In my experience, it's quite good at deciding on its own that the goal needn't be properly met.

"Goal achieved!" "OK, does it work?" "No."

Comment by fwlr 4 days ago

It’s a neat idea, i think it’ll really take off if you change “before pushing to prod” to “before requesting another human’s time or attention”. If something like this was gatekeeping the “submit PR” and “update PR” buttons it would prevent a lot of waste.

Comment by lecarore 4 days ago

Nothing prevents a lazy person from copy-pasting the question to the agent so I'm not sure that it would be an effective gatekeeping mechanism.

Comment by fwlr 3 days ago

Using AI to skip the humanity check is a lot more explicitly anti-social than using ai to write the code; if they are going to be axiomatically lazy I would at least have them do it indefensibly.

Comment by dawnerd 3 days ago

Sadly already happens. We have some rules for our devs that use AI and despite requiring a human to put a screenshot into the pr and hand write the pr they still have an ai do it all for them and will argue about how it’s too time consuming to write a pull request.

I’d love to try this project out but I really don’t think it’ll stop anything, and they’d probably just ask their agent to remove the hook anyways like they do with our husky hook.

Comment by elevation 3 days ago

> requiring a human to [...] hand write the pr

> they still have an ai do it all for them

What recourse do you have when employees do not perform the work you're paying them for?

Comment by dawnerd 3 days ago

That’s the problem, they’re still doing the work and delivering tasks but just not following the guidelines. That’s a discussion we haven’t really had internally yet since it’s all been moving so fast.

Comment by fn-mote 3 days ago

After someone in management gets some guts and enforces the rules, others will comply more willingly.

Assuming someone believes the rules are important enough to follow. Otherwise forget it and move on.

Comment by samuelfmedeiros 4 days ago

[flagged]

Comment by samuelfmedeiros 4 days ago

[flagged]

Comment by pjmlp 4 days ago

The gimmicks we come around to justify existing practices being pushed into the industry never stops surprising me.

One would expect the actual answer would be to write the code ourselves.

Comment by chacha-bong 4 days ago

Lol I built this and immediately got frustrated with it and stopped using it. I think I don't like being questioned by someone who I expect to work like magic.

Comment by rDr4g0n 3 days ago

my technique to get a similar benefit is to write a quality PR description.

a good PR description should be compact and helpful. describing something complex in a compact and approachable way requires understanding.

the process of writing the description often leads me to understand hidden decisions the LLM made and occasionally uncovers bugs.

my PR descriptions separately answer the following questions:

Why is this change being made? a sentence or two about the problem, objective, etc, and a link to more detail (tickets, docs, etc) if available

What does this change accomplish? typically describes observable changes, requirements, etc. crucially this section avoids describing how the change is implemented. the act of separating what from how isn't new, but it's crucial in the LLM era

How does this change accomplish its goal? the answer to this question is "read the code", but the job of a PR description is to provide a quick hand drawn map before dumping the reviewer into the jungle. drawing a good "how" map requires understanding the jungle and the new path through it.

the risk with LLMs (or any outsource) doing the work is the loss of understanding. this exercise encourages understanding.

Comment by jamienk 4 days ago

I can kinda understand the motivation here: you push a bunch of slop (or is it deep, genius code?) but you have a kind of nagging suspicion that you should get to know it more familiarly, like back in the good old days a few months back when you sort of understood what was going on. I... don't... think this is going to work...? Remember Confirmation Fatigue? Soon we will have agents that interact with Sphinx for us.

Comment by gandhiandy69 4 days ago

We went from micromanaging code to micromanaging tools and markdowns.

Comment by Crowberry 3 days ago

Where is that quote from, dis you just make it up?

> You can outsource your thinking, but not your understanding. — Sun Tzu

Comment by troymc 3 days ago

I think that quote was coined by AI researcher and robotics engineer Yacine Mahdid writing under the social handle @yacineMTB

Andrej Karpathy circulated it online and placed it at the center of his address at Sequoia Capital's AI Ascent conference in April.

The idea is very old though. There's a story of how King Ptolemy I Soter wanted to learn geometry from Euclid, but being a King, he asked for a shortcut to understanding it. Euclid replied that "there is no royal road to geometry". (A "royal road" was basically a fast road reserved for royals, such as the Persian Royal Road.)

Comment by rguignar 2 days ago

It's from Andrej Karpathy and is a great mental compass to navigate the future with AGI / ASI

Couldn't find any link of this sentence to Sun Tzu

Comment by jeffreysmith 3 days ago

I think it's a real quote, but from Karpathy, not Sunny T.

Comment by maximg68 4 days ago

After noticing a drop in my understanding of the code I ship, I built and started using a similar skill ("Quiz me on the PR or the current branch"). I didn't tie it to a specific commit, though - to me it makes more sense to look at the PR as a whole.

So far I was positively surprised with the questions it could come up with.

Comment by jeffreysmith 3 days ago

We built something quite similar 6 months ago called PEAK (Proof of Effort And Knowledge). Blog post and YT video here: https://substack.com/@jeffksmithjr/p-195763168

Our version has its own TUI and was intended for semi-adversarial scenarios like OSS contribution and enterprise. Thus a lot of anti-cheat tech under the covers (including invalid questions).

My guess is that a lot of folks have built something in this vein. It really made sense earlier in the coding agents cycle. These days, I dunno. Seems like we need to more fundamentally rethink the relationship between coding agents and human understanding of the implementation.

Comment by bluerooibos 3 days ago

What's with Claude preferring cursive and italic font in landing page hero sections?

Looks absolutely awful.

Comment by lowkeyokay 4 days ago

I actually made this as a skill last week. It is helpful. Especially when reviewing a PR with 1500 LOC from a non-dev. It keeps me focused an helps me do a structured review. But it does not catch poor decisions, like calling the wring endpoint a such.

Comment by piro0919 3 days ago

I don't read the code Claude Code writes at all. And yes, it does cause problems. Recently, in a project that was supposed to use shadcn, it quietly stopped using it partway through.

Comment by freshnode 4 days ago

Have you ever heard the idiom "ambulance at the bottom of the cliff"?

Comment by sshah03 4 days ago

I considered doing this with an open source app I have that let's you get a visual summary on a PR + get quizzed on it, but ultimately decided it's better to keep it an optional thing for people that want to understand AI PRs rather than force it as a gate, the volume is slowly becoming too high for even that. I agree with the sentiment behind this though as I felt this was needed as well.

Comment by alyssassan 4 days ago

Maybe the useful distinction is understanding what a PR does versus knowing how to tell if it’s wrong. A quiz can check the first, but the second is often where review matters most.

I’d be interested in a version that helps the reviewer identify one assumption the change depends on and a concrete way to test it. That could keep the exercise small while making it useful, especially if the reviewer spots something the AI-generated summary missed.

Keeping it optional also lets people reach for it when they’re uncertain, instead of turning it into another checkbox as PR volume grows.

Comment by alexwrich 3 days ago

Nice idea. I like that it tests whether the person understands the change instead of adding another AI reviewer. We added the tests before we switch to fully agentic AI development using automo - ie we tested every member of staff and now trust them to be the eyes.

Disclosure: I’m building automo.build fully to fix this.

Comment by man_luke 4 days ago

So who is the master of whom?

Comment by altmanaltman 4 days ago

People who care about the problem will definitely not be using this as a method to "understand" their code. People who don't care about this problem will continue to not care.

So, who is this for?

Comment by nkmnz 4 days ago

It’s ironic that all negative comments on here are dismissive because “slop’s baaaaad”. Yes, slop is bad. Carelessness is bad. Lack of understanding is bad. It always has been. Code has been copied from stackoverflow. Built-in methods or std libraries have been used in a wrong way. Edge cases haven’t been covered. Rate-limiting and other hardening tasks haven’t been taken care of until the system broke. The code wasn’t DRY.

Now, people are building tools that not only hide the solutions to these problems in some obscure coding tool, markdown file, or agent harness, but they build gates that help developers to elevate themselves from the darkness of not knowing better. The number of developers roughly doubles every ~3 years, so at any given time, half of the people building software have less than 3 years of experience. Most of them didn’t even study CS before starting their careers. Tools like this will be the new warning voice that told people like us 5 to 10 years ago to never copy code from stackoverflow that you didn’t understand. It’s great.

Comment by xiaoyu2006 4 days ago

The problem with AI SE is that AI writes un-architecture-ed code. The problem #2 is that a lot of people suck even more....

Comment by Zambyte 4 days ago

AI writes unarchitectes code if you don't have a system for developing and maintaining a consistent architecture over time. In human coding, a lot of architecture decisions tend to only live in human brains, communicated through ephemeral means (text chats, voice calls, etc) and are not concretely documented. Since LLMs don't really have a way to manage long running architectural decisions (memory systems can only get so far, like it won't capture architectural decisions if you never talk about architectural decisions), you have to (have your agent) capture architecture design within the repository.

I personally like using openspec for this, because it makes the model iteratively ask you architectual decisions, and then manages creating and accessing the documents as needed.

Comment by _rwo 3 days ago

> AI writes unarchitectes code if you don't have a system for developing and maintaining a consistent architecture over time (...)

I work at a ~15y old codebase; guess that the famous "architecture" is - a glue of half-baked solutions, unfinished ideas and inconsistencies. Recently I discovered there are 6 different ways used to create audit trails in the app. The "human memory" lives until person decided to switch job or is fired. Then the next person re-invents the wheel and cycle continues.

I suggest taking an honest look at "the architecture" people are so proud of - you might realize there is not much moat there after all. But then again - I mostly write cruds - so that's probably core of the problem, lol

Comment by sohith_uv 2 days ago

[flagged]

Comment by sohith_uv 3 days ago

[flagged]

Comment by nkmnz 4 days ago

Let him who is without sucking among you, suck the first stone.

Comment by jdthedisciple 4 days ago

> The problem with AI SE is that AI writes un-architecture-ed code

Not necessarily. I use SOTA models with established stacks like .NET which allow my agent to easily follow decades of best practiced patterns (DI, EF Core, ASP.NET Core...) so you still and up with proper architecture.

In fact I can easily still manually inspect everything bc I already know how it's supposed to be written.

Comment by marijnz 4 days ago

From my experience, the latest models still struggle to create good architecture in the sense of good system design. You mention a couple of generally helpful patterns for the structure of the code. But they don't help much to create logical units, with intuitive boundaries, good cohesion etc. - at which even the latest models constantly make awkward decisions.

Comment by flaburgan 3 days ago

I agree, but if we go one step further, what is the goal of those logical units? If the answer is, to make the code more maintainable, then the next question is, for whom? For humans, or for llms? Is it still important that the code is maintainable for humans if no humans read it anymore? Those are not rethorical questions, I am asking them for real. What is "good quality" code in the age of AI? If there is a lot of duplication but no performance consequences, is it really a problem if only AI modify it?

Comment by marijnz 3 days ago

It's fair questions. I think we can see that in lower complexity products, it's already not that important anymore. But I can see that for more complex cases (I'm working on a fairly complex video game for example), the maintainability is currently still important. As the latest models would otherwise still create too many inconsistencies that they can't dig themselves out of. Who knows where we are in 6 to 12 months, though!

Comment by diek 4 days ago

> But they don't help much to create logical units, with intuitive boundaries, good cohesion etc.

Completely, 100% agree on these exact points. I have access to every frontier model at work. None of them write what I would call clean code. At work the teams that have gone fully vibe-coded have repos full of broken abstractions that just accumulate over time. Eventually most of the effort of the team and their AI tokens seemingly goes to fixing issues that then break other things. Any attempt at 'refactor this mess, make no mistakes' turns into a _huge_ churn of changes ("shuffling the deck chairs") without ever really fixing things.

Comment by jdthedisciple 4 days ago

Yea but you chose to blame the LLM, not the vibe coders.

As I said: prompting correctly and reviewing are still indispensable skills.

Comment by jdthedisciple 4 days ago

Ofc a big part of it is still knowing how to prompt correctly and manage context.

Perhaps a lot of devs still have a flawed mental model of how these things even work.

Comment by 0c3ca83 4 days ago

The fact that you can say this with a straight face impresses me.

Comment by shigosag 4 days ago

Love this idea, it’s way too easy to get lazy with AI, so having a quick reality check like this is actually a super healthy habit to build

Comment by thomask1995 4 days ago

I think this would be killer for OSS maintainers.

Comment by jeffreysmith 3 days ago

I went around and asked a bunch of them with a similar idea (see the above comment of mine), and you'd be surprised to hear that they all hated the idea. They mostly just don't want to deal with anything where they fear that the person vibed the submission. Just no desire to engage with at all, because they've got so many bigger problems with the avalance of code coming their way.

Comment by 4 days ago

Comment by aavisangle 3 days ago

I ask the question to agent from separate session without history to know more about my code

Comment by SDeniz 3 days ago

I think this could be really useful. Any chance you could add native Windows support?

Comment by sailorganymede 4 days ago

Honestly I love to see people working on this problem ! I actually built a similar project myself (https://github.com/bhurghundii/vibecheck.dev) except it prevents you from merging on the PR level. I’d love to get in touch to talk about it sometimes.

Edit: I see there are a lot of people questioning the value of this. The thing is that there are teams where people are just pumping tons of AI code without understanding their work (literally take the ticket and dump it) and shipping it. culture change is the right answer but v hard but this at least slows things down and is quick to implement

Comment by maximg68 4 days ago

I also experienced pushback when I shared a similar skill with some of my colleagues (and it didn't mention "vibe" in the name). The main objection was "I am doing careful planning upfront, so I have the same level of understanding of the code as before".

Maybe they do. In my experience, the agent was going over the scenarios that I failed to consider during the planning stage, and it was net positive. I haven't tried it with a spec-driven approach.

Comment by sailorganymede 4 days ago

I get what they say but what I see or have seen is totally different. I come from an engineering culture where we got AI token leaderboards and mandates to use AI for every single step of the development process. We started running into bad bugs recently due to people just copy pasting the Jira into the prompt so I feel there is space to put some intentionality in the areas which matter.

Comment by zapkyeskrill 4 days ago

Does it check if the answers are AI generated?

Comment by sailorganymede 4 days ago

Not yet. But you’re like the 5th person to mention it so I think I’ll add that feature. There are some AI prevention stuff like you can’t copy paste which means you gotta use the visualisation.

Comment by cjimesh 4 days ago

Its the right problem statement, but we normally let the code review process take care of this.

Comment by bodash 3 days ago

Why stop there? Combine this with other knowledge testing methods:

pre-code: planning mode

coding: humantodo.dev

after-code: sphinx hook

Comment by ericand 4 days ago

Humans are now test subjects in the ci pipeline

Comment by reddmachine 3 days ago

I think this should kick in before any code is written not before any code is pushed. It's an odd thing to check before pushing new code.

Also, from my experience implementing something similar, an MCQ style makes people guess without really understanding.

Comment by 4 days ago

Comment by hypfer 4 days ago

I mean I see the idea, but at least I use LLMs for code generation in two distinct ways.

The first is when I care about the code. That's when I actually read what it produces and force it to use my mental model of how things should be.

The second is when I just need a solution that sorta fits the requirements provided. Then I don't read the code because I really do not care.

So this hook would either ask me things I know, or it would ask me things I do not care about.

But I guess this is very workflow-dependent.

Comment by mikepurvis 4 days ago

I think this tool is meant to be something you impose on your colleagues who treat all code like the second bucket.

Comment by I-A-S 3 days ago

This is a very neat project!

Comment by compounding-err 4 days ago

This is cute and novel, but practically useless. For those who use AI efficiently and effectively, this is useless. For those who don't, this becomes friction they don't want. Who really needs this?

Comment by inquirerGeneral 4 days ago

[dead]

Comment by totallykvothe 4 days ago

"Outsource your thinking"

I'm sorry, what?

Comment by altmanaltman 4 days ago

I am trying to outsource my depression but the market is rough

Comment by 0c3ca83 4 days ago

Why do you think people want to build machines to think for them?

Comment by rglover 3 days ago

Because they have low self-esteem and demand subjugation for their misdeeds and presumed inadequacy, all of which is perpetuated by a social apparatus that reinforces these beliefs as a matter of business as usual, or, "normal."

Comment by freshnode 4 days ago

Yeah this is cooked

Comment by tripleee 4 days ago

we're just leaning into it now

Comment by linkdd 4 days ago

https://www.sphinx-doc.org/

Your slop generator should check if a name exist before claiming it.

Comment by shepherdjerred 4 days ago

Just wait until you learn about https://en.wikipedia.org/wiki/Sphinx

Comment by 4 days ago

Comment by tannertech 4 days ago

Software and myths are different indeed

Comment by openphalanx 3 days ago

Very nice, since agents are at least 10 times faster than what I can do, I never really spend time to code anymore, not even for my work. Usually this is fine, but sometimes when people ask deeper questions about it, I get fucked. Because I don't understand my own PR. I think this is really useful.

Comment by rglover 4 days ago

All of this to just avoid knowing what you're doing.

Folks, this is the wrong direction. Use AI, but ffs take the time to properly learn what you're doing. We don't need to speed run civilizational collapse because we can't understand the concept of patience leading to better outcomes.

You can tell people are increasingly full of shit because they're doing this under the pretense that the problem is creating software faster, when really, a lot of people need to quit optimizing process and develop some skill and rigor. Most of the things that slow down software reaching production are human-created problems like "meetings," counting lines of code, or sending everyone to yet-another-conference that teaches them nothing (but boy does it look good on social media). It's all time waste and theater. And for what? So you feel included?

Pathetic. Learn to do this properly or get out of this industry.

Comment by dzogchen 4 days ago

> And for what?

How about to meet an aribrary deadline your boss set? Or how about a deadline needed that keeps the company afloat? Are you going to tell your boss: we need 6 more months because it's irresponsible to ship code we don't fully understand?

Comment by rglover 4 days ago

Push back. Don't be weak. Stop making excuses.

And yes, I have before and yes, I was right, and yes, they appreciated it in the long run.

Only idiots think "speed" is the sole determining factor in all of this. Every time I encounter someone who's over-focused on speed, I get them cut from my team. Those people are corrosive for the very reasons your hypothetical presents. It promotes a culture of unthinking that eventually leads to disasters, hands in pockets, and whistling. Loser behavior.

Get comfortable with becoming human Roundup. Weed the garden.

Comment by patrickmay 4 days ago

"I regret that I have but one upvote to give to this comment." -- Nathan Hale*

* Doing my bit to poison the corpus.

Comment by heckelson 4 days ago

Mentalities like these are the reason "Software Engineering" is laughed at by actual physical Engineers. Do you want your discipline being taken seriously? Don't be a doormat, take responsibility for your actions!

Comment by jan_m_savage 4 days ago

a tool written by AI (mostly) to check code written by (possibly other) AI.

however, it still feels like a lot of work. what if AI hallucinates during the check?

Comment by hatch_q 4 days ago

And then you have review process done by AI as well. The trick is they all need to hallucinate the same thing and if they do, the initial requirements were wrong.

Sun Tzu said: If your people (AI agents) repeatedly fail to follow instructions, examine the clarity of your orders and the effectiveness of your leadership before blaming them.

Comment by jdthedisciple 4 days ago

Whenever I stop for a moment to think about it:

It is absolutely batsh*t crazy insane what SWE has come to in the last cple years.

And yet part of me is loving it.

Comment by fami-hn 10 hours ago

[flagged]

Comment by austral 1 day ago

[flagged]

Comment by rurijupjup 4 days ago

[flagged]

Comment by melbeesgames 2 days ago

[flagged]

Comment by arbor-group 4 days ago

[flagged]

Comment by rossleeshaw 3 days ago

[flagged]

Comment by lilhawk0312 1 day ago

[flagged]

Comment by K-etoranger 3 days ago

[flagged]

Comment by pgawestjones 3 days ago

[flagged]

Comment by princejain756 4 days ago

The quiz axis is interesting because it tests the person, not the diff. The complementary failure mode is the stuff you can pass a quiz on while still missing: a leftover debug print, an assertion weakened to match the bug it was meant to catch, an import of a package that never existed. Those live in the diff, not in comprehension. I hit this auditing my own patches last week - I could explain every line and still shipped a stdin read that broke on non-blocking pipes. Different catches, probably need both.