Vibe coding: Empowering and imprisoning

I was working on a new project and I wanted to try out a new frontend framework (data-star.dev). What you quickly find out is that LLMs are really tuned to like react and their frontend performance drops pretty considerably if you aren't using it. Like even pasting the entire documentation in context, and giving specific examples close to what I wanted, SOTA models still hallucinated the correct attributes/APIs. And it isn't even that you have to use Framework X, it's that you need to use X as of the date of training.

I think this is one of the reasons we don't see huge productivity gains. Most F500 companies have pretty proprietary gnarly codebases which are going to be out-of-distribution. Context-engineering helps but you still don't get near the performance you get with in-distribution. It's probably not unsolvable but it's a pretty big problem ATM.

> You can’t make anything truly radical with it. By definition, LLMs are trained on what has come before. In addition to being already-discovered territory, existing code is buggy and broken and sloppy and, as anyone who has ever written code knows, absolutely embarrassing to look at.

I don't understand this argument. I mean the same applies for books. All books teach you what has come before. Nobody says "You can't make anything truly radical with books". Radical things are built by people after reading those books. Why can't people build radical things after learning or after being assisted by LLMs?

I agree with the notion that LLMs may just end up repeating coding mistakes of the past because they are statistically likely mistakes.

I'm reminded of an old quote by Dijkstra about Fortran [1]: "In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included."

I've encountered that same problem in some older scientific codes (both C and Fortran). After a while, the bugs somewhat become features because people just don't know to question them anymore. To me, this is why it is important to understand the code thoroughly enough to question what is going on (regardless of who or what wrote it).

[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD04xx/EWD498...

Well the article went in a totally different direction to what I expected.

I do feel imprisoned by my vibe coded apps. But it's because I've lost control of them. I don't understand the code that was written well enough to work on it myself any more. So now I'm at the mercy of the AI to maintain it. When there's a bug to fix I just type it into the LLM and pray and hope that it will be able to solve it, because if it can't, I'm screwed.

It's actually a big reason I'm sticking with tools like Aider for things I care about and not going fully agentic yet. Having an agent do reams of work and then submit hundreds of lines of code for me to examine is counter productive to my longer term aims. I'm happy with the 2-3x improvement I get from lesser tools that still leave me in control.

> How does our imagination shrink when we consider our options of what we create with code to be choosing between the outputs of the LLM rather than starting from the blank slate of our imagination?

This has been my biggest hesitancy with adopting these technologies. All of the things of which I’m most proud of building were built from a foundation of deep understanding of several domains, not from the solutions of a series of one offs problems, but from the process of solving them.

There are too much of both fear and optimism in what's is essentially a better compiler and google.

Eventually we will gravitate back to square one, and business people are not going to be writing COBOL or VISUAL BASIC or the long list of eventual languages (yes this now include natural ones, like English) that claim to be so easy that a manager would write it. And Googling/Prompting remain a skill that surprisingly few has truly mastered.

Of course all the venture capital believe that soon we'll be at AGI, but like the internet bubble of 2001 we can awkwardly stay at this stage for quite a long time.

I’ve had a similar thought about language, and the evolution or lack thereof with LLMs.

With the printing press+internet, one might argue that we’ve helped cement current languages, making it harder for language to evolve naturally.

(A counterpoint may be slang/memes/etc. which has likely increased the velocity of new words for any given language.)

In either case, one might see LLMs as further cementing language, as it’s the thing the machines understand (until their next training run).

Assuming we struggle to make LLMs that learn in realtime, one might suspect that these amazing new tools might further cement the status quo, meaning less new words than before.

With all that said, I think I’ve come to the conclusion that LLMs will likely speed up the evolution of language.

The hypothesis being, that future generations will develop communication that the robots can’t read, at least at first.

A never-ending game of cat and mouse; while the cat is on v6, the mouse is on v7. Ad infinitum.

I am not sure I am doing it the right way or if there's a right way, but what I do is generate small files one at a time, with a functional system of stateless input/output. That way I can focus first on architecture, then on stateless input/output Lego blocks, and just ask the AI to generate the Lego blocks. Like this, everything is easy to keep in mind and update and change after experiencing the tech and having emotions telling me what to change. It has been working great for me. It strikes a good balance between simple and robust, fast to build, plus easy to update. The other thing is that since I haven't spent too much energy on writing the code, it is very easy to detach emotionally from it when reality gives me new info that requires throwing out part of the system.

Just finished my first fully vibe coded app. A proxy for Claude code to transparently switch between providers on the fly. Zero lines written by me

It works, but I wouldn’t give it to someone else to use because I have little idea how to fix or troubleshoot it beyond shouting at an LLM.

It’s weird.

I don’t think the authors thesis about „radical“ apps is right though. You can definitely arrange simple vibe coded components in novel ways. Much of the business logic out there powering billion dollar companies isn’t exactly mind bending innovative code

> A lot is still very uncertain, but I come back to one key question that helps me frame the discussion of what’s next: What’s the most radical app that we could build? And which tools will enable me to build it? Even if all we can do is start having a more complicated conversation about what we’re doing when we’re vibe coding, we’ll be making progress towards a more empowered future.

Why not ask ChatGPT?

But how much of this article was written by an LLM? Cliches, listicles, fluffy abstractions abandoned and not developed...

Was there anything original in it? I'd like to ask this article, what was your knowledge cut-off date?

“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” ― Frank Herbert, Dune

I'm not sure that "radical" apps aren't more often built because we don't know how, than because we don't have the time, funding, or risk budget to do it.

For those cases, I think LLM-assisted coding has the ability to drastically change the usual formula and help bring into being projects that people would previously only daydream about, hoping that the world aligns with their vision one day and they can magically spin up a team to work on it.

Coding agents are fast becoming at least a part of that team. If you're idea is in a domain where they've had a lot of high-quality training code, they can already do a pretty amazing job of getting a project off the ground. If you're a programmer with at least some domain knowledge and can guide the design and push the agent past tough spots, you can keep the project going when the LLMs get bogged down.

I think at the very least, we're going to see some incredibly impressive prototypes, if not complete implementations, of OSes, programming languages, hypermedia systems, protocols, etc. because one passionate programmer threw a lot of LLM time them.

Basically lots of people are going to be able to build their own TempleOS now. Some of those might end up being impactful.

[dead]

The entire premise which he summarizes as:

> A huge reason VCs and tech tycoons put billions into funding LLMs was so they could undermine coders and depress wages

is just pure speculation, totally unsupported, and almost certainly untrue, and makes very little sense given the way LLMs and ChatGPT in particular came about. Every time I read something from Anil Dash it seems like it's this absolutely braindead sort of "analysis".