How much can you delegate to agents?

Posted by duck 4 days ago

Comments

Comment by parasti 4 days ago

This post is full of things opposite of what I've found. An agent (claude code at least) will certainly refactor feature flags better than I ever could and is less likely to miss special cases. What the agent struggles with is business knowledge - context I hold in my head as a human communicating with other humans and working on a codebase over a long period of time.

Comment by DanielHB 4 days ago

One of the main problems I face is that business knowledge is not explicitly written down and formulated. No matter how much you describe something in human text format those are not formal rules about how something should work. If you can't inspect the output of the LLM you do not have full understanding of the business.

Most "classic" computer systems (code that runs on a CPU) are not strictly formal[1], but they are orders of magnitude more formal than human language.

[1] things like distributed systems, memory corruption, performance characteristics, etc can make a system behavior not predictable

Comment by ithkuil 4 days ago

That's true. But we always desired that more business knowledge be shared in written form between humans (people come and go, work across time zones, ...).

Presumably agents can both help maintaining/mediating such communication and also benefit from the context they acquire through that

Comment by embedding-shape 4 days ago

> Presumably agents can both help maintaining/mediating such communication

What happens in practice, is that people use agents to write such communication down, don't review it, and then the wrong information persists, and future agents will regurgitate it, until it reaches a human who already know it's wrong who then goes "Huh, where did this come from?".

Comment by jolaflow 4 days ago

Interesting. When it comes to auditing, I think there is also another layer to be verified - the intent. We have tooling for static analysis and automated tests, but drift in intent or vision is hard to detect.

After leaving agents unsupervised for longer periods of time, the biggest questions tend to be "how did it arrive at this conclusion?" and "did we drift from the initial vision?". Typically these questions are hard to answer as you only review the final state of the workflow.

I recently built an issue tracker that does offer time-traveling as a core feature. Somewhat unexpectedly this turned out to provide a much sought for overview allowing for a better workflow audit, as detailed here: https://dev.to/ljtn/vision-drift-addressing-the-next-problem...

Would be interested to hear your thoughts. I guess vision drift would classify as something that is hard to reverse, because you often don't realize it has happened until much later.

Comment by ithkuil 4 days ago

I see vision drift happen routinely in most (all) human led projects.

Comment by jolaflow 4 days ago

That is also very true, and I think time-travel capabilities would help detect that too.

Comment by steve1977 4 days ago

Would behaviour driven development be helpful here?

Comment by jolaflow 4 days ago

I think behavior driven development probably can make intent more explicit and standardized. However, I think they operate at different levels, or on a different axis if you will. BDD helps make each point in time easier to reason about wile being able to traverse the time axis helps understand how the workflow evolved.

Comment by ulfw 4 days ago

I have never once delegated anything to an agent and my life is still going on. Funny how that goes.

Comment by Incipient 4 days ago

I find agents are definitely useful as a blind audit/adversarial attack on a set of changes. A more comprehensive prompt, but in essence "go come up with a set of changes you'd make to achieve -set of requirements and then compare to what has actually been done and report back". I have CC so am more liberal with this towards the end my week when I have spare tokens!

Edit: I have no clue if they earn their tokens, but as I get them 'free' on the sub I haven't needed to evaluate.

Comment by ChicagoDave 4 days ago

I work on my MacBook in direct terminal sessions or in a container with no safeguards.

I save the container for well-planned units of work that can easily be repeated and validated on a branch.

I use direct sessions when I expect and the plan dictates that Claude might go off target and I need to hold the steering wheel.

Fable has increased my use of the container option.

Comment by ChulioZ 4 days ago

How do you draw the line between a human review at level 1 and a human safety gate at level 2?

Comment by SOLAR_FIELDS 4 days ago

This is a very long form way of saying “understand the work you are dispatching, and if you don’t, ask questions until you do”

Comment by Frieren 4 days ago

> “understand the work you are dispatching, and if you don’t, ask questions until you do”

My rant about this, not directed to the commenter but in general:

I wish people would have been so interested on discussing and planning with teammates like they seem with AI tools.

Because this is nothing new. Communication, knowing what you are doing, planning... was already a mandatory step for any engineering task.

And if people that use planning with AI then ignore their teammates, other teams, etc. then it is time to reconsidering their way of working.

Comment by SOLAR_FIELDS 4 days ago

Dealing with humans is way harder than dealing with a machine that is programmed to agree with you

Comment by sixtyj 4 days ago

Yes, shouting at clouds is easier with LLM :) But those clouds are deaf, sometimes.

Comment by h2aichat 4 days ago

Should the best agents be bottlenecked by the least verifable part of the workflow?

Comment by _davide_ 4 days ago

This post seems to be created out of thin air rather than real experience and data

Comment by apical_dendrite 4 days ago

There are so many other factors. Will people be pissed if they expected your output and got AI slop instead? Will you lose context on something that you really need to understand?