Show HN: A local merge queue for parallel Claude Code agents

Posted by funador 3 days ago

I have been pushing up to 90 commits a day on a MacBook Air via 4-5 parallel agents. As you can imagine when all the agents try to build, test and run dev servers on an 8GB machine it is the fast lane to a force quit and restart. I also did not want to pay the CI minutes on 90 pushes a day.

So I designed a local merge queue to have all commits land one at a time and fully tested. Hopefully this helps other folks with more modest machines. Appreciate any feedback.

Comments

Comment by barrkel 3 days ago

A decent idea, however it seems to me a good chunk of it is because of git limitations. Have you looked jj?

I'm having a much better time with jj and a workspace per subagent than I was with git and worktrees.

It's still useful to have the CI gating on updating the master branch pointer, but you largely stop working with branches once you switch to jj.

Comment by grahamas 2 days ago

What would be the gotchas of doing this? I just spent some time looking into it and it looks ideal for my workflow, but I’d never heard of jj before right now. As far as I can tell the problems would be:

1. Worse-for-humans PR history (I currently use this for major feature checkpoints and human-involved sanity checks and review)

2. Maybe some tooling issues with IDEs? Looks like it’s all there, but maybe getting it set up so that the agents aren’t committing every little thought they have might be a bit finicky? I use VS Code if anyone has a workflow recommendation there.

Comment by barrkel 2 days ago

JJ stacked changes are just a chain of git commits. To jj, a branch is just a pointer to a commit, like git. Code review is no different to whatever you normally do for git. Push to your review system and review commit by commit Gerrit style, or as a single unit, or as stacked PR (I have not done this however as I don't use GitHub any more).

Comment by kazinator 2 days ago

git worktrees are a horrible misfeature; it's better to just clone multiple times (which you can do locally---space is saved by use of hard links!)

So that is to say, we clone some remote upstream down to /path/A. Then we can clone file:///path/A to /path/B locally. Both A and B are fully fleded git repos; no monkeying around with worktrees. The objects are shared between them with hard links. You can edit B/.gitconfig to point it to have the same upstream as A for pushing.

Unlike worktrees, you can blow away a repo with "rm -rf", and not worry that you are pulling the rug from under something which depends on it. If I have several trees, A, B, C, one of which is the parent, while the other are worktrees, removing any one of them randomly is Russian roulette: if we do "rm -rf B" and that happens to be the parent repo, worktrees A and C are no longer functional and need to be recovered. If A, B, and C are independent clones (even with objects hardlinked among themselves), this isn't a problem.

Comment by ithkuil 2 days ago

You're right about everything you said, but there is a flip side: with work trees you can:

1. find/list all related "clones" of the repo

2. Ensure that the same branch is not being worked on in different checkouts (which is useful if you do rebases as part of your workflow)

Worktrees force you distinguish the "main" repo from the swarm of checkouts, and if you locate or name directories with a rule you can always tell which are which.

For my own use, I built a little helper to create, list and destroy worktrees that fit in my workflow with Claude code:

https://github.com/mkmik/ccwt

Comment by kazinator 2 days ago

> Ensure that the same branch is not being worked on in different checkouts

worktrees ensure that the same local tracking branch is not being the basis for different trees. So when you do want to work on some independent things in parallel which share the same upstream branch, you have go through the annoyance of giving the local branches different names: master-2, master-alt, master-bug2359 ...

With multiple git repos instead of work trees, you just use the same name. All three of your repos can be on rev-3-branch: the same local name for the remote origin/rev-3-branch.

The most common example of this is multiple repos of the same upstream that are all on the default branch like master.

You can easily apply rebase workflow independently on all of them, and it's easily possible to add them to each other as remotes, to move commits sideways. If we commit something in repo A on master, and do a "git fetch A" in repo B, we can then easily "git cherry pick A/master" to get that commit into B's master.

Comment by yiyingzhang 2 days ago

What you described sounds like this: https://github.com/GenseeAI/gensee-crate. It offers whole workspace fork and merge. But the merge is still somewhat cumbersome and require manual overseeing.

Comment by funador 2 days ago

Giving up branches is the dream. This feels like the nudge I needed to give jj a proper go. Thank you.

Comment by ElijahLynn 2 days ago

Definitely going to look into this!

I just built something pretty much in the same vein of this with Claude over the past 2 months. It's a custom deployment system that uses work trees and each work tree has to to run end-to-end tests (Playright, Supabase, Astro) and other tests and create a stamp based on the contents of the entire tree. And then if another work tree has the same digest that matches the stamp then it can bypass the tests. So nothing can land on main without tests passing. Which is all in in enforced with a pre-push hook that checks the digest.

Then once it's on main it is deployed to Dev and then from there it can be promoted to prod.

I like the idea of just having one branch and the work trees funnel into it.

But definitely going to have my setup analyze this and see what can be improved with it or if I should just throw out mine and use that one.

Comment by funador 2 days ago

Sweet, let me know how it goes. Really appreciate any feedback

Comment by 2 days ago

Comment by throwaw12 2 days ago

idea looks nice, but can you share what's your workflow to push 90 commits/day?

I understand each commit might have different size, but I am guessing your commits == Pull Request (because you need this constant merge queue to rebase, merge, test) and each are around 40-80 lines of code (additions and removals), and you are probably not reviewing the code because at this rate even if one commit takes 2 minutes to review we are talking about 3 hours of non stop code review.

if you are not reviewing each code, probably your commits are small items and 10-20 of them are a part of single PR, then how do you sustain working across multiple projects?

Comment by funador 2 days ago

Definitely not reviewing the code when 90 commits go in a day :)

I have local test suite of unit / arch / e2e / build / lint / tsc when it merges to dev. Then the same suite runs on a merge to main which pushes to prod. I have post main-push e2e that checks for regressions and automatically rolls back.

Each suite run takes 3-4 minutes so I can usually push through 15 or so commits an hour fully tested and cleanly.

I do not use PRs with this workflow, and if something breaks I add tests to prevent it happening again rather than altering the setup.

Most of the commits are for a side project (https://hola.career/) so I afford to a bit more loose than I would in professional capacity.

Comment by orsorna 3 days ago

I have CI set up so that I can just let that handle builds and exclusive locks for me, but there are particular limitations (network connectivity is mandatory). I did this because I wanted to avoid something like you built, which theoretically can't scale as well as a dev box (much easier to extend deploy environments versus my local machine). I can see that you are working within resource constraints though which is admirable :)

Comment by funador 3 days ago

Exactly, have to shave a few corners when your laptop doesn't even have a fan :)

Comment by dpc_01234 2 days ago

SelfCI: https://radicle.network/nodes/radicle.dpc.pw/rad:z2tDzYbAXxT... , supports git & jj

Comment by funador 2 days ago

That looks like a much slicker version, will check that out

Comment by otekengineering 2 days ago

set up a channel for agents to communicate and you shouldn't have issues with messy merges.

your force quitting is probably more from your build pipeline than anything agents are doing.

i use an m2 8gb air with no performance issues at up to 10 parallel claude code interactive sessions (ghostty), each with deep/transient subagent heirarchies that mix claude and codex.

very rarely need to force quit. only seen a couple restart-level failures, and those were on me for being cowboy with subagent delegation logic, resulting in 100s of claude moles popping up faster than i could whack them with force close.

Comment by funador 2 days ago

I do have a pretty heavy pipe line of local tests and running more than one of those at once is what spurred me to write this package. Especially the build step.

Comment by 4b11b4 2 days ago

In codex threads can inspect each other, rename themselves , send each other messages. communication is default primitive

Comment by vasanthrb 2 days ago

One thing I'm wondering: have you tried this on a team with multiple humans as well, or is it mainly optimized for a single developer coordinating a bunch of Claude Code agents?

Comment by funador 2 days ago

There are a ton of places this package would not be suitable :)

I have not used this on a team, and what you would end up with is likely a giant PR with all those local commits that no one wants to review. Eventually someone might hold their nose and press the approve button. But that would probably be the end of the package at that workplace.

This is optimized for one engineer that does not have the machine capacity to run multiple lanes of their check test suites locally at once and also does want to pay for GHA minutes on a free plan to run those suites in the cloud.

Comment by 4b11b4 2 days ago

Lately I just use an isolated worktree per conversation and I have all these threads sitting which need to be merged then I just let codex orchestrate that itself.

I tell each thread to name itself with a prefix and it's picked up by a merge queue thread which instructs them merge one at a time. Each thread has to look at the main ref as it's changed since they did their work and determine what kind of merge needs to happen. To safely merge a a variety of merge types I have a commit skill which details the process for forward reconstruction, for example.

As of yesterday I have a single voice orchestrator thread which then can poke other threads as well as the merge queue thread.

Sometimes I just do stuff in working copy when I know exactly where in the history it's going to land (I rewrite history extensively to know cleanly where that it) and I'll git absorb / jj absorb it manually.