AX – Google’s Open Agentic Orchestrator

I don't really like the oversubscription of agent pods though, as you can no longer trust the k8s pod identity as being from a singular workload. Haven't seen a solution to this for ax yet and it is a barrier to adoption for us.

I'd prefer a K8s Operator or plugin (like Istio) to get all the pack.

Google operates at such a scale with a wide surface area of serious production considerations that even “ergonomic” solutions internally feel extremely heavyweight externally.

Source: I’m an Xoogler

That said, I think Google's ADK ecosystem and this new AX platform is promising--I would expect Google to maintain this and other tooling around this for years to come.

To the Googlers out there: is Google using this at any capacity for internal projects?

IMO, you want the flexibility to create either: (a) permanent devbox VMs, and (b) per-task VMs

Agent sandbox platforms tend to be tuned for the latter, which sometimes involves VMM hackery for fast boot, snapshotting VM filesystem and RAM, etc.

Some workflows are a lot simpler if the multiple agents share a VM. These are workflows where agents must share state. A simple one we have: making related changes in our public OSS repo and our private repo, and then testing the change.

And other times you want to split up the tasks onto isolated VMs so they don't interfere with each other (ie run two dev servers without database or port collisions).

I tweeted a bit about this (https://x.com/dbmikus/status/2099264325231771878) and had a little debate with folks about ephemeral vs persistent VMs for agents

[1]: https://github.com/gofixpoint/amika

Now the stuff people are coming up with is: how do you do authorization in this model? do you need a full sandbox all the time or can it be a workflow? how do you specify an agent is it a prompt or does it have some kind of control flow structure? How do you coordinate among many running agents?

I would say thats where we are now is there’s loads of people all solving the same problems a bit like when CoreOS, Kube etc. were all competing.

I've been working on https://lullabot.github.io/sandbar/latest/ which works with Proxmox for VMs (and lima for locals or regular linux hosts over ssh). There's a diagram in https://lullabot.github.io/sandbar/latest/why/#recommended-w... with what we're currently recommending. Though, after some feedback, I'm in the process of integrating a colleague's web-based review tool as it turns out many preferred fully reviewing locally instead of using draft PRs.

It's got some opinions in terms of default tools for our team and industry so it may not fit yours. Forgive some of the AI-isms in the docs, I want to get the UX and feature set to a solid place before doing a full review.

Yes, but, wrong layer here. Giving the agent a computer use (a la bash) is what folks are after. A temporary sandbox with lots of control knobs and security bits is how you do that in (as you noted) an enterprise.

The problem is that you'll end up wanting to run 2 or 3 (or 20, 100, 10,000) agents at once and that gets very hard with a single VM.

There's also an argument that you should be using a separate sandbox for each code operation a LLM performs (or at least each set of related operations). That's even harder to do with conventional VMs.

VMs are better for personal assistant work, GUI clicktesting, investigating bugs in your personal dogfooding dev instance and anything you haven‘t yet made repeatable and fast to set up.

Sandboxes are better when you need resource isolation or security and have a graph of tasks to work through. My agents often starve each other on one VM, so if they don‘t need any of the above it‘s just easier to isolate them.

Everyone is working in this area, including me [0], but either option really isn‘t that convenient to use yet. It‘s a bit of a „isn‘t Dropbox just FTP on a VM“ moment right now.

[0] https://github.com/madeinorbit/podium

Nothing at all. You'll know when you've outgrown it.

> what the general workflow is now that people are converging to?

Graph-based workflows where agents pick up work as it becomes available, structured output, while you manage the work queue and outcomes. Maybe? IDK really, it's all moving quite fast.

Works pretty well for me but I haven't put any effort into promoting it

- Do multiple tasks in the same context window / session, conflate different changes into the same prompt

- Repo mixed with old markdown files from previous tasks, excel and word docs and 300 playwright screenshots

- 5 tools all calling each other, test and deployment scripts are all markdown skills

Personally I prefer a ticketing system and isolated work trees

Probably we'll converge on a virtualised IO / Storage layer running microVMs beneath for isolation and security. Keep the network and storage layer separate for compatibility running a variety of stuff and a second security boundary.

I outgrew this when I wanted to bring different sets of skills and templates to different machines, wanted to be able to share a small number of credentials, different agents in different machines, different egress rules etc. I wrote https://github.com/pjlsergeant/byre which gives you a TUI and some machinery for doing this easily on top of Docker or Podman.

Only complaint is that connecting the agent harness to my local model took more work getting configured right than I'd like, but that's been true of most harnesses I've tried as well. Most assume you're using a cloud model and local model configuration is a bit of an afterthought.

Always Pi.

https://cloud.google.com/blog/products/ai-machine-learning/a...

We have used a few of Google's (smaller) open source projects, and in the last 2-3 years most of them are getting fewer updates if any updates at all. Some became very bad tech debt and we had to spend a lot of time migrating them.

Of course, that is the nature of open source projects (written in the license terms), and there is nothing to complain. But it's important to point out these days Google's open source project are not any more trustworthy than a one man's project in terms of support and maintainability. Personally I would stay away from them as far as possible. Especially if you look at what happened to Android, Gemini CLI etc.

(To be honest, even if it were officially supported by Google, that barely means anything. https://killedbygoogle.com/)

Google has around 200,000 employees. They probably haven't heard of most things Google releases.

We met quite a large number of customers in the last few months, and several teams inside Google, who are asking for a stack that runs on any cluster.

There are several reasons for a layer like this.

(a) Today, it's extremely hard to deliver large agentic applications to someone else's compute, and it kills adoption for products that need to run closer on customer data plane, e.g. large scale code scanning product.

(b) It's tedious to build systems that set up the environment, deal with networking policies, provide integrations for session storage, memory curation, skills retrieval and mounting, etc while all you want to do is to focus on building applications.

(c) People want a truly transparent stack for compliance/auditing, and all other non functional capabilities most people don't usually think about until they have to.

The web page presents ax as a typical developer tool, but it's actually not for developers.

>A Model is not a model. It is a named model configuration: ...

Remember kids, a model is not a model.

I think you could get by with 1 computer, but it’ll have to have a pretty decent machine.

Between agents running tests, CI, docker image builds, an average $400 mini PC won’t cut it.

Don’t forget also many older mini PCs don’t support KVM. Some newer ones don’t support AVX/ mongodb.

It’s not so easy to buy any old hardware sadly.

A standalone machine is nice if you need more compute resources or if you want an always-on machine you can connect to from your laptop, phone, etc.

It doesn't look like Google's AX is quite the plug-and-play fit for running agents on a computer you own, since it requires setting up a K8S cluster, etc.

I think what's needed is something like a zero-setup combo of Tailscale and Firecracker

I'm trying to work towards that with my startup (https://github.com/gofixpoint/amika) but the bring-your-own-computer part doesn't work quite yet.

Next we also need an instruction style guide and CoC. It's important to treat your agents with respect. I almost forgot, the YAML template meta-language to YAML the YAML. Then we will need a foundation employing 12 FTEs to maintain it all and of course to run the certification process. You are certified, right? Statistics show a 10x increased chance of an agent going rogue and hacking competitors if it has been mistreated or been run in an unvalidated sandbox. It goes without saying the sandbox certification process is separate and must be repeated yearly by a trusted third party auditing company.

Then put a “sandbox” around these agents, that word has many meanings. In this case they fence the network traffic, so likely some kind of allowlist for network requests so that the agent doesn’t exfil crap to random websites. They also limit the resource limits of the sandbox, so that is beneficial to the cost of running these agents.

must look cool for people who want to solve every problem with k8s

it starts with interesting misnomers like "Task" which is not a work item but a sandbox.

"billions of tasks" is a "solution" to problem nobody has (maybe some RL labs? but they solve it other way and with orders of magnitude better optimizations).

freezes design too early – unless they'll actually focus on developing it and make tons of breaking changes it looks shit.

shared state in the same workspace, identity, authority, etc – stuff like that needs to be solved

https://grapheneos.social/@GrapheneOS/117282080803799576

> Google should not be gatekeeping security patches to the standard Android platform code from Android OEMs but that's what they've started doing.

After about 3 messages in any given chat, a follow-up to "rewrite that with a more friendly tone" will result in a letter for a completely different job from another chat within the notebook.

Yes (perhaps not etcd, but the combination of kube-apiserver and etcd)

Substrate does not use CRDs for anything on the hot path of actor scheduling or resumption.

Personally I think that this kind of workload is better off not being tied too much into kubernetes. I've worked with crossplane and other controller who put a lot of load on the k8s-apiserver and etcd and can easily slow the whole machinery down / grind them to a halt.

btw, agent-substrate is in the process of being moved to CNCF: https://github.com/cncf/sandbox/issues/523

tens, hundreds, maybe a couple thousand at a time? absolutely.

Not sure if this is an extension of tech they already have had in their systems, but I've experimenting with it to build my own orchestrator and it's been a pretty neat set of tools and abstractions so far.

Overall I agree though, this is a bit of an abuse of that concept.

EDIT: I'm sure op is familiar with this workflow but I'm being overly verbose to clarify what I think they mean and my thoughts.

Even if you confine yourself to a dev workstation, having 5 agents concurrently building testing deploying code makes your computer loud and/or hot.

EDIT: if I HAD to use YAML, I'd prefer KYAML: https://dev.to/mechcloud_academy/goodbye-yaml-hell-meet-kyam...

With that said, I’ll somewhat disagree with you. I’ve been down the path you’re talking about and while it is incredibly flexible and powerful, it became too difficult to maintain, and too inconsistent between workflow runs, and a pretty hefty waste of tokens to use AI on things that could instead be handled by deterministic scripts. I ended up creating an orchestrator for myself that uses skills as the primary way to tell agents how to execute a step in a workflow, but also directly orchestrates running scripts and managing state in a deterministic way rather than leaving it all up to agents.

If you are using Qwen 27B you need very prescriptive skills.

If you are using Astra you usually want very minimal skills (because it will follow them but be unnecessarily constrained in some contexts)

If you are using Fable then it depends - it will take the skills as general guidelines but ignore them a lot more than Astra does. Sometimes this is good, sometimes not at all.

But for most things, I find resuming with memory is more trouble than it's worth. If you always resume from memory, you lose the ability to control the state of a VM. It's much easier to define which services should run than to define which active RAM state should be purged

Similar to why "did you try turning it on and off again?" is good for system reliability.

This is going to be decomposed. I believe the plan is to offer resumption with disk state only as well (and the RAM snapshots will need to be discarded from time to time, if you update the underlying code of the agent, or switch CPU types).

By the way, the RAM snapshots are not kept in RAM, they are serialized to disk, or uploaded to object storage.

For further isolation, I like to use nono inside a smolvm instance.

The website makes me think the contrary: It is described as “low opinion” and explicitly mentions that the running tasks don’t even have to be AI agents. Can you explain in what ways you’re more locked in than the website suggests?

Scion at the same time talks much more about concrete agents, giving me the opposite initial impression.

https://axllm.dev/