Celld: Self-hosted, distributed Durable Objects
Posted by calvinfo 1 day ago
Comments
Comment by sakesun 1 day ago
> Pull requests are disabled. Coding agents make it too easy to send a large,
> low-context change that costs maintainers more time than it saves.
> Thoughtful contributions are welcome; please understand the code,
> keep the patch focused, and respect the review time you are asking for.
>
> Send a git format-patch attachment to ...Comment by ryanrasti 12 hours ago
> At this time, we are not seeking outside contribution.
> AI has made writing code easy. The hard part, today, is not writing the code, but reviewing it, making sure quality stays high, and keeping the product coherent. In that light, unfortunately, external code contributions are "donating" the easy part of the job, while creating more of the hard work.
Feels very weird, but is logically sound: owners know exactly what they want and so they can work with Claude et al to iterate on features faster than with most drive-by contributors.
It reads like an "end of an era" but I imagine the steady state will be somewhere in the middle: high trust, high context contributors will still be able to contribute meaningful work.
Comment by aitchnyu 20 hours ago
Comment by dbmikus 1 day ago
Comment by 7bit 19 hours ago
Comment by szmarczak 20 hours ago
Comment by crabmusket 7 minutes ago
Comment by somerandom1 1 hour ago
celld's README states "Object-storage compare-and-swap ensures that exactly one node owns a cell at a time," but I'm skeptical this actually holds at the point where data is written to storage - I had an AI skim through the code with me, and the actual segment writes looked like plain, unconditional PUTs with no epoch check.
In many cases, I think using Cloudflare's Durable Objects is probably the right call instead.
Comment by rtpg 21 hours ago
Comment by wett 20 hours ago
More practical example (serverless WebSockets) https://youtu.be/FgWVoryZ8PU
Comment by thefreeman 15 hours ago
Comment by pelagicAustral 21 hours ago
Comment by jitl 1 day ago
Comment by rough-sea 1 day ago
celld is the full distributed system (albeit single tenant). It distributes DOs (cells) across any number of VMs. The databases for each DO are in object storage with RPO=0 guarantee.
Comment by kentonv 1 day ago
https://github.com/cloudflare/workerd/pull/6780
But honestly I love that there are multiple implementations now.
Comment by jitl 1 day ago
ah well, we have celld!
Comment by kentonv 1 day ago
The goal of this new design is to scale to a cluster while being operationally very easy to set up. Ideal for self-hosting.
NFSv4 is an easy first step, convenient because it's broadly understood, has many implementations, and requires no client libraries. I could imagine a follow-up to support LiteFS instead of NFS could make a lot of sense, though it'll get more complicated.
But yes, celld is definitely ahead of us here. No doubt about that.
Comment by keybits 18 hours ago
Comment by khalidx 1 day ago
So happy to see support for running durable objects outside of one provider. Upvoted.
The "durable object" concept has been repeatably demonstrated to be a valuable abstraction.
"Each object is its own SQLite database, addressed by name and replicated to an S3-compatible bucket you own" -- this concept can take you a long way, both in its power and simplicity.
Comment by jeremycarter 1 day ago
https://www.microsoft.com/en-us/research/wp-content/uploads/...
Comment by semicognitive 1 day ago
Comment by crabmusket 3 minutes ago
There was also Jamsocket and Plane, though it looks like they've shut down after acquisition: https://github.com/jamsocket/plane
This seems to be the first one which is providing drop-in compatibility for Cloudflare's JS-side APIs, including JSRPC etc.
Comment by dzonga 16 hours ago
Comment by jumploops 1 day ago
It worked really well! Excited to see more options outside of Cloudflare.
Comment by ironqcold 9 hours ago
It's just making it way easier to spam maintainers with code.
Good contributors are probably worth even more now.
Comment by meowtimemania 12 hours ago
Comment by hhthrowaway1230 1 day ago
Comment by wmf 1 day ago
Comment by myshapeprotocol 1 day ago
Comment by vmg12 1 day ago
Comment by wmf 1 day ago
Comment by jauntywundrkind 1 day ago
CloudFlare OS: https://blog.cloudflare.com/cloudflare-os/ https://news.ycombinator.com/item?id=49182996
And yes, celld is isolates! The very lightweight within v8 isolation boundary. Hence the "very low idle cost". The deno team rolled their own new runtime! This one without deno_core! Shout out to this excellent 2022 post: https://deno.com/blog/roll-your-own-javascript-runtime https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990
It's super super exciting having an age where many pieces of software can all run with low profiles. Very timely, again, for the unbelievably order of magnitude (and thats 10 to the not 2 to the, for you fellow computer nerds) ish ram crunch we are in. Next up, at some point, ideally we get some v8-like runtimes where we can share libraries across multiple isolates! Separating the shared code from the shared data, so we have multiple instances of the library, seems much harder. (It feels like wasm has some/much promise here and I eagerly await clearer signals that code-sharing while sandboxing is indeed possible)
Comment by reactordev 1 day ago
Comment by wartywhoa23 1 day ago
Comment by tdrz 1 day ago
I'm probably missing your point, but aren't wasm memories achieving exactly this? Just an example from wasmtime: https://docs.wasmtime.dev/examples-multimemory.html
Comment by 7bit 18 hours ago
Comment by jiveturkey 1 day ago
i would prefer a thing that was more self-contained, not dependent on a black box service layer underneath.
much apologies if i just have a poor understanding.
Comment by mrkeen 16 hours ago
It's like fixing all the problems with democracy by putting everyone in charge of their own 1-person election.
Comment by zuzululu 1 day ago
cloudflare gives you that instant geo-sync across the world that is hard to beat
with Celld do i need to buy bare metal in major continents
Comment by jitl 1 day ago
the stuff is great when it works but Durable Objects can be quite expensive. whenever i get too excited about em all it takes is a little time trying to price it out to calm me down.
> do i need to buy bare metal
my first idea would be to run celld on AWS Kubernetes deployed to local zones https://docs.aws.amazon.com/eks/latest/userguide/auto-local-...
it’s not “region: earth” like cloudflare but perhaps worth the trade off.
Comment by weego 17 hours ago
Comment by jitl 12 hours ago
Comment by wmf 1 day ago
Comment by vmg12 1 day ago
Comment by zuzululu 1 day ago
thats hard to beat
Comment by benatkin 1 day ago
So where it's overkill, this is where Celld is a good fit. It's a 100% open play, despite mentioning three hosted APIs: S3, Cloudflare Workers, and Durable Objects. The thing is, that Deno tried making similar ones. Deno Deploy is analagous to S3, and Deno KV is analagous to Durable Objects. However, with Celld the only dependencies are being able to run a binary and some S3-compatible storage. Cloudflare provides S3 compatible storage, but this README doesn't mention that, except in an example. Perhaps Ryan is leaving the door open for Deno to be acquired by Cloudflare.
FWIW here's an old Deno blog post comparing Durable Objects to other stuff: https://denoland.medium.com/deno-kv-vs-cloudflare-workers-kv...
Comment by jauntywundrkind 1 day ago
on the other hand, deno land folks stole from (used open source from) the best. the coordination layer/control-plane is all S3 CAS of dumb json files, which is a fantastic common-mode infra requirement for most orgs anyways, & perhaps a durable control-plane substrate you'd feel comfortable having someone else run (such as aws or others), while you run the data-plane (workers) yourself. and then for the durable objects themselves, they used litestream, which is a pretty top pick, excellent way to get radical distribution (but actual topology not included, some assembly required)! https://hn.algolia.com/?q=litestream
there's no reason this wouldn't run fine on most hosting, you definitely don't need bare metal. the virtues of v8 sandboxing / isolates! no need for vm's at all, no nested vm difficulties if you are trying to host on a shared host! but if you're asking questions like this, i want to again point you back to my top point.
Comment by q3k 1 day ago
... but that S3 _is_ the control plane and consensus layer, no? You're just pushing this down the stack to whoever runs that S3 clone.
Comment by jitl 1 day ago