Show HN: Walrus – a Kafka alternative written in Rust

Posted by janicerk 8 days ago

Comments

Comment by Barathkanna 5 days ago

Walrus isn’t trying to replace Kafka, but it does beat Kafka in a few narrow areas. It’s a lightweight Rust-based distributed log with a fast WAL engine and modern I/O (io_uring), so the operational overhead is much lower than running a full Kafka stack. If you just want a simple, fast log without JVM tuning, controllers, or the entire Kafka ecosystem, Walrus is a lot easier to run. Kafka still wins on ecosystem, connectors, and massive scale, but Walrus is appealing for teams that want the core idea without the complexity. Really impressed by the direction here, great work!!.

Comment by alexmorley 5 days ago

IMO for those requirements Redpanda has been the go-to for the last 5 years or so but I agree this is still a nice take and potentially even lighter.

Comment by lionkor 6 days ago

Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project.

We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.

Comment by kylecazar 5 days ago

"Consumption" works too :)

Assuming topics are consumed in your version, a la Kafka.

Comment by sgt 6 days ago

Imagine talking to your clients about tech stacks and "we're running tuberculosis" comes up... while people are dying from it.

Comment by lionkor 5 days ago

You just say "well, the alternative was Kafka" and they'd surely get it. Or not. Either way we imagined it to be hilarious.

Comment by ramses0 5 days ago

t10s, pronounced "tíos" or a stuttering "t- tents" on your geo. :-D

Comment by 5 days ago

Comment by dbacar 4 days ago

Is it me or is this analogy a bit too dark/insensitive?

Comment by enether 5 days ago

For Rust-based Kafka alternatives, I like Tansu[1]. It at least provides Kafka API parity, and critically also gives users a pluggable backend (embedded SQLite, S3 for low cost diskless type workloads and Postgres because just use Postgres)

It’s nice to try and out innovate Kafka, but I fear the network effect can’t be beaten unless the alternative is 10x better.

Something like Warpstream’s architecture[2] had a shot at dethroning Kafka, but critically even they adopted the Kafka API. Sure enough, Apache Kafka introduced a competing feature[3] within two years of warpstreams launch too.

[1] - https://github.com/tansu-io/tansu [2] - https://www.warpstream.com/ [3] - https://topicpartition.io/blog/kip-1150-diskless-topics-in-a...

Comment by roncohen 6 days ago

As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work!

Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging.

Notes:

- Would love to see it in an S3-backed mode, either entirely diskless like WarpStream or as tiered storage.

- Love the simplified API. If possible, adding a Kafka compatible API interface is probably worth it to connect to the broader ecosystem.

Best of luck!

Comment by seanhunter 6 days ago

It says on the github page

   " It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination."

So I guess that's a "yes" to raft?

Comment by zbentley 6 days ago

GP asked about data plane consensus, not metadata/control plane.

Comment by EdwardDiego 6 days ago

They asked about data plane replication - e.g., leader -> followers. Unless I misunderstood them.

Comment by nubskr 6 days ago

Hi, the creator here, I think its a good idea to have S3 backed storage mode, its kinda tricky to do it for the 'active' block which we are currently writing to, but totally doable for historical data.

Also about the kafka API, I tried to implement that earlier, I had a sort of `translation` layer for that earlier, but it gets pretty complicated to maintain that because kafka is offset based, while walrus is message based.

Comment by EdwardDiego 6 days ago

TBH I don't think anyone can utilise S3 for the active segment, I didn't dig into Warpstream too much, but I vaguely recall they only offloaded to S3 once the segment was rolled.

Comment by zellyn 5 days ago

The Developer Voices interview where Kris Jenkins talks to Ryan Worl is one of the best, and goes into a surprising amount of detail: https://www.youtube.com/watch?v=xgzmxe6cj6A

tl;dr they write to s3 once every 250ms to save costs. IIRC, they contend that when you keep things organized by writing to different files for each topic, it's the Linux disk cache being clever that turns the tangle of disk block arrangement into a clean view per file. They wrote their own version of that, so they can cheaply checkpoint heavily interleaved chunks of data while their in-memory cache provides a clean per-topic view. I think maybe they clean up later async, but my memory fails me.

I don't know how BufStream works.

The thing that really stuck with me from that interview is the 10x cost reduction you can get if you're willing and able to tolerate higher latency and increased complexity and use S3. Apparently they implemented that inside Datadog ("Labrador" I think?), and then did it again with WarpStream.

I highly recommend the whole episode (and the whole podcast, really).

Comment by EdwardDiego 2 days ago

Thanks, I have been highly remiss in catching up on that podcast :)

Comment by nubskr 5 days ago

s3 charges per 1,000 Update requests, not sure how it's sustainable to do it every 250ms tbh, especially in multi tenant mode where you can have thousands of 'active' blocks being written to

Comment by zellyn 5 days ago

Guess it beats doing it every 250ms for every topic…

Comment by k_bx 6 days ago

There's also Iggy https://github.com/apache/iggy

Never tried it, but looks promising

Comment by spetz 5 days ago

Thank you for the mention! BTW, we're currently working on VSR (Viewstamped Replication) to provide the proper clustering :)

Comment by tormeh 6 days ago

Looks like it has a solid amount of contributors. Exciting! Some other attempts like Fluvio seem to have lost momentum.

Comment by carverauto 5 days ago

iggy is amazing

Comment by yencabulator 18 hours ago

I see no mention of recovering redundancy of stored data after a node failure.

Comment by teleforce 6 days ago

For Kafka alternative written in C++ there's Redpanda [1],[2].

Redpanda claim of better performance but benchmarks showed no clear winner [3].

It will be interesting to test them together on the performance benchmarks.

I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus).

It's the very architecture of Kafka itself due to the notorious head of line problem (check the top most comments [4].

[1] Redpanda – A Kafka-compatible streaming platform for mission-critical workloads (120 comments):

https://news.ycombinator.com/item?id=25075739

[2] Redpanda website:

https://www.redpanda.com/

[3] Kafka vs. Redpanda performance – do the claims add up? (141 comments):

https://news.ycombinator.com/item?id=35949771

[4] What If We Could Rebuild Kafka from Scratch? (220 comments):

https://news.ycombinator.com/item?id=43790420

Comment by nubskr 6 days ago

In the current benchmarks, I only have Kafka and rocksdb wal, will surely try to add redpanda there as well, curious how walrus would hold up against seastar based systems.

Comment by chaotic-good 5 days ago

I don't see any mentions of p99 latency in the benchmark results. Pushing gigabytes per second is not that difficult on modern hardware. Doing so with reasonable latency is what's challenging. Also, instead of using custom benchmarks it's better to just use the OMB (open-messaging benchmark).

Comment by EdwardDiego 6 days ago

> It's the very architecture of Kafka itself due to the notorious head of line problem

Except a consumer can discard an unprocessable record? I'm not certain I understand how HOL applies to Kafka, but keen to learn more :)

Comment by thinkharderdev 5 days ago

> Except a consumer can discard an unprocessable record?

It's not the unproccessable records that are the problem it is the records that are very slow to process (for whatever reason).

Comment by pixelpoet 5 days ago

Yeah but how is a C++ based project supposed to hit the HN frontpage?

Comment by selkin 4 days ago

By having a use-after-free bug that caused a noticeable and catastrophic incident.

Comment by pixelpoet 4 days ago

Like that Cloudflare outage from the other day? In any case, I'm sure there's good reason to believe that, as Rust gains popularity, there totally won't be proportionally more high profile issues in stuff made with it.

Comment by selkin 4 days ago

You could write a lot of bugs in Rust. Or in any memory-safe language. Just not some bugs. It's just that starting a non-toy project using a non-safe language today is really in the "you could, but didn't stop think if you should" category.

Comment by pixelpoet 4 days ago

Fair enough, and I do appreciate Rust's contributions to the spectrum of programming language considerations. Personally I'd like to become a Zig guy, though.

There's also Fil-C BTW, and in normal C++ there are GCC or Clang (I forget which) extensions for detecting threading issues, even good old Valgrind is under-appreciated and under-used. In general one wants to adopt best practices and be proactive, rather than relying on the language to solve all problems (of course).

Comment by drob518 5 days ago

Or it’s I/O-bound.

Comment by jeberle 5 days ago

That was my first reaction. It's not like Java is terribly slow, so rewriting it in a slightly faster language seems like an empty exercise.

Comment by throw10920 5 days ago

What value does this provide other than being written in Rust? I'm pretty sure all of the five "key features" listed are either present in alternatives or already in Kafka itself.

Comment by gethly 5 days ago

I never understood the popularity of Kafka. It's just a queue with persistent storage(ie. not in-memory queu with ram-size limited capacity) after all.

Comment by mrkeen 5 days ago

A queue with persistent storage is like a ledger whose entries don't vanish when you read them, or a git branch whose commits stick around for longer than 24-72 hours.

It's popular because it didn't have any competition while it built up its ecosystem. And even though there's competitors now, I haven't had time to check them out, and they still brand themselves as "Kafka-alternatives".

Comment by sumtechguy 5 days ago

Most of the other ones at the time it was pop and the data was gone. You had to jump thru some hoops to make it work as persistent. Not 'hard' but just more annoying. Kafka has that out of the box. Where kafka starts to come apart is how to set it up. Its configuration is a bit tedious to setup.

Comment by lucyjojo 4 days ago

a huge amount of distributed programs work exceedingly well with the processing on a log structure. and kafka did it better than other at the times.

Comment by optician_owl 3 days ago

That's funny.

- Let's create a new Kafka in rust. Yeah!

- Let's create a Kafka client that's ready to use. Blah.

Comment by WD-42 5 days ago

Are those dips in the write throughput comparison for Kafka GC pauses?

Comment by oulipo2 6 days ago

Nice! How does it compare to Redpanda, NATS, etc?

Comment by Zambyte 5 days ago

Funnily enough I have been toying around with implementing a NATS compatible server using the new Zig Io interfaces (including the 0.16.0 preview) :)