Replacing Protobuf with Rust

> What I find particularly ironic is that the title make it feel like Rust gives a 5x performance improvement when it actually slows thing down.

Rust didn't slow them down. The inefficient design of the external library did.

Calling into C libraries from Rust is extremely easy. It takes some work to create a safer wrapper around C libraries, but it's been done for many popular libraries.

This is the first and only time I've seen an external library connected via a Rube Goldberg like contraption with protobufs in the middle. That's the problem.

Sadly they went with the "rewrite to Rust" meme in the headline for more clickability.

>> But had they written their software in C, they wouldn't have needed to do any conversion at all. It means they could have titled the article "How we lowered the performance penalty of using Rust".

That's not really fair. The library was doing serialization/deserialization which was poor design choice from a performance perspective. They just made a more sane API that doesn't do all that extra work. It might best be titles "replacing protobuf with a normal API to go 5 times faster."

BTW what makes you think writing their end in C would yield even higher performance?

I wonder why they didn't immediately FFI it: C is the easiest lang to write rust binding for. It can get tedious if using many parts of a large API, but otherwise is straightforward.

I write most of my applications and libraries in Rust, and lament that most of the libraries I wish I would FFI are in C++ or Python, which are more difficult.

Protobuf sounds like the wrong tool. It has applications for wire serialization and similar, but is still kind of a mess there. I would not apply it to something that stays in memory.

Given they heavily used LLMs for this optimization, makes you wonder why they didn’t use them to just port the C library to rust entirely. I think the volume of library ports to more languages/the most performant languages is going to explode, especially given it’s a relatively deterministic effort so long as you have good tests and api contracts, etc

> they had to use a Rust-to-C binding library, that uses Protobuf for portability reasons.

That sounds like a performance nightmare, putting Protobuf of all things between the language and Postgres, I'm surprised such a library ever got popular.

Protobuf does something important that copying memory cannot do: a protocol that can be changed separately on either end and things can still work. You have to build for "my client doesn't send some new data" (make a good default), or "I got extra data I don't understand" (ignore it). However the ability to upgrade part of the system is critical when the system is large and complex since you can't fix everything to understand your new feature without making the new feature take ages to roll out.

Protobuf also handles a bunch of languages for you. The other team wants to write in a "stupid language" - you don't have to have a political fight to prove your preferred is best for everything. You just let that team do what they want and they can learn the hard way it was a bad language. Either it isn't really that bad and so the fight was pointless, or it was but management can find other metrics to prove it and it becomes their problem to decide if it is bad enough to be worth fixing.

TIL serializing a protobuf is only 5 times slower than copying memory, which is way faster than I thought it’d be. Impressive given all the other nice things protobuf offers to development teams.

Yep.

Just doing memcpy or mmap would be even faster. But the same Rust advocates bragging about Rust speed frown upon such unsecure practices in C/C++.

Why don't we use standardized zero-copy data formats for this kind of thing? A standardized layout like Arrow means that the data is not tied to the layout/padding of a particular language, potential security problems like bounds checks are automatically handled by the tooling, and it works well over multiple communication channels.

How we used Claude and bindgen to make Rust catch up with C's 5x performance.

If they made the headline something on the line of "replacing protobuf with a native, optimized implementation" would not get the same attention as putting rust in the title to attract the everything-in-rust-is-better crowd.

It's devbait, not many of us can resist bikeshedding about the title which obviously doesn't accurately reflect the article contents. And the article contents are self-aware enough to admit this to itself too, yet the title remains.

I was equally confused by the headline.

I wonder if it's just poorly worded and they meant to say something like "Replacing Protobuf with some native calls [in Rust]".

Correct, this has very little to do with Rust. But it wouldn't have made the front page without it.

The title would suggest that it was already written in Rust; that it was the rewrite in Go that brought five times faster.

Yes you are absolutely right. The article even outright admits that Rust had nothing to do with it. From the article:

> Protobuf is fast, but not using Protobuf is faster.

The blog post reads like an unserious attempt to repeat a Rust meme.

In my experience capn proto is much less ergonomic.

I mean, cap'n'proto is written by the same person who created protobuf, so they are legit (and that somewhat jokish claim is simply that it requires no parsing).

You're saying we choose Protobufs [1] because Google maintains it but not FlatBuffers [2]?

[1] - https://github.com/protocolbuffers/protobuf: Google's data interchange format

[2] - https://github.com/google/flatbuffers: Also maintained by Google

I also thought I could trust mega Corp. That's why I put all my code on their platform, code.google.com, and not on this obscure platform without any business model, github. Well, that sucked. And why should I use protobuf, when I just need to share structs and arrays in memory (aka zero copy) with a version field? Like everyone else does for decades?

Yet they've yet to release their internal optimisation that allows zero-copying string-type fields.

Mmmm, I don't know maybe because your library DIDN'T EXIST before November 2025? Or perhaps for any other million reasons why people use Protobuf, and don't use Cap'n'proto and other 0-serialise libraries, like requiring a schema, established tooling for language of their choice, etc?

I don't know, but I have a guess. Someone didn't want to deal with unsafety of dealing with memory allocated in C code. Serialize/deserialize makes it easy, no need for unsafe, no need to learn all the quirks of the C-library allocating the memory.

I had experience with writing safe bindings to structures created in C library, and it is a real pain. You spend a lot of times reverse engineering C code to get an idea of the intent of those who had wrote the code. You need to know which pointers can address the same memory. You need to know which pointers can be NULL or just plain invalid. You need to know which pointers you get from C code or pass to it along with ownership, and which are just borrowed. It maybe (and often is) unclear from the documentation, so you are going to read a lot of C code, trying to guess what the authors were thinking when writing it. Generating hypotheses about the library behavior (like 'library never does THIS with the pointer') and trying to prove them by finding all the code dealing with the pointer.

It can be easy in easy situations, or it can be really tricky and time consuming. So it can make sense to just insert serialization/deserialization to avoid dealing with C code.

That's the experimental feature I was talking about! :)

Thank you so much for the kind words!

I dunno, are you sure you can manually write correct de/serializaiton for JSON and XML so strings, floats and integer formats correctly get parsed between JavaScript, Java, Python, Go, Rust, C++ and any other languages?

Do you want to maintain that and debug that? Do you want to do all of that without help of a compiler enforcing the schema and failing compiles/CI when someone accidentally changes the schema?

Because you get all of that with protobuf if you use them appropriately.

You can of course build all of this yourself... and maybe it'll even be as efficient, performant and supported. Maybe.

Well, protobuf allows to generate easy to use code for parsing defined data and service stubs for many languages and is one of the faster and less bandwidth wasting options

Type safety. The contract is the law instead of a suggestion like JSON.

Having a way to describe your whole API and generate bindings is a godsend. Yes, it can be done with JSON and OpenApi, yet it’s not mandatory.

Besides the other comments already here about code gen & contracts, a bigger one for me to step away from json/xml is binary serialization.

It sounds weird, and its totally dependent on your use case, but binary serialization can make a giant difference.

For me, I work with 3D data which is primarily (but not only) tightly packed arrays of floats & ints. I have a bunch of options available:

1. JSON/XML, readable, easy to work with, relatively bulky (but not as bad as people think if you compress) but no random access, and slow floating point parsing, great extensibility.

2. JSON/XML + base64, OK to work with, quite bulky, no random access, faster parsing, but no structure, extensible.

3. Manual binary serialization: hard to work with, OK size (esp compressed), random access if you put in the effort, optimal parsing, not extensible unless you put in a lot of effort.

4. Flatbuffers/protobuf/capn-proto/etc: easy to work with, great size (esp compressed), random access, close-to-optimal parsing, extensible.

Basically if you care about performance, you would really like to just have control of the binary layout of your data, but you generally don't want to design extensibility and random access yourself, so you end up sacrificing explicit layout (and so some performance) by choosing a convenient lib.

We are a very regularly sized company, but our 3D data spans hundreds of terabytes.

(also, no, there is no general purpose 3D format available to do this work, gltf and friends are great but have a small range of usecases)

In most languages protobuf is eaiser because it generates the boilerplate. And protobuf is cross language so even if you are working in javascript where json is native protobuf is still faster because the other side can be whatever and you are not spending their time parsing.

It's not just about traffic. IoT devices (or any other low-powered devices for that matter) also like protobuf because of its comparatively high efficiency.

Protobuf is fantastic because it separates the definition from the language. When you make changes, you recompile your definitions to native code and you can be sure it will stay compatible with other languages and implementations.

I never used it, coding since 1986.

Become a 5X Hacker News reader with this One Weird Trick.

Thank you!

I was terrified until it worked. The Postgres "ABI" is relatively stable - the parser only really changes between major versions and we bake the whole code into the same executable - largely thanks to the work done by team behind pg_query!

The output is machine-verifiable, which makes this uniquely possible in today's vibe-coded world!

Explicit tail call optimization is in the works but I don't think it's available in stable jut yet.

The `become` keyword has already been reserved and work continues to happen (https://github.com/rust-lang/rust/issues/112788). If you enable #![feature(explicit_tail_calls)] you can already use the feature in the nightly compiler: https://play.rust-lang.org/?version=nightly&mode=debug&editi...

(Note that enabling release mode on that link will have the compiler pre-calculate the result so you need to put it to debug mode if you want to see the assembly this generates)

> I vaguely recall that there's a Rust macro to automatically convert recursive functions to iterative.

Isn't that just TCO or similar? Usually a part of the compiler/core of the language itself, AFAIK.

Title is as nonsensical as "We replaced Windows with ARM CPUs"

I loved, every clickbait title should come with a tldr just like this one.