Jepsen: NATS 2.12.1

> 2. Delayed Sync Mode (Default)

> In the default mode, writes are batched and marked with needSync = true for later synchronization filestore.go:7093-7097 . The actual sync happens during the next syncBlocks() execution.

However, if you read DeepWiki's conclusion, it is far more optimistic than what Aphyr uncovered in real-world testing.

> Durability Guarantees

> Even with delayed fsyncs, NATS provides protection against data loss through:

> 1. Write-Ahead Logging: Messages are written to log files before being acknowledged

> 2. Periodic Sync: The sync timer ensures data is eventually flushed to disk

> 3. State Snapshots: Full state is periodically written to index.db files filestore.go:9834-9850

> 4. Error Handling: If sync operations fail, NATS attempts to rebuild state from existing data filestore.go:7066-7072"

https://deepwiki.com/search/will-nats-lose-uncommitted-wri_b...

People always think "theory is overrated" or "hacking is better than having a school education"

And then proceed to shoot themselves in the foot with "workarounds" that break well known, well documented, well traversed problem spaces

- ACKed messages can be silently lost due to minority-node corruption.

- A single-bit corruption can cause some replicas to lose up to 78% of stored messages

- Snapshot corruption can propagate and lead to entire stream deletion across the cluster.

- The default lazy-fsync mode can drop minutes of acknowledged writes on a crash.

- A crash combined with network delay can cause persistent split-brain and divergent logs.

- Data loss even with “sync_interval = always” in presence of membership changes or partitions.

- Self-healing and replica convergence did not always work reliably after corruption.

…was not downvoted, but flagged... That is telling. Documented failure modes are apparently controversial. Also raises the question: What level of technical due diligence was performed by organizations like Mastercard, Volvo, PayPal, Baidu, Alibaba, or AT&T before adopting this system?

So what is next? Nominate NATS for the Silent Failure Peace Prize?

There is also this [1] which Aphyr collabed on which you might find interesting if you haven’t seen it yet.

[1] https://antithesis.com/resources/reliability_glossary/

Many applications do not require true durability and it is likely that many applications benefit from lazy fsync. Whether it should be the default is a lot more questionable though.

Yes, exactly.

The kind of failure that a system can tolerate with strict fsync but can't tolerate with lazy fsync (i.e. the software 'confirms' a write to its caller but then crashes) is probably not the kind of failure you'd expect to encounter on a majority of your nodes all at the same time.

> They will refuse, of course, and ever so ashamed, cite a lack of culture fit. Alight upon your cloud-pine, and exit through the window. This place could never contain you.

https://aphyr.com/posts/340-reversing-the-technical-intervie...

Yes, and you shouldn't even need a fixed interval. Just queue up any writes while an `fsync` is pending; then do all those in the next batch. This is the same approach you'd use for rounds of Paxos, particularly between availability zones or regions where latency is expected to be high. You wouldn't say "oh, I'll ack and then put it in the next round of Paxos", or "I'll wait until the next round in 2 seconds then ack"; you'd start the next batch as soon as the current one is done.

One of the most used DB in the world is Redis, and by default they fsync every seconds not every operations.

I like that, and it allows me to build things around it.

For us when we used it back in 2018, it performed well and was easy to administer. The multi-language APIs were also good.

The trouble is that you need to specifically optimize for fsyncs, because usually it is either no brakes or hand-brake.

The middle-ground of multi-transaction group-commit fsync seems to not exist anymore because of SSDs and massive IOPS you can pull off in general, but now it is about syscall context switches.

Two minutes is a bit too too much (also fdatasync vs fsync).

Wait, isn't that the whole point of acknowledgments? This is not acknowledgment, it's I'm a teapot.

> I wouldn't trust a product that doesn't default to safest options

This would make most products suck, and require a crap-ton of manual fixes and tuning that most people would hate, if they even got the tuning right. You have to actually do some work yourself to make a system behave the way you require.

For example, Postgres' isolation level is weak by default, leading to race conditions. You have to explicitly enable serialization to avoid it, which is a performance penalty. (https://martin.kleppmann.com/2014/11/25/hermitage-testing-th...)

I got very deep into using NATS last year, and then realized the choices it makes for persistence are really surprising. Another horrible example if that server startup time is O(number of streams), with a big constant; this is extremely painful to hit in production.

I ended up implementing from scratch something with the same functionality (for me as NATS server + Jetstream), but based on socket.io and sqlite. It works vastly better for my use cases, since socketio and sqlite are so mature.

I understand that you need to make money. But you'll have to have a proper self-hosting offering with paid support as well before you're considered, at least by me.

I'm not looking to have even more stuff in the cloud.

[0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s...

[1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

To implement backpressure without relying on out of band signals (distributed systems beware) you need to have a deep understanding of the entire redis streams architecture and how the the pending entries list, consumers groups, consumers etc. works and interacts to not lose data by overwriting yourself.

Unbounded would have been fine if we could spill to disk and periodically clean up the data, but this is redis.

Not sure if that has improved.