Show HN: Lightweight Task queue on Erlang/OTP, SQLite-backed, no overengineering

Posted by ent1c3d 6 days ago

Setting up Kafka or such enterprise oriented software with their clusters or dedicated servers is heavy and bothering enough that most small teams or indie hackers skip it entirely and making compromise to use in-memory queues.

I wanted something in between: a persistent queue that is simple to run (one binary, which makes one sqlite db), gets real fault isolation and crash recovery due to Elixir, easy to inspect (open ezra.db in any SQLite browser and see every task), and requires no new client library - it speaks the Redis Streams wire protocol, so any Redis client in any language just works out of the box.

Very short demo video: [https://www.youtube.com/watch?v=MLYyD3DVWmE]

Comments

Comment by hankbond 3 days ago

Maybe this says more about me than your project, but I find the use of implementing this with the Redis Streams wire protocol very interesting and creative. Being able to leverage the surface area of existing client SDKs should really help adoption and reduce what you have to maintain in order to widely integrate.

I agree that in many areas there often is not much in-between "roll yourself in-memory" and "enterprise grade maxed out scalability focus".

Comment by ent1c3d 1 day ago

You perfectly got the point - building and maintaining SDKs for all popular languages is very time-consuming, and it also adds a barrier between your project and potential users, who have to get familiar with it, install it, and so on.

But, for sure, having your own SDKs has advantages too - you don't depend on anyone else's protocol decisions, have less dependency, eyc.

Btw, the first time I saw one company use another company's SDK, which I really liked, was in 2023, when DeepSeek used OpenAI's SDK and still think it was brilliant.

Comment by bhaney 4 days ago

> This project is maintained by a single author and pull requests are not accepted

Save yourself the headache of people not reading this and just disable pull requests in the repo settings

Comment by ent1c3d 2 days ago

that's a smart point. Just did it, thanks!

Comment by 3dedb728-3f77 1 day ago

So lightweight... Using docker and containers. What even is the point.

Comment by advertum 3 days ago

Using the Redis Streams wire protocol means any existing Redis client works without changes. The queue is stored on disk, so tasks survive a restart. No new client library to add. One question: if a worker stops in the middle of a task, does the message go back to the queue for another worker?

Comment by ent1c3d 2 days ago

[dead]

Comment by neoecos 4 days ago

Oban is really awesome, are you inspired by it?

Comment by ent1c3d 1 day ago

Maybe kinda shameful, but have to admit I just heard about Oban from you - and it seems an amazing thing. thanks for mentioning it!

Re inspiration: no direct inspiration really, just needed this tool myself for my own projects. But I guess this project as my engineering taste was kind of influenced/validated by bright minds like Richard Hipp, Joe Armstrong, and Derek Sivers

Comment by skrebbel 4 days ago

Title said “no overengineering” so I doubt it.

Comment by xlii 4 days ago

I would argue that Oban isn't overengineered.

If so should we also consider PostgreSQL overengineered?

It's a shame OP decided to use Elixir as base, many ecosystems don't have mature task queues (e.g. for Rust I had to roll my own: simple_queue) so the space IMO would be more welcoming.

On OTP doubt anything can even make a dent in Oban user base.

Comment by gamache 4 days ago

Satisfied former Oban user here. Oban is engineered. Your use-case may be petty, though, like your comment.

Comment by abrookewood 4 days ago

Congrats on the launch. Using the Redis protocol was a pretty clever choice. Does it have to run as a stand-alone server?

Comment by ent1c3d 1 day ago

Thanks, really glad you liked the idea to use redis protocol!

as an answer: tare actually two ways to run it. As a standalone binary, it's one process you run(you can easily run it via docker too) on its own(which speaks on Redis protocol over TCP) - and any language connects to it like a Redis server.

Or, if you're on Elixir, you can skip the separate process entirely and add it directly to your app's supervision tree - it runs inside your own app's process, no network hop, just function calls.

Comment by memset 3 days ago

This is really cool! I built something very similar only I replicated the AWS sqs protocol instead.

https://github.com/poundifdef/smoothmq

Comment by ent1c3d 1 day ago

oh, that's seems very nice project! I just wonder what it uses as storage/where it saves the tasks?

Comment by cpursley 4 days ago

This is nice. For those wanting to stay on Postgres for DAG type of workflows, check out pgmq based PgFlow: https://github.com/agoodway/pgflow