Trade-offs vs Redis:
- Not for multi-region distributed systems
- Best for single server or small clusters
Happy to answer any questions about the architecture!
nakovet 3 days ago [-]
We are using a service that abstracts redis from us and requires to be treated like a critical dependency, think RDS, Aurora, Postgres, if they are down the whole site is down. Every job push is a call to this service. Upgrading the service = downtime.
For us this is resulted in a big weak point on our architecture because when the service reboots both job pushing and job pulling stops, with the pushing being on the API side bringing the API down. With containers we could have multiple of them running at the same time, but the shared reading/writing of the abstract Redis locks itself.
We are considering BullMQ, because the architecture is sane:
* job push: API writes to Redis
* job pull: Worker reads from Redis then writes the completion.
How do you see this issue for Bunqueue? What happens when it goes down for 5 minutes, can the jobs be enqueued? Can you run multiple instances of it, failover?
Our throughput (jobs/sec) is small we do have 100k+ scheduled jobs anywhere from minutes to months from now.
kernelvoid 3 days ago [-]
Transparent answer about bunqueue's architecture.
Current state: bunqueue is single-server with SQLite persistence.
If the server goes down for 5 minutes, clients cannot push/pull during that window. However: the client SDK has automatic reconnection with exponential backoff + jitter, all data is safe on disk (SQLite WAL mode), and on restart active jobs are detected as stalled and re-queued automatically. Delayed/scheduled jobs resume from their run_at timestamps.
For your use case (100k+ scheduled jobs, low throughput): well-optimized. We use MinHeap + SQLite indexes for O(k) refresh where k = jobs becoming ready, not O(n) scan.
What bunqueue does NOT have today: no clustering, no multi-instance with shared state, no automatic failover, no replication.
What it does have: S3 automated backups (compressed, checksummed) for disaster recovery. A "durable: true" option for zero data loss on critical jobs. Zero external dependencies.
Roadmap: HA is something we're actively working toward. Native HA with leader election and replication. Managed cloud offering with automatic failover and geographic distribution.
Bottom line: if you need true HA today, BullMQ + Redis Sentinel/Cluster is the safer choice. bunqueue is for when you want simplicity, high performance (~100k jobs/sec), and can tolerate brief downtime with automatic recovery.
nakovet 2 days ago [-]
Thank you for your detailed reply! Appreciated it
stephenr 2 days ago [-]
> Best for single server or small clusters
How would this work for anything _but_ a single server?
kernelvoid 2 days ago [-]
Fair point, that was misleading. bunqueue is single-server only today. “Small clusters” is on the roadmap but I should’ve been clearer it’s not there yet.
tombert 2 days ago [-]
At Apple, on iTunes, we used an Oracle database to do job queue stuff. Initially I will admit I made fun of it because I wanted to use a "real" queue like RabbitMQ or ActiveMQ or something, but I have to admit that it worked fine and to be fair it did predate both of those.
Anyway, it made me realize that there's really no reason you can't use a SQL database as a backing store for queue stuff. I should try building my own at some point.
dawnerd 2 days ago [-]
I remember in my early dev days thinking I was clever for creating a queue. Was right in MySQL handing forum posts back when building your own forum software was a normal thing. It worked fine. Was it optimal? Not really but neither was the entire app. Didn’t matter at the time and probably doesn’t matter today either for most apps.
kernelvoid 2 days ago [-]
True, most apps will never hit the scale where it matters. But when you do, retrofitting a queue is painful.
kernelvoid 2 days ago [-]
This is exactly why I built bunqueue — a job queue for Bun backed by SQLite. No Redis, no external dependencies, just bun:sqlite with WAL mode for concurrent access. Handles 100k+ jobs/sec on a single node.
The SQL-as-queue pattern is definitely underrated. Great to hear it worked well at that scale.
lelanthran 2 days ago [-]
> This is exactly why I built bunqueue — a job queue for Bun backed by SQLite. No Redis, no external dependencies, just bun:sqlite with WAL mode for concurrent access. Handles 100k+ jobs/sec on a single node.
If you're too lazy to even write your own comments, I suspect you're too lazy to have written your own software.
At least preface your comment with "The LLM says" or preface your submission with "The LLM wrote this software".
mpalmer 2 days ago [-]
To be honest I don't see anything egregious here.
pyrolistical 2 days ago [-]
Why do you think that was written with a LLM?
MitziMoto 2 days ago [-]
Because it has an emdash and people can't fathom that real people use emdashes. Like the LLMs didn't learn it from somewhere.
kovek 1 days ago [-]
I know people who don't speak English fluently who like to use LLMs to translate to English.
lelanthran 2 days ago [-]
"No $X, just $Y. $CONCLUSION"
kernelvoid 2 days ago [-]
No magic, just good engineering. That’s it.
kernelvoid 2 days ago [-]
The comment sounds "polished" because I've probably described this project dozens of times at this point.
When you repeat the same thing over and over you naturally end up with a tight version of it. That's not an LLM, that's just how it works when you talk about something a lot.
And honestly even if I did use an LLM to write a comment on HN, so what? The code is what matters.
Go run the benchmarks, read the source, open an issue if something breaks.
That's the part that actually counts.
lelanthran 2 days ago [-]
> The comment sounds "polished" because I've probably described this project dozens of times at this point.
I didn't say it sounded "polished", I said exactly the opposite.
> And honestly even if I did use an LLM to write a comment on HN, so what?
If we wanted to chat with bots, we know where to find them.
tombert 2 days ago [-]
> And honestly even if I did use an LLM to write a comment on HN, so what? The code is what matters.
Part of what makes these forums fun is human responses. LLMs write "good enough" text but they come off as robotic and inhuman. The only reason to go onto one of these forums is to communicate with people. If I wanted to talk to a robot, I would talk to ChatGPT, which I can do as often as I want.
kernelvoid 2 days ago [-]
I get the concern, but there’s a spectrum.
Using an LLM to polish grammar vs. having it generate opinions wholesale are different things.
tombert 2 days ago [-]
I'm not necessarily saying that you used an LLM, but em dashes aren't used that often when regular people are typing. I use Grammarly all the time and they've never suggested that I add an em dash, and it's often a sign of a low-effort "ask ChatGPT for a response to this comment".
Again, not necessarily saying that's what you did, just that that's the red flag.
Rendered at 14:38:32 GMT+0000 (Coordinated Universal Time) with Vercel.
The idea: for single-server deployments, SQLite can handle 100k+ ops/sec with WAL mode, so why add infrastructure?
Features: priorities, delays, retries, cron jobs, DLQ, job dependencies, BullMQ-compatible API.For us this is resulted in a big weak point on our architecture because when the service reboots both job pushing and job pulling stops, with the pushing being on the API side bringing the API down. With containers we could have multiple of them running at the same time, but the shared reading/writing of the abstract Redis locks itself.
We are considering BullMQ, because the architecture is sane: * job push: API writes to Redis * job pull: Worker reads from Redis then writes the completion.
How do you see this issue for Bunqueue? What happens when it goes down for 5 minutes, can the jobs be enqueued? Can you run multiple instances of it, failover?
Our throughput (jobs/sec) is small we do have 100k+ scheduled jobs anywhere from minutes to months from now.
Current state: bunqueue is single-server with SQLite persistence.
If the server goes down for 5 minutes, clients cannot push/pull during that window. However: the client SDK has automatic reconnection with exponential backoff + jitter, all data is safe on disk (SQLite WAL mode), and on restart active jobs are detected as stalled and re-queued automatically. Delayed/scheduled jobs resume from their run_at timestamps.
For your use case (100k+ scheduled jobs, low throughput): well-optimized. We use MinHeap + SQLite indexes for O(k) refresh where k = jobs becoming ready, not O(n) scan.
What bunqueue does NOT have today: no clustering, no multi-instance with shared state, no automatic failover, no replication.
What it does have: S3 automated backups (compressed, checksummed) for disaster recovery. A "durable: true" option for zero data loss on critical jobs. Zero external dependencies.
Roadmap: HA is something we're actively working toward. Native HA with leader election and replication. Managed cloud offering with automatic failover and geographic distribution.
Bottom line: if you need true HA today, BullMQ + Redis Sentinel/Cluster is the safer choice. bunqueue is for when you want simplicity, high performance (~100k jobs/sec), and can tolerate brief downtime with automatic recovery.
How would this work for anything _but_ a single server?
Anyway, it made me realize that there's really no reason you can't use a SQL database as a backing store for queue stuff. I should try building my own at some point.
The SQL-as-queue pattern is definitely underrated. Great to hear it worked well at that scale.
If you're too lazy to even write your own comments, I suspect you're too lazy to have written your own software.
At least preface your comment with "The LLM says" or preface your submission with "The LLM wrote this software".
When you repeat the same thing over and over you naturally end up with a tight version of it. That's not an LLM, that's just how it works when you talk about something a lot.
And honestly even if I did use an LLM to write a comment on HN, so what? The code is what matters.
Go run the benchmarks, read the source, open an issue if something breaks.
That's the part that actually counts.
I didn't say it sounded "polished", I said exactly the opposite.
> And honestly even if I did use an LLM to write a comment on HN, so what?
If we wanted to chat with bots, we know where to find them.
Part of what makes these forums fun is human responses. LLMs write "good enough" text but they come off as robotic and inhuman. The only reason to go onto one of these forums is to communicate with people. If I wanted to talk to a robot, I would talk to ChatGPT, which I can do as often as I want.
Using an LLM to polish grammar vs. having it generate opinions wholesale are different things.
Again, not necessarily saying that's what you did, just that that's the red flag.