A queue, to understand queues

I could configure a queue, but I couldn't explain what happens to a message when the consumer crashes halfway through it. Reading docs didn't fix that. Building one might.

One binary, one disk, the Rust standard library and a checksum crate. At-least-once delivery. It had to survive kill -9 at any moment without losing an acknowledged message.

An append-only log split into fixed-size segments. Producers append to the end. Each consumer group keeps one number: the offset of the next record it will read.

Calling fsync for every message made the disk the bottleneck. I batch instead: records wait up to 5 ms, then one fsync covers all of them.

A producer only hears "ok" after its batch is on disk, so every write can wait up to 5 ms longer. I chose durability over latency and wrote the number in the README.

Each record is a length, a CRC32 and the payload. On start, the last segment is scanned and cut at the first record whose checksum fails, which is where the crash happened.

My first consumer saved its offset before processing the message. A crash between those two lines lost the message silently. The tests passed because none of them crashed in that gap.

At-least-once is decided by the order of two lines in the consumer, not by the broker. Commit after the work, and make the work idempotent, because now duplicates will arrive.