标签：#hermitmq

I built HermitMQ entirely in Go. The main feature is ditching heavy wrappers like JSON in favor of a custom 29 byte binary protocol. Additionally, data transmission over the network uses a direct file to socket copy mechanism. I will go into detail about the architecture, data storage approaches, benchmark numbers, and show how it is implemented in code. The full source code for the HermitMQ project is available on GitHub: https://github.com/ekhidirov/hermitmq The problem with standard brokers and the cost of serialization When the message counter exceeds hundreds of thousands per second, the main problem for a Go developer is the garbage collector. If every message is parsed via standard JSON, the application starts allocating a massive number of small objects in memory. The GC wakes up too frequently, eating up CPU time and causing network latency spikes. To avoid triggering the garbage collector at every turn, I completely abandoned standard serialization libraries. Every message is packed into a custom header of exactly 29 bytes. In code, the message structure looks extremely simple: type Message struct { Magic byte Timestamp uint64 Offset uint64 KeySize uint32 PayloadSize uint32 RecordCount uint32 Key [] byte Payload [] byte } The first byte is a magic number for version checking and instantly discarding bad packets. Next come 8 bytes for the timestamp in nanoseconds and 8 bytes for the offset, which the broker fills in itself to maintain message order. Then come the key and payload sizes, 4 bytes each. Finally, 4 bytes are reserved for the record count to support batching. The broker reads the stream using the binary package and reuses buffers via sync.Pool. As a result, under standard loads, we achieve practically zero memory allocation. Being honest about allocations and plans for zero serialization To be completely honest: although the broker is incredibly frugal under standard loads, a memory management compromise still remains. An absolute victory over al