标签：#queue

Five independent clients on one free AI server will produce 429s and a thundering herd unless you add a fair queue. We fixed it with a client-side asyncio queue that capped concurrency at two, prioritized interactive work, and dropped 429s from 23 to 0 on a 100-request mixed workload. Disclosure: This article was prepared as part of MonkeyCode's product outreach. What Failed When Five Developers Shared One Server We shared one MonkeyCode free server for code review and refactoring. Each of us ran our own scripts. Nobody coordinated. The first symptom was latency: requests that took two seconds started taking thirty. Then came the 429s. Then came the retries. Retries made everything worse. The server spent more time rejecting requests than answering them. The timeline compressed quickly: Day 1: two developers, no issues Day 3: four developers, latency doubles Day 5: five developers, 429s appear Day 6: retries cause a thundering herd Day 7: the team stops using the server The root cause was not the server. It was the absence of coordination. Five independent clients hammered one endpoint. Each client assumed it was the only user. The server had no way to prioritize. HTTP 429 is the standard “too many requests” signal; we treated it as a retry cue instead of backpressure. That is how a shared free endpoint turns into a retry storm. The deeper problem was architectural. Each of us built a separate integration. Each integration had its own retry logic. Under load those retries multiplied. The server received about five times the intended traffic, not because we needed five times the work, but because five clients were guessing independently. Contrast the two modes we actually ran: Uncoordinated: five scripts, five retry loops, unbounded in-flight calls, no shared view of queue depth. Coordinated: one process, one priority heap, two in-flight calls, explicit rejection when the queue is full. The first mode failed in a week. The second mode is what we shipped. How We Built

Short answer: for a small edtech SaaS sending a weekly digest in Europe and the US, persist one idempotent delivery job per customer and week, then start with a polling worker; adopt queue push or subscription delivery only when measured queue delay, regional isolation, or worker operations justify a public HTTPS receiver. The transport is not the guarantee. A public webhook can be retried, a subscriber can redeliver, and a polling loop can crash after sending but before recording success. In all three designs, the hard boundary is the same: a durable job identity, an atomic claim, an expiring lease, and a delivery operation that tolerates repetition. Get those right first. The easiest setup is then the one with the fewest independently failing parts your team must operate, not the one with the shortest quick-start page. This matters for a weekly digest because duplicates damage trust while an omitted message is difficult to notice. A customer who was active at the cutoff must map to a stable key such as customer_id + digest_week ; changing from polling to push must not change that identity. What delivery guarantee does the weekly digest actually need? “Exactly once” is an application outcome, not a useful promise to infer from a queue label. There are at least four moments to distinguish: eligibility is calculated, a job is committed, a worker claims it, and the downstream delivery system accepts it. A process can stop between any two writes. If it stops after acceptance but before the job is marked complete, retrying is the conservative action, and that retry can duplicate the digest unless the downstream operation accepts the same idempotency key. Write the contract before choosing a transport: Every active customer at the weekly cutoff gets one durable job. A job may be attempted more than once. The same digest_key is used on every attempt and is unique in the ledger. A claim expires, so a stopped worker cannot own work forever. Operators can distinguish pending

What Is a Message Queue? A message queue is a buffer that stores messages between producers and consumers. Producers send data to the queue, and consumers read from it. The queue decouples the two sides so they don't need to know about each other. This is a core pattern in distributed systems. Think of it like a restaurant ordering system. You (the producer) write your order on a ticket and put it on a spindle. The kitchen (the consumer) picks tickets off the spindle when they're ready. You don't shout at the chef, and the chef doesn't wait for you. The spindle is the queue. Why Use a Message Queue? Three big reasons: Decoupling : Producers and consumers evolve independently. You can change one without touching the other. Buffering : Producers can run faster than consumers. The queue absorbs spikes and prevents overload. Scaling : You can add more consumers to handle more load, or more producers to generate more work. Core Concepts Producer : Sends messages. Consumer : Receives messages. Queue : Stores messages until consumed. Broker : The server that hosts the queue (e.g., RabbitMQ, Kafka, Redis). Acknowledgment : When a consumer tells the broker it successfully processed a message. Dead Letter Queue : Where messages go if they can't be processed after retries. Simple Example with Redis Redis has a simple list-based queue using LPUSH and BRPOP . Here's a minimal Python example using redis-py . import redis import time r = redis . Redis ( host = ' localhost ' , port = 6379 ) # Producer r . lpush ( ' tasks ' , ' send_email ' ) r . lpush ( ' tasks ' , ' generate_report ' ) # Consumer (blocking pop) while True : task = r . brpop ( ' tasks ' , timeout = 5 ) if task : print ( f " Processing: { task [ 1 ]. decode () } " ) time . sleep ( 1 ) # simulate work else : break This is a simple FIFO queue. It works for basic cases but lacks features like acknowledgments, retries, and routing. Real-World Example with RabbitMQ RabbitMQ is a full-featured broker. Here's a producer an

Safe retry: idempotency key để retry một request không biến thành hai lần charge Trong hệ phân tán, retry là mặc định — client, gateway, load balancer, queue consumer đều retry khi timeout hoặc lỗi tạm thời. Vấn đề là nhiều thao tác quan trọng không idempotent tự nhiên: một request POST /charges gửi hai lần thì trừ tiền khách hai lần, một message OrderCreated xử lý hai lần thì ship hai đơn. Idempotency key là cơ chế để server nhận diện "cùng một intent" giữa các lần retry và chỉ thực hiện side effect một lần , trong khi vẫn trả về response giống hệt cho mọi lần gọi lặp — về mặt hiệu ứng thấy được từ bên ngoài, đây là cái người ta hay gọi là "exactly-once effect" (dù ở tầng transport vẫn là at-least-once). Cơ chế hoạt động Client sinh một identifier duy nhất cho mỗi thao tác (thường là UUIDv4) và đính kèm request — quy ước phổ biến là HTTP header Idempotency-Key (Stripe API dùng đúng tên này, và IETF có draft draft-ietf-httpapi-idempotency-key-header chuẩn hoá cùng tên header). Server dùng key làm identity của thao tác trong một cửa sổ TTL: Nhận request với key K . Tra K trong idempotency store. Nếu tồn tại và request cũ ở trạng thái terminal (đã có response), trả lại response đã lưu — không chạy lại business logic. Nếu tồn tại nhưng đang in-flight , trả 409 Conflict (hoặc chờ, tuỳ contract). Nếu chưa tồn tại, INSERT bản ghi với unique constraint trên key, chạy business logic, persist response, commit. Điểm cốt lõi là bước insert + bước business logic + bước lưu response phải nằm trong cùng một transaction boundary — hoặc chí ít, phải có cơ chế đảm bảo không có window mà một retry khác nhìn thấy "chưa có key" trong lúc lần đầu vẫn đang chạy dở. CREATE TABLE idempotency_keys ( key TEXT NOT NULL , user_id BIGINT NOT NULL , request_hash TEXT NOT NULL , -- fingerprint payload status TEXT NOT NULL , -- in_flight | succeeded | failed response_code INT , response_body JSONB , created_at TIMESTAMPTZ NOT NULL DEFAULT now (), locked_until TIMESTAMPTZ , PRIMARY KEY ( user_id ,

Async processing qua message queue: vì sao đẩy việc nặng ra khỏi request path, và cái giá phải trả bằng eventual consistency Async processing là mô hình tách một request thành hai giai đoạn: request handler nhận việc, xác nhận với client, rồi giao phần xử lý thật cho một worker chạy ngoài request path — thường qua một message queue (RabbitMQ, AWS SQS, Kafka, Redis Streams, hoặc queue trên nền Redis như BullMQ/Sidekiq). Lý do dev gặp nó trong việc thật rất cụ thể: một endpoint gọi payment provider mất 3s, gửi email confirm mất 1s, resize ảnh mất 5s — nếu làm tuần tự trong request, p99 latency của endpoint là tổng các con số đó, và một downstream chậm hoặc chết đủ để làm timeout hết thread pool của app server. Đẩy vào queue thì request trả về trong vài chục ms; nhưng đổi lại, cái "xong" mà client thấy không còn nghĩa là việc đã thực sự hoàn thành. Cơ chế hoạt động Ba thành phần: producer (thường là API server) đóng gói việc thành message rồi publish vào broker; broker (RabbitMQ/SQS/Kafka…) giữ message trong queue có persistence tuỳ cấu hình; consumer/worker poll hoặc được push message, xử lý, rồi ack để broker biết xoá. Nếu worker chết trước khi ack, broker redeliver — đây là gốc của semantic at-least-once : mỗi message được giao ít nhất một lần, có thể nhiều lần. Exactly-once trong hệ phân tán chỉ đạt được ở lớp application bằng cách consumer viết idempotent, không phải bằng cấu hình broker. Ví dụ với RabbitMQ + Node ( amqplib ): // producer — trong HTTP handler const ch = await conn . createConfirmChannel () await ch . assertQueue ( ' image.resize ' , { durable : true }) app . post ( ' /upload ' , async ( req , res ) => { const jobId = crypto . randomUUID () const payload = Buffer . from ( JSON . stringify ({ jobId , s3Key : req . body . key })) await ch . sendToQueue ( ' image.resize ' , payload , { persistent : true , // ghi xuống disk, sống sót broker restart messageId : jobId , // để consumer dedupe contentType : ' application/json ' , }) // đợi broker confirm đ