标签：#systems

The Problem We Were Actually Solving Our event platform at Veltrix ran a treasure hunt game that gave users real-world rewards. It started as a simple Rails app with a PostgreSQL counter column for each hunt. By 3 AM on Black Friday, that counter column became a single point of failure. Every leaderboard update blocked the entire leaderboard query because PostgreSQL row-level locks escalated to table-level for SERIAL columns. Our error rate jumped from 0.2% to 18% under 2000 concurrent writes. The system didnt just slow down; it started failing writes with could not serialize access due to concurrent update deadlocks. We lost $47K in rewards payouts before we could scale up the database. What We Tried First (And Why It Failed) Our first fix was to shard the PostgreSQL counter by hunt ID, splitting the hot row into 1024 partitions. That reduced the lock contention, but introduced new problems. Each hunt now needed its own sequence, and our Rails code had to route writes to the correct shard. The shard routing introduced 400ms extra latency on leaderboard queries because we had to union results across 1024 tables. Meanwhile, PostgreSQL sequences had gaps up to 1024 when nodes restarted, so our reward payouts were off by thousands on high-traffic hunts. Our Redis cache didnt help because the leaderboard queries were point lookups against 1024 tables, and Redis couldnt pipeline those efficiently. The Architecture Decision We ripped out the PostgreSQL counter and replaced it with a Kafka Streams-based event sourcing system called HuntStream. Every hunt action (point earned, reward claimed) became an immutable event in a Kafka topic. We built a materialized view on top of RocksDB that consumed the topic and maintained the current leaderboard state in memory. The materialized view was partitioned by hunt ID, which meant leaderboard queries only hit one RocksDB partition per hunt. We used RocksDBs built-in caching to keep hot leaderboards in memory, and fall back to disk fo

Microsoft announced Azure Linux 4.0 and Azure Container Linux at Open Source Summit. Azure Linux 4.0 is a Fedora-based general-purpose server distribution for Azure VMs, the first time Microsoft has offered a supported Linux beyond container hosting. Azure Container Linux is an immutable container-optimized host built on Flatcar. By Steef-Jan Wiggers

n fan-out microservice architectures, slow-but-completing requests accumulate across services and drive p99 latency far higher than per-service metrics suggest. This article presents an adaptive hedging mechanism that uses DDSketch for real-time quantile estimation, windowed rotation to handle distribution drift, and a token-bucket budget to prevent load amplification. By Prathamesh Bhope

In part 1 , the single-document case was easy. In part 2 , two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through. So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines. 1. Network latency Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store. You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write. The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below. 2. Deadlock You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get. So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table: Transaction A locks key X, then needs Y. Transaction B locks Y, then needs X. Both block until timeout or intervention. The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time. So nobody builds it. Instead teams impose a standing discipline: always acquire locks

The Problem We Were Actually Solving During load testing at 50k concurrent hunters hitting the hunt endpoints, p99 latencies stayed under 200ms. But at 270k concurrent users in production, the hunt page suddenly took 1.8 seconds to load, triggering cascading 502s from our CDN. The error surfaced in Datadog as hunt_page_render_time_bucket{le=2.0} = 42% while le=0.5 dropped to 18%. The fingerprints were identical across three regions: high latency correlated exactly with Redis cache miss rate spiking from 12% to 48% during the hunt start window. Our cache-aside pattern with a 30-second TTL was amplifying miss storms. We discovered that the treasure hunt start time was synchronised by marketing campaigns. When the clock struck 10:00:00 UTC, 270k users hit the endpoint within 30 seconds. Each request would check the cache (miss), fetch from PostgreSQL, render the page, and write the cache entry. But PostgreSQL couldnt keep up with 9k queries per second during that window, causing query queueing and connection exhaustion. The Redis layer, designed for 150k ops/sec, was not the bottleneck. The database was. What We Tried First (And Why It Failed) Our first attempt was to increase Redis TTL from 30 seconds to 5 minutes. This reduced cache misses from 48% to 24%, and p99 latency improved to 650ms. But at 320k concurrent users, the latency still spiked to 1.4s because the underlying database queries were still hitting the same table with the same indexes. The Redis layer was masking symptoms, not solving the root cause. Next, we tried database read replicas. We spun up three read replicas and routed hunt queries to them using a weighted service mesh. This worked for a few minutes, but then we hit replication lag. The replicas fell 800ms behind primary, causing hunt pages to display stale treasure locations. Our operators started getting customer complaints about seeing the wrong treasure coordinates. We rolled back within 15 minutes. We even tried increasing PostgreSQL share

The Problem We Were Actually Solving Treasure hunts in Hytale arent just about generating loot. Theyre about generating simultaneous loot across thousands of players while keeping the world state consistent. We started with the assumption that events are stateless notifications: a hunt starts, we fire an event, clients react. That model worked fine when we had 200 concurrent players. At 2,000 players, the event bus turned into a 40 MB/s firehose of JSON blobs. Each loot drop required serializing the entire chunk state—blocks, entities, metadata—so clients could render the drop in real time. The JVMs G1GC couldnt handle the allocation rate. Every 47 minutes, a GC cycle would pause for 4.2 seconds, the chunk cache would fragment, and the server would hard crash with an OutOfMemoryError in net.minecraft.server.MinecraftServer#processQueue. The real problem wasnt the hunt logic. It was the architectural laziness of treating events as a catch-all glue layer instead of a boundary layer with explicit interfaces. What We Tried First (And Why It Failed) We tried Kafka as the event bus. The plan was to shard hunts by region and stream loot drops as compacted topics. The first run worked for about 6 hours before the compacted topics started to bloat. Each hunt was generating 700 KB of serialized chunk state per drop. At 30 drops per hunt per minute, thats 21 MB per hunt per minute. With 400 active hunts, the brokers couldnt keep up. The lag grew to 12 seconds, clients started rubber-banding, and we got a flood of Discord reports: You sank my boat! The event stream was now the bottleneck, not the event source. Next, we tried Redis Streams with a Lua script to aggregate loot drops per chunk. Within 30 minutes, we hit the 4 GB maxmemory limit because Lua scripts were stacking dropped items in memory while waiting for the next batch. The script was elegant—O(1) per drop—but the memory footprint made it unusable in production. Finally, we tried a sidecar service: a small Go process

The argument sounds reasonable: fewer lines of code mean fewer bugs. Simpler to review, easier to...

Why Manual Memory Management Still Matters ? It is now easier than ever to write software...