标签：#STEM

Google has shared details of its fleet wide large scale A/B experimentation system designed to standardize experiment assignment, exposure logging, and configuration propagation across distributed services. The approach enables consistent measurement across products, reduces experiment conflicts, and improves reliability of data driven decision making at scale. By Leela Kumili

There is a moment every embedded Linux developer hits eventually. You have spent days building something that works beautifully — a sensor pipeline, a streaming server, an MQTT client — and then you reboot the device and everything is silent. Nothing started. You SSH in, manually run your script, and it all comes back to life. The hardware is fine. Your code is fine. You just have no way of automatically running it. That is the gap systemd fills. It is the init system on virtually every modern Linux distribution, and on embedded Linux systems like the Raspberry Pi it is what decides what runs at boot, what gets restarted if it crashes, and where all the logs go. Once you understand how to write a service file, your applications stop being fragile scripts you need to babysit and start being first-class system services that survive reboots, network drops, and unexpected crashes. This tutorial builds up from the simplest possible service file to a production-ready configuration, explaining every line along the way. By the end you will have a service running your own Python application, logging to the system journal, and automatically restarting itself after failures. See Complete Tutorial in Github: Systemd Services Tutorial What systemd Actually Does Before writing any configuration, it helps to understand what problem systemd is solving, because the design of service files makes much more sense once you see the underlying model. When your Raspberry Pi boots, the Linux kernel starts and immediately hands control to process ID 1 — the very first user-space process. On modern systems, that process is systemd . Everything that happens next — mounting filesystems, bringing up the network, starting your application — is orchestrated by systemd. It reads configuration files called unit files that describe what should be started, when, in what order, and what to do if something goes wrong. A service file is just one type of unit file (there are also unit files for timers, so

Blockchain performance is determined by consensus rules. There is no acceleration layer within the protocol. One of the most common misconceptions about blockchain technology is the belief that transaction speed can be dramatically increased through special tools, hidden settings, or external services. While applications can improve user experience and optimize how information is presented, they cannot change the fundamental rules that govern how blockchain networks process transactions. At the core of every blockchain is a consensus mechanism. Consensus is responsible for ensuring that independent participants agree on the validity and order of transactions before they become part of the permanent ledger. Whether a network uses Proof of Work, Proof of Stake, or another consensus model, transaction processing remains tied to the protocol rules that all participants follow. Every transaction moves through a structured lifecycle: submit → validate → confirm Submission introduces the transaction to the network. Validation ensures that the transaction complies with protocol requirements and contains legitimate data. Confirmation establishes agreement across the network and records the transaction as part of the blockchain. These stages are not optional. They are essential to maintaining consistency and trust within decentralized systems. Because blockchain performance is governed by consensus, there is no protocol-level acceleration layer that can bypass validation or force immediate finality. No application can override consensus. No service can remove verification requirements. No external process can alter the execution sequence established by the protocol. What users often interpret as slow performance is usually the result of network conditions such as congestion, validator workload, transaction prioritization, or fee market activity. These factors can influence confirmation times, but they do not change the underlying rules of the system. Blockchain networks are d

There is a version of organizational life that feels, from inside, like maturity. Meetings reach conclusions. Decisions move forward without extended debate. The leadership team operates with visible coherence. Escalations are rare. When concerns are raised, they are quickly absorbed into the existing framework and resolved without disruption. Everything functions exactly as designed. That is what makes it difficult to notice that something essential has stopped occurring. The more stable the system appears, the more completely it has eliminated the conditions under which instability would be visible. The Epistemic Function of Conflict Conflict in organizations is not primarily a social problem. It is an epistemic mechanism. When a decision is challenged, something precise occurs: the decision's internal logic is forced into the open. Its assumptions are made explicit. Its evidence is tested against contrary interpretation. The challenge does not guarantee a better outcome — but it generates information the unchallenged decision never produces. Conflict is not disruption of the system. It is how the system verifies itself against reality. Remove the disagreement, and the system continues deciding. It simply stops testing whether its decisions are sound. The absence of challenge feels like confidence. It is blindness — a blindness that is, from inside, indistinguishable from clarity. What Fills the Vacuum When legitimate conflict disappears, the space does not remain empty. It fills with the performance of conflict. Meetings still contain discussion. Questions are still asked. Concerns are occasionally raised. But the texture has changed in ways that experienced practitioners feel before they can articulate. Questions are asked to signal engagement rather than to probe assumptions. Concerns are framed to demonstrate awareness rather than to force resolution. Debate occurs within the boundaries of what the system has already decided is acceptable to debate. The ritual

Windborne Systems' newest weather forecasting model beats the best government predictions by days.

Shopify introduced GraphQL Cardinal, a new execution engine replacing depth-first traversal with breadth-first execution. The redesign improves large-scale GraphQL performance with up to 15x faster field execution, 6x lower GC overhead, and +4s P50 latency gains. It focuses on execution-layer efficiency and batched resolver processing for high-cardinality commerce queries. By Leela Kumili

The Theorem That Changed How We Think About Databases In 2000, Eric Brewer stood at a conference and proposed a conjecture that would reshape distributed systems forever: "You can only guarantee two of these three properties at the same time: Consistency, Availability, and Partition Tolerance." Two years later, Seth Gilbert and Nancy Lynch proved it mathematically. It became known as the CAP Theorem — and every distributed system architect since has had to wrestle with it. It sounds abstract. But once you understand it, you'll never look at a database choice the same way again. You'll understand why Amazon DynamoDB and Google Spanner make opposite architectural choices. You'll know why your bank uses PostgreSQL while Twitter uses Cassandra. Let's break it down from first principles. The Three Properties C — Consistency Every read receives the most recent write, or an error. There's only one version of the truth — all nodes agree. Not the same consistency as ACID . CAP consistency (linearizability) means every read reflects the latest write across all nodes. ACID consistency means transactions don't violate database constraints. Different concepts, same confusing word. A — Availability Every request receives a non-error response — though it might not be the most recent data. The system is always up and answering. Note: "Available" in CAP doesn't mean "fast." It means "responds without error." A system that always returns a (possibly stale) answer is Available. P — Partition Tolerance The system continues operating even when network messages between nodes are lost or delayed. A partition is when part of your distributed system can't communicate with another part. The Unavoidable Truth: P Is Not Optional Here's the insight that makes CAP actually useful: In any real distributed system, partitions will happen. Networks fail. Cables get cut. Data centers lose connectivity. AWS regions go down. Since you must tolerate partitions (or have a single-server system, which does

Event sourcing gets sold on immutability. You don't update, you don't delete, you only append, so the history is permanent. It mostly isn't. The events are immutable because your code agrees not to touch them, not because anything actually stops it. Underneath they're still rows in Postgres, and rows have a DBA with write access. A migration that "cleans up" old data. A 2 a.m. query run against the wrong connection. A backup restored with slightly different bytes in it. Change one of those rows and a replay won't blink. The aggregate rebuilds, the projections rebuild, everything looks fine. Usually the first person to notice is a customer whose balance is off, and by then the trail is cold. Chain each event into the next The trick is small. Give every row two extra columns: a hash of its contents, and the hash of the row before it. #1 AccountOpened prev=00000… hash=70be4f… │ ▼ #2 AmountDeposited prev=70be4f… hash=796018… │ ▼ #3 AmountWithdrawn prev=796018… hash=6a0260… The hash is SHA-256(previousHash || json(payload)) . Nothing exotic. The point is that each hash depends on the one before it. Edit a payload and its hash stops matching. Rewrite that hash to cover for the edit, and now the next row's pointer is wrong. You can't fix one without breaking the next. About forty lines of it Appending an event hashes it together with the previous one: public HashChainedEntry Append ( object payload ) { var previousHash = _entries . Count == 0 ? GenesisHash : _entries [^ 1 ]. Hash ; var hash = ComputeHash ( previousHash , payload ); var entry = new HashChainedEntry ( _entries . Count + 1 , payload , previousHash , hash ); _entries . Add ( entry ); return entry ; } internal static byte [] ComputeHash ( byte [] previousHash , object payload ) { var payloadJson = JsonSerializer . SerializeToUtf8Bytes ( payload , payload . GetType ()); var combined = new byte [ previousHash . Length + payloadJson . Length ]; Buffer . BlockCopy ( previousHash , 0 , combined , 0 , previousHash .

In November 2023 we ran our first global Hytale servers on Google Kubernetes Engine using Veltrix 3.2 as our configuration orchestrator. The Treasure Hunt Engine—a service that fans spawn to claim event loot—started crashing every time search volume exceeded 12 k RPM. Grafana showed a steady climb of 503 errors on /hunt/claim until the autoscaler maxed out at 32 G1 CPU cores and still couldnt keep up. Operators kept filing tickets that boiled down to one sentence: We click the map, nothing happens. We never saw the actual error because the ingress controller was swallowing it and returning a generic Too many requests. What we tried first (and why it failed) Our first move was to crank up the nginx-ingress-controller replicas from 3 to 12 and switch the load-balancer tier from GKE Standard to Premium. The 503 rate dropped to 8 k RPM, but now the p99 latency on claims spiked from 80 ms to 420 ms. The culprit was a recursive call in the hunt service: every claim required a round trip to the player-profile service to validate tier eligibility, and that service was on a shared Postgres 15.4 cluster with 3 k TPS of unrelated traffic. The error stack in Jaeger was literally tracing_id=7f3a1c8… server=profile-db pool_timeout . We tried adding connection pooling with PgBouncer, but the hunt service was using raw libpq and refused to reuse connections—no matter how many times we told it. The Architecture Decision We ripped the validation out of the synchronous path and made the hunt engine publish an event called HuntTierCheckRequired to a dedicated Kafka topic player-events-tier . The hunt service would respond to the client with a 202 Accepted immediately, then the loot-claim worker would listen to that topic and, if the tier passed, publish HuntLootReady . The worker ran in the same pod but on a separate goroutine with a 60-second TTL so we didnt leak memory if the tier service hung. We moved the player-profile service to an SSD-backed CloudSQL instance and gave it 32 GB R

I lost more than 60 days but I made a huge update All OnemanBSD OS is now 64 bits. I also removed all bloat like Rust, Wayland and Qt dependencies from all app ports used in the system. (mpv was changed to gnome-mplayer because of that). New apps included (+ fixed source code of everything): audacity (audio editor) godot (game editor) deadbeef (mp3 player) ted (word processor) Also some bug fixes. Here is the link again: http://bialamusic.com/onemanBSD/ Here are my thoughts: I feel that we as humans are loosing control over technology that was created by humans. This is rising some serious and strange questions. I think we now need tools to power the individual so we can fight back. Not corporations. Not governments. Not organizations. Consider this my bullet in this battle. It is now in your hands so you can be active developers not just passive users.

E-food delivery is a trillion-dollar market . And most of that trillion is not going to farmers, store owners, or the people who actually move food around. It's going to the infrastructure layer sitting between them — the platform tax, the per-order cut, the SaaS subscription that charges you to exist inside someone else's garden. The walled garden isn't accidental. It's the product. What if food delivery was a protocol, not a platform? Not an app. Not a marketplace. A protocol — like HTTP, like SMTP — that any node can speak, that no single company owns, and that costs near zero to run. That's what DIFP is. The Djowda Interconnected Food Protocol. An open wire format for connecting food ecosystem participants — farms, stores, restaurants, wholesalers, delivery nodes, end users — directly to each other, without a platform in the middle extracting rent at every step. The spec covers: Presence & discovery — participants announce themselves to their spatial cell, others find them by location Orders, asks, and donations — not just commerce, but demand signals and surplus distribution in the same protocol Spatial routing — the MinMax99 grid maps the entire planet into ~500m cells; every message knows where it's going Decentralized registry — nodes find each other through a federated lobby system, no central server required Version 0.4 of the spec dropped a few weeks ago. Today we're publishing the first working implementation. The gRPC preview — what we built DIFP-gRPC is a skull implementation of the full protocol stack over gRPC. Thin, end-to-end, every domain wired — nothing production-hardened yet, everything clearly marked for what it is. What's inside difp.proto — the entire DIFP v0.4 spec as a single protobuf file. Two services, 30+ message types, the full DifpEnvelope wrapper with a oneof payload that covers every domain: message DifpEnvelope { string id = 1 ; string type = 2 ; // "trade.ask" | "presence.announce" | "node.ping" | … string version = 3 ; MessageSen

Java LLD: Designing a Thread-Safe Parking Lot with Strategy Pattern Designing a parking lot is a staple of Java LLD and machine coding interviews, yet most candidates fail to write production-grade code. As an ex-FAANG interviewer, I've seen countless designs fall apart under concurrent traffic or when asked to support multiple slot allocation algorithms. If you're prepping for interviews, I've been building javalld.com — real machine coding problems with full execution traces. The Mistake Most Candidates Make Monolithic locking on the entire ParkingLot class: Using a global synchronized keyword on the entry method, which serializes all gate entries and destroys system throughput. Hardcoding slot-finding logic: Mixing spatial layout algorithms (like nearest-to-entrance or smallest-available-fit) directly inside the ParkingLot or Gate classes, violating the Open-Closed Principle. Thread-safety as an afterthought: Relying on raw List<Slot> iterations without synchronization, causing race conditions where multiple cars are assigned to the exact same physical slot. The Right Approach Core mental model: Decouple capacity management from slot selection by using a Semaphore for gate-keeping and the Strategy Pattern for thread-safe slot allocation. Key entities: ParkingLot , Gate , Slot , Vehicle , ParkingStrategy ( SmallestFitStrategy , NearestEntranceStrategy ), and StrategyFactory . Why it beats the naive approach: It isolates concurrency concerns (preventing overbooking) from business rules (how we choose a slot), making the system highly performant and easily extensible. The Key Insight (Code) public class EntryGate { private final Semaphore semaphore ; private final ParkingStrategy strategy ; public EntryGate ( int capacity , ParkingStrategy strategy ) { this . semaphore = new Semaphore ( capacity ); this . strategy = strategy ; } public synchronized Ticket park ( Vehicle vehicle ) { if (! semaphore . tryAcquire ()) throw new ParkingFullException (); Slot slot = strat

Most software projects focus on producing outputs. Most AI projects focus on producing predictions. But real organizations don't operate on outputs or predictions alone. They operate on decisions. A decision has consequences. A decision creates risk. A decision consumes resources. A decision changes the future state of a system. Over the last few months, I've been studying and building systems around a simple question: How can we make decisions more explainable, auditable, and repeatable? This led me toward concepts such as: event-driven architectures decision logging risk evaluation pipelines audit trails feedback loops operational intelligence systems Instead of asking: "Can we predict what will happen?" I'm becoming more interested in asking: "Can we explain why a decision was made?" and "Can we reproduce that decision six months later?" Current areas I'm exploring: Financial decision systems Risk infrastructure Event-driven architectures Blockchain compliance workflows Operational intelligence platforms One of the projects I'm currently building is an Event-Driven Decision Logging System (EDDL), designed to explore how organizations can record, audit, and replay critical decisions over time. Still learning. Still building. Still refining my understanding of how complex systems operate under uncertainty. Looking forward to sharing the journey here. systemsdesign #architecture #backend #fintech #softwareengineering #eventdriven #riskmanagement

The Problem We Were Actually Solving Our event platform at Veltrix ran a treasure hunt game that gave users real-world rewards. It started as a simple Rails app with a PostgreSQL counter column for each hunt. By 3 AM on Black Friday, that counter column became a single point of failure. Every leaderboard update blocked the entire leaderboard query because PostgreSQL row-level locks escalated to table-level for SERIAL columns. Our error rate jumped from 0.2% to 18% under 2000 concurrent writes. The system didnt just slow down; it started failing writes with could not serialize access due to concurrent update deadlocks. We lost $47K in rewards payouts before we could scale up the database. What We Tried First (And Why It Failed) Our first fix was to shard the PostgreSQL counter by hunt ID, splitting the hot row into 1024 partitions. That reduced the lock contention, but introduced new problems. Each hunt now needed its own sequence, and our Rails code had to route writes to the correct shard. The shard routing introduced 400ms extra latency on leaderboard queries because we had to union results across 1024 tables. Meanwhile, PostgreSQL sequences had gaps up to 1024 when nodes restarted, so our reward payouts were off by thousands on high-traffic hunts. Our Redis cache didnt help because the leaderboard queries were point lookups against 1024 tables, and Redis couldnt pipeline those efficiently. The Architecture Decision We ripped out the PostgreSQL counter and replaced it with a Kafka Streams-based event sourcing system called HuntStream. Every hunt action (point earned, reward claimed) became an immutable event in a Kafka topic. We built a materialized view on top of RocksDB that consumed the topic and maintained the current leaderboard state in memory. The materialized view was partitioned by hunt ID, which meant leaderboard queries only hit one RocksDB partition per hunt. We used RocksDBs built-in caching to keep hot leaderboards in memory, and fall back to disk fo

Microsoft announced Azure Linux 4.0 and Azure Container Linux at Open Source Summit. Azure Linux 4.0 is a Fedora-based general-purpose server distribution for Azure VMs, the first time Microsoft has offered a supported Linux beyond container hosting. Azure Container Linux is an immutable container-optimized host built on Flatcar. By Steef-Jan Wiggers

n fan-out microservice architectures, slow-but-completing requests accumulate across services and drive p99 latency far higher than per-service metrics suggest. This article presents an adaptive hedging mechanism that uses DDSketch for real-time quantile estimation, windowed rotation to handle distribution drift, and a token-bucket budget to prevent load amplification. By Prathamesh Bhope

In part 1 , the single-document case was easy. In part 2 , two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through. So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines. 1. Network latency Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store. You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write. The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below. 2. Deadlock You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get. So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table: Transaction A locks key X, then needs Y. Transaction B locks Y, then needs X. Both block until timeout or intervention. The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time. So nobody builds it. Instead teams impose a standing discipline: always acquire locks

The Problem We Were Actually Solving During load testing at 50k concurrent hunters hitting the hunt endpoints, p99 latencies stayed under 200ms. But at 270k concurrent users in production, the hunt page suddenly took 1.8 seconds to load, triggering cascading 502s from our CDN. The error surfaced in Datadog as hunt_page_render_time_bucket{le=2.0} = 42% while le=0.5 dropped to 18%. The fingerprints were identical across three regions: high latency correlated exactly with Redis cache miss rate spiking from 12% to 48% during the hunt start window. Our cache-aside pattern with a 30-second TTL was amplifying miss storms. We discovered that the treasure hunt start time was synchronised by marketing campaigns. When the clock struck 10:00:00 UTC, 270k users hit the endpoint within 30 seconds. Each request would check the cache (miss), fetch from PostgreSQL, render the page, and write the cache entry. But PostgreSQL couldnt keep up with 9k queries per second during that window, causing query queueing and connection exhaustion. The Redis layer, designed for 150k ops/sec, was not the bottleneck. The database was. What We Tried First (And Why It Failed) Our first attempt was to increase Redis TTL from 30 seconds to 5 minutes. This reduced cache misses from 48% to 24%, and p99 latency improved to 650ms. But at 320k concurrent users, the latency still spiked to 1.4s because the underlying database queries were still hitting the same table with the same indexes. The Redis layer was masking symptoms, not solving the root cause. Next, we tried database read replicas. We spun up three read replicas and routed hunt queries to them using a weighted service mesh. This worked for a few minutes, but then we hit replication lag. The replicas fell 800ms behind primary, causing hunt pages to display stale treasure locations. Our operators started getting customer complaints about seeing the wrong treasure coordinates. We rolled back within 15 minutes. We even tried increasing PostgreSQL share

The Problem We Were Actually Solving Treasure hunts in Hytale arent just about generating loot. Theyre about generating simultaneous loot across thousands of players while keeping the world state consistent. We started with the assumption that events are stateless notifications: a hunt starts, we fire an event, clients react. That model worked fine when we had 200 concurrent players. At 2,000 players, the event bus turned into a 40 MB/s firehose of JSON blobs. Each loot drop required serializing the entire chunk state—blocks, entities, metadata—so clients could render the drop in real time. The JVMs G1GC couldnt handle the allocation rate. Every 47 minutes, a GC cycle would pause for 4.2 seconds, the chunk cache would fragment, and the server would hard crash with an OutOfMemoryError in net.minecraft.server.MinecraftServer#processQueue. The real problem wasnt the hunt logic. It was the architectural laziness of treating events as a catch-all glue layer instead of a boundary layer with explicit interfaces. What We Tried First (And Why It Failed) We tried Kafka as the event bus. The plan was to shard hunts by region and stream loot drops as compacted topics. The first run worked for about 6 hours before the compacted topics started to bloat. Each hunt was generating 700 KB of serialized chunk state per drop. At 30 drops per hunt per minute, thats 21 MB per hunt per minute. With 400 active hunts, the brokers couldnt keep up. The lag grew to 12 seconds, clients started rubber-banding, and we got a flood of Discord reports: You sank my boat! The event stream was now the bottleneck, not the event source. Next, we tried Redis Streams with a Lua script to aggregate loot drops per chunk. Within 30 minutes, we hit the 4 GB maxmemory limit because Lua scripts were stacking dropped items in memory while waiting for the next batch. The script was elegant—O(1) per drop—but the memory footprint made it unusable in production. Finally, we tried a sidecar service: a small Go process

The argument sounds reasonable: fewer lines of code mean fewer bugs. Simpler to review, easier to...