标签：#systems

Earlier parts examined the performance characteristics of sequential and random access under single-threaded execution, and noted in passing the destructive effect of random access on the TLB. This part devotes full attention to the TLB: what it is, why a TLB miss is more severe than a cache miss, why a page table walk constitutes one of the longest dependency chains a CPU can encounter, how huge pages fundamentally alter TLB reach, and where memory-level parallelism falters in the face of TLB misses. Page Boundaries: Where the Prefetcher Halts Part III, in its discussion of prefetchers, noted a hard constraint: a prefetcher must not cross page boundaries on its own authority. The operating system manages virtual memory in units of pages (typically 4 KB, i.e., 64 cache lines). When a program reaches the end of one page and is about to step into the next, the prefetcher cannot proceed. The reason is that the next page may not reside in physical memory (it may have been swapped out to disk), or it may be an entirely invalid virtual address — if the prefetcher were to speculatively initiate an access to the next page, it would trigger a page fault: the OS would have to suspend the process and swap the page in from disk; in the case of an invalid address, the OS would terminate the process outright. From a security standpoint, the prefetcher neither can nor is permitted to autonomously cross page boundaries without TLB approval. Hence a performance brake appears every 4 KB — even when traversing an array sequentially, after every 64 cache line accesses the prefetch pipeline must pause and await confirmation of an address translation. This is not to say that modern CPU prefetchers are completely unable to cross pages. Intel's Next Page Prefetcher and AMD's equivalent mechanism can consult the TLB when approaching a page boundary — if the address mapping for the next page is already registered in the TLB, the prefetcher receives clearance to continue prefetching across th

Introduction Modern cloud-native systems generate an enormous amount of telemetry data every second. Applications, containers, Kubernetes clusters, APIs, databases, and infrastructure components continuously emit metrics, logs, and traces to help engineering teams understand system behavior and troubleshoot issues. While observability has become essential for operating distributed systems reliably, it has also introduced a new challenge: managing the scale, cost, and quality of telemetry. OpenTelemetry (OTel) has emerged as the industry standard for collecting and processing observability data. It provides a vendor-neutral framework for instrumenting applications and exporting telemetry to different observability backends. However, simply adopting OpenTelemetry is not enough. Without proper optimization strategies, organizations often face excessive telemetry ingestion costs, noisy dashboards, high-cardinality metrics, trace overload, and inefficient debugging workflows. This article explores practical approaches for optimizing observability using OpenTelemetry. It focuses on metrics, logs, and traces individually while also discussing broader optimization strategies across the telemetry pipeline. Understanding the OpenTelemetry observability pipeline OpenTelemetry provides a unified framework for generating, collecting, processing, and exporting telemetry data. At its core, the OTel ecosystem consists of SDKs, instrumentation libraries, collectors, processors, and exporters. Applications generate telemetry using OpenTelemetry SDKs or auto-instrumentation agents. This telemetry is then sent to the OpenTelemetry Collector, which acts as a centralized telemetry processing layer. The collector can receive telemetry from multiple sources, enrich it with metadata, apply filtering or sampling, and export it to one or more observability backends. The observability pipeline typically follows this flow: Application → OTel SDK → OTel Collector → Observability Backend The Open

Have you ever wondered how Discord keeps your channel messages available even when a server goes down? Or how Amazon DynamoDB serves petabytes of data with single-digit millisecond latency? The unsung hero powering almost all these distributed systems is hashing — a simple but powerful technique that makes even load distribution, fast lookups, and seamless scaling possible. As more applications move to distributed cloud architectures, understanding hashing for distributed systems is no longer optional for developers. Choosing the wrong hashing algorithm can lead to cascading failures, cache stampedes, and expensive downtime. This guide breaks down every core hashing technique, real-world use cases, best practices, and common pitfalls to avoid in 2026. Table of Contents What is Hashing in Distributed Systems? Core Hashing Algorithms Explained Traditional Modulo Hashing Consistent Hashing Virtual Nodes (VNodes) Rendezvous Hashing (HRW) Jump Consistent Hash Maglev Hashing Multi-Probe Consistent Hashing Consistent Hashing with Bounded Loads Real-World Applications of Distributed Hashing Head-to-Head Algorithm Comparison Best Practices for Distributed Hashing Common Pitfalls to Avoid Conclusion References What is Hashing in Distributed Systems? Hashing in distributed systems is the practice of mapping data keys (e.g., user IDs, object keys, channel IDs) to server nodes using a deterministic hash function. The core goals are: Distribute load evenly across all nodes to avoid hotspots Enable fast lookups (O(1) or O(log N)) without a central coordinator Minimize data movement when nodes are added or removed during scaling Support fault tolerance by simplifying replication across nodes The simplest implementation is modulo-based hashing , where node_id = hash(key) % N and N is the total number of nodes. While trivial to implement, it suffers from a fatal flaw: the rehashing problem. When N changes (a node is added or removed), nearly all keys are remapped to new nodes, causin

Pinterest introduced MIQPS, a URL normalization system that identifies which query parameters affect page identity using rendered content fingerprints. It reduces duplicate processing across millions of domains by replacing rule-based approaches with offline analysis, anomaly detection, and runtime parameter maps, improving ingestion efficiency and scalability in large-scale content pipelines. By Leela Kumili

Somewhere along the way, computing became disposable. Devices became sealed. Systems became rented. Ownership became licensing. Repairability disappeared. Infrastructure moved away from the user and into distant cloud platforms. And I think we lost something important because of it. Lately, I’ve found myself becoming increasingly interested in hardware again. Not just software. Not just cloud systems. But actual computing devices. Servers. Home infrastructure. Repairable machines. Set-top systems. Local AI appliances. Sovereign computing. Because I believe the next era of computing will belong to people who own their infrastructure again. The Provider Box Realization One thing that kept sticking in my head was this: Almost every home in America already has a provider box. A Comcast box. An AT&T gateway. A router. A modem. A streaming box. People are already comfortable with the idea of a dedicated computing appliance sitting in their home quietly powering their digital life. That realization changed how I thought about computing infrastructure. What if those boxes worked for the user instead of the provider? What if they: hosted local AI, managed home storage, coordinated smart devices, powered media systems, handled automation, protected privacy, synchronized intelligently, and operated as sovereign infrastructure? That idea became part of the thinking behind KiwiHome. The Return Of Home Infrastructure For a long time, the industry moved toward centralization. Everything shifted toward: SaaS, subscriptions, streaming, cloud storage, cloud intelligence, and rented operational environments. Convenient? Absolutely. But also fragile. If: pricing changes, services disappear, companies shut down, APIs get revoked, or platforms change policies, entire workflows collapse overnight. I think people are starting to feel that tension. Especially creators. Especially businesses. Especially technical users. That’s why I believe we’re going to see a major resurgence in: home serv

Imagine you are building a cloud-native backend for a high-frequency trading platform or a core banking ledger. To ensure mathematical immutability and prevent silent data tampering, compliance mandates that every transaction for a specific financial account must be cryptographically chained. This means the signature of Transaction #50 must explicitly include the cryptographic hash of Transaction #49. You cannot sign them out of order, and the backend is strictly responsible for generating and validating this chain. This introduces a massive distributed systems headache: How do you enforce strict, sequential ordering while maintaining the concurrency required to scale a modern cloud architecture? Let's walk through the evolution of this system, deconstruct exactly how the standard event-driven approach fails in production, and examine the Staff-level architecture required to fix it. Phase 1: The MVP & The Database Bottleneck In the early days, traffic is low. An account might see one transaction every few minutes. The "Happy Path" is simple: The API receives a deposit request for Account A. The API queries Postgres for the last_signature_hash . The API computes the new hash in memory: SHA(last_hash + new_transaction_data) . The API writes the new transaction and updates the state. The Pitfall: The Thundering Herd To prevent two concurrent requests from reading the same previous hash, you wrap the database operation in a pessimistic lock: SELECT ... FOR UPDATE . This forces the database to serialize requests at the row level. When a massive partner bank initiates a bulk sync, dumping 5,000 transactions for a single corporate account onto the API in two seconds, 4,999 concurrent threads immediately hit the FOR UPDATE lock and block. The database connection pool is instantly exhausted, latency spikes platform-wide, and the MVP dies. The Insight: The database must be your last line of defense, not your primary queueing mechanism. Contention must be solved upstream in me

Table of Contents The Integration Problem That Broke Industry 4.0 MCP: The Vertical Connection Layer How MCP Connects to Servers, Tools, and Databases MCP in Real World Industrial Automation ACP: The Horizontal Communication Layer How ACP Works Under the Hood ACP in Real World Industrial Coordination The Six Precise Differences How They Work Together: The Complete Stack Decision Framework for Industrial AI Architects 1. The Integration Problem That Broke Industry 4.0 Industry 4.0 promised connected factories, intelligent automation, and seamless data flow between machines, systems, and humans. The technology arrived. The connectivity did not. The reason is a number called N times M. An enterprise manufacturing facility might have 12 AI agents across quality, maintenance, and planning — and 28 data sources including ERP, MES, SCADA, IoT sensors, databases, CAD repositories, and supplier APIs. Without a standard protocol: 12 agents multiplied by 28 data sources equals 336 custom integrations. Each integration is bespoke code. Each breaks when either side updates. Each requires maintenance. Each represents a point of failure and a security surface that must be independently managed. IBM VP Armand Ruiz stated this precisely: "Without a common standard, every integration is costly duct tape." MCP and ACP together replace 336 pieces of duct tape with two standard protocols — one governing how agents connect to systems, one governing how agents connect to each other. The smart manufacturing market is projected to reach 374 billion dollars by 2025 at 11.8 percent CAGR. Over 50 percent of companies in industrial automation are expected to adopt MCP-based connectivity. The integration problem is not theoretical. The solution is being deployed at scale right now. 2. MCP: The Vertical Connection Layer MCP connects agents to tools and data — the vertical integration layer. It handles the connection between an AI agent and everything it needs to interact with in the external worl

Hello Dev.to Hello everyone, This is my very first post here at Dev.to. I am curious about how companies go about running their business in the backstage and, specifically, what kind of challenges do they face in their inventory management, warehousing, workplace technologies, and overall business systems. There have been plenty of times that I witnessed how little operational inefficiencies turn into big business problems when left untreated for too long. I intend to share my observations and learnings regarding the topics of inventory management, asset management, workflow improvement, and business systems used by companies to keep themselves organized and efficient. Please note that I am not here to pretend having all the right answers. It's simply an attempt at learning and exchanging ideas within the industry. See you around!

Most engineers build distributed systems hoping nothing breaks. Amazon S3 was engineered under the opposite assumption: that something is already broken, right now, and the system needs to be fine with that. That one mindset shift explains almost everything about how S3 works — and why it's one of the most reliable pieces of infrastructure on the planet. I went through a deep-dive conversation with Mai-Lan Tomsen Bukovec, VP of Data and Analytics at AWS, and extracted the engineering philosophy underneath the product. Not the marketing version. The real one. Here's what actually matters. 1. Hardware failure is not an emergency. It's Tuesday. S3 manages hundreds of exabytes of data across tens of millions of hard drives, spread across 120 Availability Zones in 38 AWS Regions. It currently stores over 500 trillion objects. At that scale, something is always failing. A disk here. A rack there. An availability zone every now and then. The math is unforgiving. So the S3 team made a deliberate architectural decision early: stop treating failure as an exception. Design it into the system as the baseline state. This means dedicated auditor and repair microservices run continuously in the background — not when something goes wrong, but always. They scan the entire fleet, inspect every byte of data, detect discrepancies, and trigger repairs automatically. No human in the loop. No incident ticket. No war room. There's also a specific property they engineer for called crash consistency — the system is designed so that after any fail-stop event, it automatically returns to a valid state without manual intervention. The failure happens. The system continues. Those two things are not in conflict. The system heals itself because it was designed to assume it's already sick. If you're building distributed systems and your failure handling is reactive — you only respond after something breaks — you've already lost. Design the repair loop as a first-class citizen, not an afterthought.

Uber introduced a high-throughput financial ledger processing system designed to handle hot account write contention at scale. Using 250ms batching, Redis coordination, and optimistic atomic updates, the system supports 30+ updates per second per account while preserving consistency and auditability, reducing multi-hour processing pipelines to minutes in its distributed accounting infrastructure. By Leela Kumili

The chatbot is tricking me!!! 💬📜⌛ When you text a chatbot, it doesn’t actually remember who you are or what you said two minutes ago. The exact millisecond it finishes typing a response, its brain completely wipes clean. To pull off the illusion of a continuous, flowing conversation, the web application secretly copy-pastes the entire past chat history, bundles it up, and blasts that whole massive block of text back into the processor every single time you hit send. Your "chat session" is an illusion maintained entirely by an ever-growing stateless prompt wrapper. You aren't interacting with a growing, adapting mind; you are repeatedly gas-lighting a brand-new entity into believing it has been talking to you for an hour. Wait, I am the one training it ??? 🚦🚸🚲 AI models are inherently blind to context; a computer doesn't instinctively know that a specific cluster of raw pixel values represents a real-world object. It requires billions of examples to be manually labeled by a human mind before the math can understand it. Every time you click on squares containing "traffic lights," "crosswalks," or "bicycles" to unlock a website, you are acting as an unpaid data annotator. You are manually labeling complex, messy real-world data points that feed directly into the computer vision systems of autonomous vehicles. The grand paradox of modern cyber security is that we force humans to act like mechanical data annotators to prove they are not computers, all so that computers can learn how to perfectly impersonate humans. The supercomputer is stupider than a toddler... 🍓👶🏻🖥️ We assume AI read letters and words the same way human eyes scan a page. It doesn't—it is entirely alphabet-blind. Before text hits the AI's brain, a parser chops strings of text into numerical blocks called "tokens." For example, the word "strawberry" isn't seen by the model as ten distinct letters; it is compressed into numerical IDs representing chunked pieces like "straw" and "berry". Because it never s

Google has shared details of its fleet wide large scale A/B experimentation system designed to standardize experiment assignment, exposure logging, and configuration propagation across distributed services. The approach enables consistent measurement across products, reduces experiment conflicts, and improves reliability of data driven decision making at scale. By Leela Kumili

Blockchain performance is determined by consensus rules. There is no acceleration layer within the protocol. One of the most common misconceptions about blockchain technology is the belief that transaction speed can be dramatically increased through special tools, hidden settings, or external services. While applications can improve user experience and optimize how information is presented, they cannot change the fundamental rules that govern how blockchain networks process transactions. At the core of every blockchain is a consensus mechanism. Consensus is responsible for ensuring that independent participants agree on the validity and order of transactions before they become part of the permanent ledger. Whether a network uses Proof of Work, Proof of Stake, or another consensus model, transaction processing remains tied to the protocol rules that all participants follow. Every transaction moves through a structured lifecycle: submit → validate → confirm Submission introduces the transaction to the network. Validation ensures that the transaction complies with protocol requirements and contains legitimate data. Confirmation establishes agreement across the network and records the transaction as part of the blockchain. These stages are not optional. They are essential to maintaining consistency and trust within decentralized systems. Because blockchain performance is governed by consensus, there is no protocol-level acceleration layer that can bypass validation or force immediate finality. No application can override consensus. No service can remove verification requirements. No external process can alter the execution sequence established by the protocol. What users often interpret as slow performance is usually the result of network conditions such as congestion, validator workload, transaction prioritization, or fee market activity. These factors can influence confirmation times, but they do not change the underlying rules of the system. Blockchain networks are d

There is a version of organizational life that feels, from inside, like maturity. Meetings reach conclusions. Decisions move forward without extended debate. The leadership team operates with visible coherence. Escalations are rare. When concerns are raised, they are quickly absorbed into the existing framework and resolved without disruption. Everything functions exactly as designed. That is what makes it difficult to notice that something essential has stopped occurring. The more stable the system appears, the more completely it has eliminated the conditions under which instability would be visible. The Epistemic Function of Conflict Conflict in organizations is not primarily a social problem. It is an epistemic mechanism. When a decision is challenged, something precise occurs: the decision's internal logic is forced into the open. Its assumptions are made explicit. Its evidence is tested against contrary interpretation. The challenge does not guarantee a better outcome — but it generates information the unchallenged decision never produces. Conflict is not disruption of the system. It is how the system verifies itself against reality. Remove the disagreement, and the system continues deciding. It simply stops testing whether its decisions are sound. The absence of challenge feels like confidence. It is blindness — a blindness that is, from inside, indistinguishable from clarity. What Fills the Vacuum When legitimate conflict disappears, the space does not remain empty. It fills with the performance of conflict. Meetings still contain discussion. Questions are still asked. Concerns are occasionally raised. But the texture has changed in ways that experienced practitioners feel before they can articulate. Questions are asked to signal engagement rather than to probe assumptions. Concerns are framed to demonstrate awareness rather than to force resolution. Debate occurs within the boundaries of what the system has already decided is acceptable to debate. The ritual

Windborne Systems' newest weather forecasting model beats the best government predictions by days.

Shopify introduced GraphQL Cardinal, a new execution engine replacing depth-first traversal with breadth-first execution. The redesign improves large-scale GraphQL performance with up to 15x faster field execution, 6x lower GC overhead, and +4s P50 latency gains. It focuses on execution-layer efficiency and batched resolver processing for high-cardinality commerce queries. By Leela Kumili

The Theorem That Changed How We Think About Databases In 2000, Eric Brewer stood at a conference and proposed a conjecture that would reshape distributed systems forever: "You can only guarantee two of these three properties at the same time: Consistency, Availability, and Partition Tolerance." Two years later, Seth Gilbert and Nancy Lynch proved it mathematically. It became known as the CAP Theorem — and every distributed system architect since has had to wrestle with it. It sounds abstract. But once you understand it, you'll never look at a database choice the same way again. You'll understand why Amazon DynamoDB and Google Spanner make opposite architectural choices. You'll know why your bank uses PostgreSQL while Twitter uses Cassandra. Let's break it down from first principles. The Three Properties C — Consistency Every read receives the most recent write, or an error. There's only one version of the truth — all nodes agree. Not the same consistency as ACID . CAP consistency (linearizability) means every read reflects the latest write across all nodes. ACID consistency means transactions don't violate database constraints. Different concepts, same confusing word. A — Availability Every request receives a non-error response — though it might not be the most recent data. The system is always up and answering. Note: "Available" in CAP doesn't mean "fast." It means "responds without error." A system that always returns a (possibly stale) answer is Available. P — Partition Tolerance The system continues operating even when network messages between nodes are lost or delayed. A partition is when part of your distributed system can't communicate with another part. The Unavoidable Truth: P Is Not Optional Here's the insight that makes CAP actually useful: In any real distributed system, partitions will happen. Networks fail. Cables get cut. Data centers lose connectivity. AWS regions go down. Since you must tolerate partitions (or have a single-server system, which does

In November 2023 we ran our first global Hytale servers on Google Kubernetes Engine using Veltrix 3.2 as our configuration orchestrator. The Treasure Hunt Engine—a service that fans spawn to claim event loot—started crashing every time search volume exceeded 12 k RPM. Grafana showed a steady climb of 503 errors on /hunt/claim until the autoscaler maxed out at 32 G1 CPU cores and still couldnt keep up. Operators kept filing tickets that boiled down to one sentence: We click the map, nothing happens. We never saw the actual error because the ingress controller was swallowing it and returning a generic Too many requests. What we tried first (and why it failed) Our first move was to crank up the nginx-ingress-controller replicas from 3 to 12 and switch the load-balancer tier from GKE Standard to Premium. The 503 rate dropped to 8 k RPM, but now the p99 latency on claims spiked from 80 ms to 420 ms. The culprit was a recursive call in the hunt service: every claim required a round trip to the player-profile service to validate tier eligibility, and that service was on a shared Postgres 15.4 cluster with 3 k TPS of unrelated traffic. The error stack in Jaeger was literally tracing_id=7f3a1c8… server=profile-db pool_timeout . We tried adding connection pooling with PgBouncer, but the hunt service was using raw libpq and refused to reuse connections—no matter how many times we told it. The Architecture Decision We ripped the validation out of the synchronous path and made the hunt engine publish an event called HuntTierCheckRequired to a dedicated Kafka topic player-events-tier . The hunt service would respond to the client with a 202 Accepted immediately, then the loot-claim worker would listen to that topic and, if the tier passed, publish HuntLootReady . The worker ran in the same pod but on a separate goroutine with a 60-second TTL so we didnt leak memory if the tier service hung. We moved the player-profile service to an SSD-backed CloudSQL instance and gave it 32 GB R

I lost more than 60 days but I made a huge update All OnemanBSD OS is now 64 bits. I also removed all bloat like Rust, Wayland and Qt dependencies from all app ports used in the system. (mpv was changed to gnome-mplayer because of that). New apps included (+ fixed source code of everything): audacity (audio editor) godot (game editor) deadbeef (mp3 player) ted (word processor) Also some bug fixes. Here is the link again: http://bialamusic.com/onemanBSD/ Here are my thoughts: I feel that we as humans are loosing control over technology that was created by humans. This is rising some serious and strange questions. I think we now need tools to power the individual so we can fight back. Not corporations. Not governments. Not organizations. Consider this my bullet in this battle. It is now in your hands so you can be active developers not just passive users.

E-food delivery is a trillion-dollar market . And most of that trillion is not going to farmers, store owners, or the people who actually move food around. It's going to the infrastructure layer sitting between them — the platform tax, the per-order cut, the SaaS subscription that charges you to exist inside someone else's garden. The walled garden isn't accidental. It's the product. What if food delivery was a protocol, not a platform? Not an app. Not a marketplace. A protocol — like HTTP, like SMTP — that any node can speak, that no single company owns, and that costs near zero to run. That's what DIFP is. The Djowda Interconnected Food Protocol. An open wire format for connecting food ecosystem participants — farms, stores, restaurants, wholesalers, delivery nodes, end users — directly to each other, without a platform in the middle extracting rent at every step. The spec covers: Presence & discovery — participants announce themselves to their spatial cell, others find them by location Orders, asks, and donations — not just commerce, but demand signals and surplus distribution in the same protocol Spatial routing — the MinMax99 grid maps the entire planet into ~500m cells; every message knows where it's going Decentralized registry — nodes find each other through a federated lobby system, no central server required Version 0.4 of the spec dropped a few weeks ago. Today we're publishing the first working implementation. The gRPC preview — what we built DIFP-gRPC is a skull implementation of the full protocol stack over gRPC. Thin, end-to-end, every domain wired — nothing production-hardened yet, everything clearly marked for what it is. What's inside difp.proto — the entire DIFP v0.4 spec as a single protobuf file. Two services, 30+ message types, the full DifpEnvelope wrapper with a oneof payload that covers every domain: message DifpEnvelope { string id = 1 ; string type = 2 ; // "trade.ask" | "presence.announce" | "node.ping" | … string version = 3 ; MessageSen