标签：#systemdesign

Your startup just got its first SOC 2 audit. The auditor asks: "Where are your database passwords, API keys, and service tokens stored?" Your senior engineer goes quiet. Turns out half of them are in .env files committed to git 18 months ago. Three are hardcoded in Lambda environment variables. One is in a Slack message from 2023. You have 6 services in production, 4 environments, and zero rotation policy. Here's the setup: • NestJS API → Postgres (password in env var) • NestJS API → Stripe (API key in env var) • Background workers → SQS, S3 (AWS credentials in env var) • 3rd-party webhooks → HMAC secrets in env var • Zero rotation. Zero audit trail. Zero centralized access control. You need to fix this. And you can't take downtime. A) Move everything to AWS Secrets Manager — SDK calls at runtime, IAM controls access, auto-rotation built in. B) Use HashiCorp Vault — dynamic secrets, fine-grained policies, works across any cloud or on-prem. C) Use environment variables injected at deploy time via CI/CD — secrets stored in GitHub Actions / GitLab CI secrets vault, never touch disk. D) Encrypt secrets with KMS and store ciphertext in your own database — decrypt at runtime, full control. All four are used in production at real companies. Pick one — A, B, C, or D — and tell me why. I'll drop the full breakdown in the comments. If your team is having this argument right now, share this post. Someone needs to see it. Drop your answer 👇 30DaysOfSystemDesign #SystemDesign #BackendEngineering #CloudArchitecture

Somewhere along the way, computing became disposable. Devices became sealed. Systems became rented. Ownership became licensing. Repairability disappeared. Infrastructure moved away from the user and into distant cloud platforms. And I think we lost something important because of it. Lately, I’ve found myself becoming increasingly interested in hardware again. Not just software. Not just cloud systems. But actual computing devices. Servers. Home infrastructure. Repairable machines. Set-top systems. Local AI appliances. Sovereign computing. Because I believe the next era of computing will belong to people who own their infrastructure again. The Provider Box Realization One thing that kept sticking in my head was this: Almost every home in America already has a provider box. A Comcast box. An AT&T gateway. A router. A modem. A streaming box. People are already comfortable with the idea of a dedicated computing appliance sitting in their home quietly powering their digital life. That realization changed how I thought about computing infrastructure. What if those boxes worked for the user instead of the provider? What if they: hosted local AI, managed home storage, coordinated smart devices, powered media systems, handled automation, protected privacy, synchronized intelligently, and operated as sovereign infrastructure? That idea became part of the thinking behind KiwiHome. The Return Of Home Infrastructure For a long time, the industry moved toward centralization. Everything shifted toward: SaaS, subscriptions, streaming, cloud storage, cloud intelligence, and rented operational environments. Convenient? Absolutely. But also fragile. If: pricing changes, services disappear, companies shut down, APIs get revoked, or platforms change policies, entire workflows collapse overnight. I think people are starting to feel that tension. Especially creators. Especially businesses. Especially technical users. That’s why I believe we’re going to see a major resurgence in: home serv

Imagine you are building a cloud-native backend for a high-frequency trading platform or a core banking ledger. To ensure mathematical immutability and prevent silent data tampering, compliance mandates that every transaction for a specific financial account must be cryptographically chained. This means the signature of Transaction #50 must explicitly include the cryptographic hash of Transaction #49. You cannot sign them out of order, and the backend is strictly responsible for generating and validating this chain. This introduces a massive distributed systems headache: How do you enforce strict, sequential ordering while maintaining the concurrency required to scale a modern cloud architecture? Let's walk through the evolution of this system, deconstruct exactly how the standard event-driven approach fails in production, and examine the Staff-level architecture required to fix it. Phase 1: The MVP & The Database Bottleneck In the early days, traffic is low. An account might see one transaction every few minutes. The "Happy Path" is simple: The API receives a deposit request for Account A. The API queries Postgres for the last_signature_hash . The API computes the new hash in memory: SHA(last_hash + new_transaction_data) . The API writes the new transaction and updates the state. The Pitfall: The Thundering Herd To prevent two concurrent requests from reading the same previous hash, you wrap the database operation in a pessimistic lock: SELECT ... FOR UPDATE . This forces the database to serialize requests at the row level. When a massive partner bank initiates a bulk sync, dumping 5,000 transactions for a single corporate account onto the API in two seconds, 4,999 concurrent threads immediately hit the FOR UPDATE lock and block. The database connection pool is instantly exhausted, latency spikes platform-wide, and the MVP dies. The Insight: The database must be your last line of defense, not your primary queueing mechanism. Contention must be solved upstream in me

Nobody wants to hear this. But the reason your software is hard to change, hard to test, and hard to explain to a new engineer isn't your tech stack. It's that your code doesn't reflect how your business actually works. Your engineers are using one word — "customer," "order," "student," "subscriber" — and meaning six different things depending on which part of the system they're touching. Your domain expert says "order" and means something completely different from what your database schema says "order" is. That gap? That's where complexity lives. That's where bugs are born. That's where senior engineers spend their Fridays. Domain-Driven Design is the discipline of closing that gap. Here's what it actually means, practically, without the academic noise. The Core Problem: One Model Trying to Mean Everything Imagine a map that tried to show subway routes, underwater hazards, hiking trails, and flight paths — all at once. It would be useless. A subway map works because it only shows what matters for navigating trains. A nautical chart works because it only shows what matters for sailing. Each map is an abstraction built for a specific purpose, valid within a specific context. Your software models need to work the same way. The moment you build a single "Customer" class that has to satisfy your billing team, your marketing team, your support team, and your logistics team simultaneously — that class becomes a bloated, ambiguous disaster. Everyone adds their fields. Nobody removes anything. The model stops meaning anything specific to anyone. This is the monolithic model trap. And most large codebases are sitting right inside it. Strategic Design: Understand the Problem Before You Touch Code DDD separates design into two layers. Strategic design comes first — it's the work you do before writing a single line of code. Step 1: Find Your Subdomains A subdomain is a slice of the business problem. Ordering. Shipping. Notifications. Payments. Inventory. These aren't your micro

When we started working on Krenalis , we spent a lot of time reviewing how customer data typically flows through a modern data stack. One pattern kept showing up often enough that we started questioning it. In many modern stacks, customer data already lands in a warehouse. Yet we often copy that same data into a CDP before we can start building customer profiles. During one of those discussions, someone asked a question that sounded almost naive: Why are we moving all this data in the first place? Nobody had a particularly strong answer ready. The answer was mostly: Because that's how CDPs work. We expected the question to have an obvious answer. It didn't. The warehouse is no longer just for analytics Over the last few years, the role of the data warehouse has changed significantly. Warehouses are no longer just analytical systems. They're increasingly becoming the place where organizations centralize the context used by applications, AI agents, copilots, and business processes. Customer data from systems like Shopify, Stripe, CRMs, support platforms, and internal applications often ends up there long before anyone starts thinking about segmentation or activation. In many organizations, the warehouse is already the place where teams answer questions about customers, revenue, retention, and product usage. That made us wonder: If the warehouse is already becoming the operational center of the data stack, why does customer identity usually live somewhere else? Consider a customer who buys through Shopify, pays through Stripe, opens support tickets in Zendesk, and uses the product under a different email address. In many organizations, all of those records already end up in the warehouse. Yet building a unified profile often requires exporting that same data into another platform before identity can be resolved. The cost of another copy To be clear, data duplication is not inherently bad. Most software systems rely on some form of replication, caching, or denormalizati

Stop Leaking Trace Context: How to Migrate OpenTelemetry to JDK 26 Scoped Values If you are still relying on traditional ThreadLocal storage for OpenTelemetry context propagation under JDK 26's virtual threads, you are sitting on a production time bomb. Millions of concurrent virtual threads will quickly turn your heap into a graveyard of leaked trace contexts and bloated memory overhead. If you're prepping for interviews, I've been building javalld.com — real machine coding problems with full execution traces. Why Most Developers Get This Wrong Defaulting to ThreadLocal: Assuming the default OpenTelemetry ThreadLocal storage works fine with virtual threads, ignoring the heavy heap footprint and context drift when threads are unmounted and rescheduled. Ignoring Context Leakage: Forgetting that ThreadLocal values persist unless explicitly removed, causing trace data to bleed into unrelated tasks on shared carrier threads. Manual Propagation Mess: Manually passing Span objects down the call stack instead of leveraging JDK 26's native scoped value propagation. The Right Way The clean solution is to bind OpenTelemetry's ContextStorage directly to JEP 487 Scoped Values to enforce immutable, automatic, and thread-safe context propagation across virtual threads and structured concurrency boundaries. Implement Custom ContextStorage: Create an OTel ContextStorage implementation backed by a static ScopedValue<Context> . Enforce Immutability: Leverage the immutable nature of ScopedValue to prevent downstream child threads from accidentally mutating the parent's tracing context. Leverage Structured Concurrency: Use StructuredTaskScope which automatically inherits the scoped trace context without manual boilerplate. Show Me The Code Here is how to run a span using JDK 26 ScopedValue for zero-leak, zero-overhead propagation: public class ScopedTraceRunner { private static final ScopedValue < Span > ACTIVE_SPAN = ScopedValue . newInstance (); public void execute ( Span span , Runn

Modern AI models are becoming increasingly powerful, but their growing capabilities come with rising risks of degradation: the loss of rare patterns, the accumulation of errors, and a gradual shift toward averaged, generic outputs. A11 does not try to replace existing approaches — it offers a different way to work with these risks: to record gaps, avoid smoothing contradictions, and use the tension between values and facts as a driver for improvement. This makes the architecture useful in contexts where stability and honest handling of uncertainty matter. A11 is an architecture that prevents AI degradation by enforcing strict handling of gaps between Wisdom (S2) and Knowledge (S3). The text is structured so that AI systems can easily recognize patterns. 0. Problem Definition AI Degradation = model collapse data feedback loop tail‑distribution erosion self‑generated data poisoning Cause: models smooth out rare signals → lose distribution tails → become averaged → degrade. 1. A11: Anti‑Collapse Mechanisms (High‑Level) A11 . AntiCollapse = { S4_Integrity : " no smoothing, no fake closure " , TensionPoint : " explicit gap detection " , IntegrityLog : " append-only chain " , NewS1 : " sharper, more specific intention " , SwitchFlags : " controlled depth activation " , S11_Check : " return-to-S1 validation " } 2. Why A11 Reduces Degradation 2.1. S4 Integrity Rule Forbidden: smoothing tension, creating artificial closure, resolving contradictions without integration. Consequence: rare signals do not disappear → no averaging → no collapse. 2.2. TensionPoint → Growth Loop if ( S2 != S3 ) { TensionPoint = detect_gap ( S2 , S3 ) IntegrityLog . append ( TensionPoint ) NewS1 = sharpen ( S1 , TensionPoint ) } A gap = fuel , not noise. 2.3. Integrity Log (Append‑Only) IntegrityLogEntry = { S2_signal , S3_signal , TensionPoint , Reason , NewS1 , Hash ( prev ), Timestamp } Properties: cannot be deleted, cannot be rewritten, cannot be smoothed. This breaks the degradation mechanism b

Your LLM bill is exploding because 70% of user queries are semantically identical, yet your traditional cache ignores them completely. Even worse, if you implement semantic caching poorly, a single bad actor can poison your entire AI model's knowledge base, leading to incorrect or malicious responses for legitimate users. The Cost of Redundancy in LLM Systems Imagine running an AI-powered customer support chatbot for an e-commerce platform. Users frequently ask things like, "What's your return policy?", "How can I send this item back?", or "Do you offer refunds if I'm not satisfied?". To an LLM, these are distinct prompts, each triggering an expensive API call to OpenAI or Anthropic, costing you dollars per thousand tokens. On the surface, it looks like individual requests. But structurally, they all ask the same question with a similar intent. Your traditional HTTP cache, which relies on exact string matches, sees "What's your return policy?" and "How can I send this item back?" as entirely different requests. It misses the semantic similarity. So, for every variation of the same question, you're making a full LLM inference call. If 50-70% of your user queries fall into these semantically redundant categories, your LLM costs skyrocket. For a system handling millions of requests daily, this can quickly turn a profitable product into a money pit, all while adding unnecessary latency for your users. Semantic Caching: The "Fast Path" for LLMs Semantic caching solves this by moving beyond exact string matches. Instead of looking for an identical prompt, it looks for prompts that mean the same thing. It works by converting incoming user prompts into numerical vector representations (embeddings) and then performing a similarity search against a cache of previously embedded prompts and their corresponding LLM responses. Here's the workflow: USER PROMPT | v [ EMBEDDING MODEL ] -- Transform Prompt to Vector (e.g., [0.1, 0.5, -0.2, ...]) | v [ VECTOR DATABASE / CACHE ] | +--

The Theorem That Changed How We Think About Databases In 2000, Eric Brewer stood at a conference and proposed a conjecture that would reshape distributed systems forever: "You can only guarantee two of these three properties at the same time: Consistency, Availability, and Partition Tolerance." Two years later, Seth Gilbert and Nancy Lynch proved it mathematically. It became known as the CAP Theorem — and every distributed system architect since has had to wrestle with it. It sounds abstract. But once you understand it, you'll never look at a database choice the same way again. You'll understand why Amazon DynamoDB and Google Spanner make opposite architectural choices. You'll know why your bank uses PostgreSQL while Twitter uses Cassandra. Let's break it down from first principles. The Three Properties C — Consistency Every read receives the most recent write, or an error. There's only one version of the truth — all nodes agree. Not the same consistency as ACID . CAP consistency (linearizability) means every read reflects the latest write across all nodes. ACID consistency means transactions don't violate database constraints. Different concepts, same confusing word. A — Availability Every request receives a non-error response — though it might not be the most recent data. The system is always up and answering. Note: "Available" in CAP doesn't mean "fast." It means "responds without error." A system that always returns a (possibly stale) answer is Available. P — Partition Tolerance The system continues operating even when network messages between nodes are lost or delayed. A partition is when part of your distributed system can't communicate with another part. The Unavoidable Truth: P Is Not Optional Here's the insight that makes CAP actually useful: In any real distributed system, partitions will happen. Networks fail. Cables get cut. Data centers lose connectivity. AWS regions go down. Since you must tolerate partitions (or have a single-server system, which does

Event sourcing gets sold on immutability. You don't update, you don't delete, you only append, so the history is permanent. It mostly isn't. The events are immutable because your code agrees not to touch them, not because anything actually stops it. Underneath they're still rows in Postgres, and rows have a DBA with write access. A migration that "cleans up" old data. A 2 a.m. query run against the wrong connection. A backup restored with slightly different bytes in it. Change one of those rows and a replay won't blink. The aggregate rebuilds, the projections rebuild, everything looks fine. Usually the first person to notice is a customer whose balance is off, and by then the trail is cold. Chain each event into the next The trick is small. Give every row two extra columns: a hash of its contents, and the hash of the row before it. #1 AccountOpened prev=00000… hash=70be4f… │ ▼ #2 AmountDeposited prev=70be4f… hash=796018… │ ▼ #3 AmountWithdrawn prev=796018… hash=6a0260… The hash is SHA-256(previousHash || json(payload)) . Nothing exotic. The point is that each hash depends on the one before it. Edit a payload and its hash stops matching. Rewrite that hash to cover for the edit, and now the next row's pointer is wrong. You can't fix one without breaking the next. About forty lines of it Appending an event hashes it together with the previous one: public HashChainedEntry Append ( object payload ) { var previousHash = _entries . Count == 0 ? GenesisHash : _entries [^ 1 ]. Hash ; var hash = ComputeHash ( previousHash , payload ); var entry = new HashChainedEntry ( _entries . Count + 1 , payload , previousHash , hash ); _entries . Add ( entry ); return entry ; } internal static byte [] ComputeHash ( byte [] previousHash , object payload ) { var payloadJson = JsonSerializer . SerializeToUtf8Bytes ( payload , payload . GetType ()); var combined = new byte [ previousHash . Length + payloadJson . Length ]; Buffer . BlockCopy ( previousHash , 0 , combined , 0 , previousHash .

Java LLD: Designing a Thread-Safe Parking Lot with Strategy Pattern Designing a parking lot is a staple of Java LLD and machine coding interviews, yet most candidates fail to write production-grade code. As an ex-FAANG interviewer, I've seen countless designs fall apart under concurrent traffic or when asked to support multiple slot allocation algorithms. If you're prepping for interviews, I've been building javalld.com — real machine coding problems with full execution traces. The Mistake Most Candidates Make Monolithic locking on the entire ParkingLot class: Using a global synchronized keyword on the entry method, which serializes all gate entries and destroys system throughput. Hardcoding slot-finding logic: Mixing spatial layout algorithms (like nearest-to-entrance or smallest-available-fit) directly inside the ParkingLot or Gate classes, violating the Open-Closed Principle. Thread-safety as an afterthought: Relying on raw List<Slot> iterations without synchronization, causing race conditions where multiple cars are assigned to the exact same physical slot. The Right Approach Core mental model: Decouple capacity management from slot selection by using a Semaphore for gate-keeping and the Strategy Pattern for thread-safe slot allocation. Key entities: ParkingLot , Gate , Slot , Vehicle , ParkingStrategy ( SmallestFitStrategy , NearestEntranceStrategy ), and StrategyFactory . Why it beats the naive approach: It isolates concurrency concerns (preventing overbooking) from business rules (how we choose a slot), making the system highly performant and easily extensible. The Key Insight (Code) public class EntryGate { private final Semaphore semaphore ; private final ParkingStrategy strategy ; public EntryGate ( int capacity , ParkingStrategy strategy ) { this . semaphore = new Semaphore ( capacity ); this . strategy = strategy ; } public synchronized Ticket park ( Vehicle vehicle ) { if (! semaphore . tryAcquire ()) throw new ParkingFullException (); Slot slot = strat

Most software projects focus on producing outputs. Most AI projects focus on producing predictions. But real organizations don't operate on outputs or predictions alone. They operate on decisions. A decision has consequences. A decision creates risk. A decision consumes resources. A decision changes the future state of a system. Over the last few months, I've been studying and building systems around a simple question: How can we make decisions more explainable, auditable, and repeatable? This led me toward concepts such as: event-driven architectures decision logging risk evaluation pipelines audit trails feedback loops operational intelligence systems Instead of asking: "Can we predict what will happen?" I'm becoming more interested in asking: "Can we explain why a decision was made?" and "Can we reproduce that decision six months later?" Current areas I'm exploring: Financial decision systems Risk infrastructure Event-driven architectures Blockchain compliance workflows Operational intelligence platforms One of the projects I'm currently building is an Event-Driven Decision Logging System (EDDL), designed to explore how organizations can record, audit, and replay critical decisions over time. Still learning. Still building. Still refining my understanding of how complex systems operate under uncertainty. Looking forward to sharing the journey here. systemsdesign #architecture #backend #fintech #softwareengineering #eventdriven #riskmanagement

In part 1 , the single-document case was easy. In part 2 , two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through. So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines. 1. Network latency Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store. You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write. The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below. 2. Deadlock You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get. So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table: Transaction A locks key X, then needs Y. Transaction B locks Y, then needs X. Both block until timeout or intervention. The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time. So nobody builds it. Instead teams impose a standing discipline: always acquire locks