标签：#STEM

One of the most common system design interview questions is: "How would you scale a web application from 100 users to 100 million users?" The answer is rarely a single technology. Instead, systems evolve through multiple stages, with each stage solving a specific bottleneck. This article walks through the typical evolution of a scalable system and explains why , how , and when each component is introduced. 1. Single Server Why Start Here? Every application starts simple. In the beginning: Traffic is low Development speed matters more than scalability Infrastructure costs should be minimal What Is It? A single machine handles everything: Frontend Backend Database Users | v Single Server ├── Application └── Database How Does It Work? User sends request. Application processes request. Database stores and retrieves data. Response is returned. Everything happens on one machine. Problem As traffic grows: CPU becomes overloaded Memory becomes insufficient Database competes with application for resources A single server becomes a bottleneck. Interview One-Liner A single server architecture is simple and cost-effective but becomes a bottleneck as traffic and resource usage increase. 2. Application and Database Separation Why Do We Need It? The application and database have different workloads. Application Server: Uses CPU Handles business logic Database Server: Uses memory and storage Handles queries Keeping them together causes resource contention. How Does It Work? Move the database to a separate machine. Users | v Application Server | v Database Server Benefits Independent scaling Better resource utilization Improved performance Example Suppose an e-commerce website receives thousands of requests. The application handles: Authentication Order processing API responses The database handles: Product data Orders User information Separating them prevents one workload from affecting the other. Interview One-Liner Separating the application and database allows each layer to scal

When organizations scale, a simple monolithic architecture often becomes difficult to maintain, deploy, and scale. This is where microservices come into the picture. However, moving to microservices introduces new challenges: How do we migrate from a monolith safely? How do we handle transactions across multiple services? How do we scale read-heavy applications efficiently? Three popular patterns solve these problems: Strangler Pattern – Monolith to Microservices Migration Saga Pattern – Distributed Transaction Management CQRS (Command Query Responsibility Segregation) – Read/Write Scalability 1. Strangler Pattern Why Do We Need It? Most companies cannot shut down a production monolith and rewrite everything from scratch. A complete rewrite is risky because: Development takes a long time. Existing customers are affected. Bugs can impact business operations. Rollback becomes difficult. The Strangler Pattern allows teams to migrate gradually with minimal risk. What Is It? The Strangler Pattern is a migration strategy where new microservices slowly replace parts of a monolithic application until the monolith is no longer needed. The name comes from the strangler fig tree, which gradually grows around another tree and eventually replaces it. How Does It Work? [Insert diagram here showing Client → API Gateway → Monolith + Microservices] Step 1 All requests go to the monolith. Client | v Monolith Step 2 Introduce an API Gateway (or Controller). Client | v API Gateway | v Monolith Step 3 Extract one module into a microservice. Client | v API Gateway |------> Order Service | v Monolith Step 4 Gradually move more modules. Client | v API Gateway |------> Order Service |------> Payment Service |------> Inventory Service | v Monolith Step 5 Eventually remove the monolith completely. Example Consider an e-commerce application. Initially, everything exists inside one application: Monolith ├── Orders ├── Payments ├── Inventory └── Users Over time: Orders become Order Service Payme

Amazon Fulfillment: The Three Tiers of Optimization Amazon processes billions of orders annually through a network of over 175 fulfillment centers globally. To maintain their 1-2 day (or same-day) delivery guarantees, they built a 3-tier optimization architecture: ┌─────────────────────────────────────────────────────────────┐ │ TIER 1: ANTICIPATORY SHIPPING (Long-term — weeks/months) │ │ → ML predicts demand → Moves inventory close to customers │ │ BEFORE they place an order │ ├─────────────────────────────────────────────────────────────┤ │ TIER 2: REGIONALIZATION (Medium-term — days/weeks) │ │ → Partitions the fulfillment network into autonomous zones│ │ → Ensures 70-80% of orders are fulfilled intra-region │ ├─────────────────────────────────────────────────────────────┤ │ TIER 3: CONDOR (Short-term — hours) │ │ → Continuously re-optimizes the fulfillment plan within │ │ a 5-6 hour window before pick-and-pack begins. │ └─────────────────────────────────────────────────────────────┘ Anticipatory Shipping — Shipping Before You Buy A Crazy but Effective Idea Amazon holds a patent (US Patent 8,615,473) describing a system that begins shipping items BEFORE a customer places an order . It sounds like science fiction, but it's a reality. Traditional Model: Customer orders → Warehouse processes → Ships → Delivered (2-5 days) Anticipatory Shipping: ML predicts: "Customers in Region X will buy 200 iPhone 16s in the next 3 days" → Amazon ships 200 iPhones from a central hub to local delivery hubs in Region X → Customer places order → The item is already locally staged → Delivered same-day! ML Model Input Features Input Feature Significance Purchase history What do they buy, and how often? Browsing behavior What are they looking at? Cart abandonment? Wishlists Explicitly desired items Seasonal patterns Winter coats in November, sunscreen in June Regional demographics High-income areas? Young families? College towns? Trending products Items going viral on social media Weathe

Every non-trivial business operation touches more than one system. An e-commerce order reserves inventory, charges a payment method, and schedules a shipment — three services, three databases. A bank transfer debits one account and credits another across two ledgers that may not even be in the same data center. A cloud VM provisioning workflow reserves a network port, allocates storage, starts the hypervisor, registers billing, and sends a notification — five services, five independent state stores. The question is: what happens when step four fails after steps one through three have already succeeded? In a monolith backed by a single database, the answer is simple: roll back the transaction. The database engine guarantees atomicity; either everything commits or nothing does. But when your workflow spans multiple services, each owning its own storage, there is no transaction boundary that wraps them all. There is no rollback button. Step one through three have already made durable changes to systems that do not know about each other, and step four's failure has left the system in an inconsistent state. This is not a pathological edge case. It is the default condition in any distributed architecture. And it gets worse: the failure might not be a hard error. The network might time out. The billing service might return a 503. You do not know whether step four applied its effect or not — you only know you did not receive a success response. Now what? This is the problem sagas were designed for. Client Inventory Svc Payment Svc Shipping Svc │ │ │ │ 1 │──reserve(item)──►│ │ │ │◄──── 200 OK ─────│ │ │ │ [reserved ✓] │ │ │ │ │ │ 2 │──────────── charge(card, $99) ────►│ │ │◄───────────────── 200 OK ──────────│ │ │ │ [charged ✓] │ │ │ │ │ 3 │─────────────────────── schedule(order) ─────────────►│ │◄─────────────────────────── 503 ──────────────────── │ │ │ │ [no record ✗] │ │ │ │ ╔══════════════════════════════════════════════════════╗ ║ ⚠ Inconsistent state ║ ║ Inventory: it

Your AI agent remembered the user's name. Then it forgot what it was doing. Here's the setup: User asks the agent: book the cheapest flight to NYC, search hotels under $150/night, then compare total trip cost. By step 3, the agent calls the LLM with 8,000 tokens of raw conversation history — and still answers as if it's turn 1. You need a memory architecture before this ships. Which one do you pick? A) In-context window only — full conversation stays in the system prompt. Simple. Breaks at ~15 turns or 8K tokens, whichever comes first. B) Vector memory store — embed past turns, retrieve the top-k by semantic similarity at query time. Works great until "NYC flight" pulls a memory about a past NYC trip instead of the current task. C) Episodic memory with summarization — compress old turns into structured event summaries, inject the relevant ones per request. More complex to build. Much harder to confuse. D) Redis session state — structured key-value store, explicit agent reads/writes. Deterministic. Requires the agent to know what to store and when. One of these collapses past 15 turns. One retrieves the wrong context at exactly the wrong moment. One is the right answer for task-oriented agents. Pick A, B, C, or D — and tell me where you've hit this in production. Full breakdown in the comments.

A common way to design an application is to begin with its nouns: User Product Order Payment Then each noun receives the standard architectural starter pack: UserController UserService UserRepository The controller receives users, the service services them, and the repository stores them somewhere responsible. This is noun-oriented architecture : treating every important thing in the domain as if it were automatically a useful software boundary. It works for simple CRUD systems. Unfortunately, most applications eventually do something. The noun becomes a drawer Consider a typical UserService : register() findByEmail() resetPassword() changeAddress() disableAccount() mergeAccounts() assignRole() calculateDiscount() These operations all involve a user. That is approximately where their similarity ends. They have different rules, dependencies, side effects, security concerns, owners, and reasons to change. They live together because User was the nearest available noun when the folders were created. As more behaviour accumulates, UserService becomes the official location for anything vaguely user-shaped. Other components depend on it. It gradually depends on authentication, email, permissions, billing, auditing, and several services added during incidents nobody wishes to revisit. The noun becomes both a dependency of everything and a consumer of everything. The folder remains impressively tidy. Name the capability, not the material A better starting question is not: What things exist in this system? It is: What must this system be capable of doing? That leads to components such as: UserRegistrar PasswordResetter AccountMerger OrderPlacer PaymentRefunder SubscriptionCanceller These are agentive names . They name the component responsible for performing a capability. Compare: UserService with: PasswordResetter UserService tells us which noun is nearby. PasswordResetter tells us what the component is for. That difference produces better architectural questions: What rules

Rate Limiting and Circuit Breakers in Distributed AI Systems Distributed AI systems are inherently complex, handling massive volumes of requests, variable latency from model inference, and dependencies on external services like GPU clusters, databases, or third-party APIs. Without proper safeguards, a single misbehaving component or a sudden traffic surge can cascade into system-wide failure. Two fundamental patterns— rate limiting and circuit breakers —provide essential protection. This post explores their roles, implementation strategies, and practical Python examples tailored for AI workloads. Why Distributed AI Systems Need These Patterns Consider a typical AI pipeline: a user sends a prompt, which hits a load balancer, then an API gateway, then an inference service (e.g., a large language model), which may call a vector database or a fine-tuning API. Each component has capacity limits: GPU inference servers can handle limited concurrent requests. External APIs (e.g., OpenAI, HuggingFace) impose rate limits. Database connections are finite. Without rate limiting, a single abusive client can exhaust resources. Without circuit breakers, a failing downstream service can cause cascading timeouts and resource exhaustion across the entire system. Rate Limiting: Controlling Request Flow Rate limiting restricts how many requests a client, user, or service can make in a given time window. It prevents resource starvation and ensures fair access. Common Algorithms Algorithm Pros Cons Token Bucket Smooth burst handling, easy to implement Memory per bucket Leaky Bucket Constant outflow rate, simple Less flexible for bursts Fixed Window Simple, low overhead Boundary spikes (reset issues) Sliding Window Smoother than fixed, accurate Slightly more complex For AI systems, token bucket is often preferred because it allows short bursts (e.g., a user sending a batch of prompts) while maintaining a long-term average. Python Implementation: Token Bucket Rate Limiter import time impor

If you’ve used S3, MinIO, or any cloud storage API, it’s easy to assume object storage is just a “cloud folder system.” That assumption is wrong — and it leads to confusion when you start working with distributed systems. Object storage is not a file system. It’s closer to a distributed key-value system with strong durability guarantees and a very specific access model . Once you understand that shift, a lot of cloud infrastructure starts to make more sense. The mental model most people start with When people first see object storage, they imagine something like this: /photos/cats.png /photos/dogs.png A hierarchical file system: folders subfolders files inside directories This is how traditional systems like ext4 or NTFS work. But object storage doesn’t actually work this way. The actual model: key → object Object storage is much simpler at its core: key → value Example: key : photos/cats.png value : <binary data> There are no real folders. “folders” are just string prefixes used for organization. That’s it. Why this design exists This model isn’t accidental. It solves real distributed system problems. Traditional file systems struggle when you try to: scale across many machines replicate data reliably handle partial failures coordinate metadata changes at scale Object storage avoids many of these problems by simplifying the model. Instead of supporting complex file operations, it focuses on: store object retrieve object delete object list objects by prefix Nothing more. The most important design choice: immutability In most object storage systems: Objects are not modified in place. If you “update” a file, what actually happens is: upload a new object replace the key pointer old object becomes orphaned (eventually cleaned up) This is a huge shift from file systems. Why this matters Immutability makes distributed systems easier because: no concurrent write conflicts on the same object replication becomes simpler caching becomes safer failure recovery is easier to rea

Every engineering team hits this conversation at some point. Someone proposes GraphQL. Someone else says REST is fine. A third person mentions gRPC and half the room goes quiet. The debate usually ends with the most senior person in the room picking what they're most familiar with. That's not a strategy — that's habit. Here's an objective breakdown of all three, when each one wins, and how to actually make the decision for your specific use case. The Core Mental Model Before comparing them, understand what each one is optimizing for: REST optimizes for simplicity and broad compatibility GraphQL optimizes for flexibility and precise data fetching gRPC optimizes for performance and strongly-typed contracts None of them is universally better. Each one is a tradeoff. The right answer depends entirely on who is consuming your API and what they need from it. REST — The Default That Still Wins Most of the Time REST (Representational State Transfer) is not a protocol. It's an architectural style built on HTTP — verbs, URLs, and status codes most developers already understand. Where REST genuinely wins: Public APIs. If external developers are consuming your API, REST is the only reasonable default. The tooling, documentation patterns, and developer familiarity are unmatched. Stripe, Twilio, GitHub — all REST. Simple CRUD services. If your resource model is straightforward, REST maps cleanly to it. No overhead, no learning curve, no ceremony. Browser-native requests. REST over HTTP works directly in the browser without any special client. Fetch it, done. Where REST struggles: Over-fetching and under-fetching. A single REST endpoint returns a fixed shape. Mobile clients that need 3 fields get 40. Separate data needs often require multiple round trips. Versioning overhead. As covered in our previous post — every breaking change forces a versioning decision. This compounds quickly on complex APIs. GraphQL — Powerful, But You Need to Earn It GraphQL is a query language for your A

Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I did not understand how Kafka works. So I built a tiny one. No dependencies. No Zookeeper. No JVM. Just Python and the core ideas. Here is what I learned. The Three Things Kafka Actually Does People say "Kafka is a message queue." That is not quite right. Kafka is a distributed commit log . It has three jobs: Accept writes from producers and append them to a log Let consumers read from any offset in that log Remember where each consumer group is up to That third one is the thing that makes Kafka different from a traditional queue. A queue forgets a message once it is consumed. Kafka remembers. You can replay. You can have 10 different consumer groups reading the same topic at different speeds. The code to implement this is smaller than you think. brokelite: A Message Broker in 120 Lines import threading import time from collections import defaultdict from typing import Dict , List , Tuple class Partition : """ Append-only log for one partition of a topic. """ def __init__ ( self ): self . _log : List [ Tuple [ int , bytes ]] = [] # (offset, message) self . _lock = threading . Lock () self . _next_offset = 0 def append ( self , message : bytes ) -> int : with self . _lock : offset = self . _next_offset self . _log . append (( offset , message )) self . _next_offset += 1 return offset def read_from ( self , offset : int , max_count : int = 100 ) -> List [ Tuple [ int , bytes ]]: with self . _lock : return [ ( off , msg ) for off , msg in self . _log if off >= offset ][: max_count ] def __len__ ( self ): return self . _next_offset class Topic : """ A topic is just N partitions. """ def __init__ ( self , name : str , num_partitions : int = 3 ): self . name = name self . partitions = [ Partition () for _ in range ( num_partitions )] def route ( self , k

Most AI demos work perfectly on a laptop. But production AI systems can become fragile when everything is handled inside one synchronous API call. A user sends a request. The API extracts text. The API chunks the content. The API generates embeddings. The API stores data. The API waits for everything to finish. This may look simple in a demo, but it quickly becomes a problem in real systems. The problem with one giant API call In many AI applications, the API is expected to do too much. For example, in a document processing or RAG pipeline, one request may trigger multiple heavy steps: text extraction chunking embedding generation indexing summarization database updates If all of this happens inside one synchronous request, the API becomes slow and fragile. If one downstream step fails, the complete request may fail. If traffic increases suddenly, the API may become overloaded. This is why event-driven architecture becomes useful for AI workloads. A better approach: API + Kafka + workers Instead of making the API do everything, we can split the workflow into smaller services. The API accepts the request and publishes an event. Background workers consume events and continue the processing asynchronously. A simple flow looks like this: User Request ↓ FastAPI ↓ Kafka / Redpanda Topic ↓ Python Worker ↓ Next Processing Stage In my practical demo, I am using: FastAPI Redpanda Python workers Docker Compose Kafka-compatible messaging Why Redpanda? Redpanda is Kafka-compatible, which makes it useful for local demos and event-driven architecture experiments. It allows us to work with Kafka-style topics, producers, and consumers while keeping the setup simple for development. What this architecture gives us This approach helps with: decoupling services handling bursty workloads moving long-running tasks to background workers improving scalability isolating failures building production-style AI pipelines This pattern is especially useful for AI systems involving: document proce

Martin Kleppmann, an associate professor at Cambridge and author of Designing Data-Intensive Applications, discusses the evolution of data systems over the last decade, mainly the shift from monolithic databases to modular building blocks. Kleppmann underlines the importance of moving from cloud-centric data storage systems to decentralised data storage similar to Bluesky’s AT protocol. By Martin Kleppmann

Quorum is an incident command plane built on Amazon Aurora DSQL. The failover story lives in another post. This one is about a narrower question that turned out to be the foundation: when several responders write to the same incident at the same moment, across regions, during the worst minutes of an outage, how do you guarantee the record never forks into two conflicting truths. The answer is two design choices that are really one choice seen from two angles: event sourcing, and DSQL's optimistic concurrency control. The data model is append-only Quorum is event-sourced across four tables. Every state change is an immutable event appended to a log, not an in-place update. The current state of an incident is a fold over its events. There is no UPDATE incidents SET status = ... ; there is an acknowledged event, a note event, a resolved event, and the status you render is computed from them. The event's UUID is its primary key and its idempotency key at the same time. A retried write carrying the same UUID cannot double-apply: the insert collides on the primary key and becomes a no-op. That property sounds minor until you remember what kind of system this is. A tool designed to survive network failure retries writes constantly, and "the responder tapped resolve twice because the first response was slow" must not produce two resolutions. Append-only also suits the domain directly. For an incident system the audit trail is the product, not a side effect. "Who acknowledged this, at what time, and what did the timeline look like at 02:14" is a first-class question for the post-incident review and a compliance requirement in regulated environments. Event sourcing gives you that for free. It also gives DSQL a write pattern it likes, which matters more than you would expect. The stack, briefly TypeScript end to end. Kysely as a typed query builder rather than an ORM, because I wanted type safety without surrendering control of the SQL: on a distributed database the exact shap

Every time you tap "Book Ride," a system makes dozens of decisions in under two seconds: Which driver? What route? What's the real ETA? This article breaks down exactly how the dispatch algorithm works — from the greedy approach that fails at scale, to the bipartite graphs, batched matching, and surge pricing mechanics that power Uber, Lyft, Grab, and Gojek today. Why a Greedy Dispatch Algorithm Fails (Closest Driver Problem) The first instinct when designing a matching system is to pair every customer with their nearest driver. However, this Greedy approach causes massive losses at a system-wide scale: Example: 3 riders (R1, R2, R3) and 3 drivers (D1, D2, D3) Greedy Matching (closest driver): R1 ← D1 (ETA 2 mins) ✓ R2 ← D3 (ETA 8 mins) ← D2 was "taken" by R1, even though D2 is closer to R2 R3 ← D2 (ETA 10 mins) ← Terrible outcome Total ETA: 2 + 8 + 10 = 20 minutes Optimal Matching (global optimal): R1 ← D2 (ETA 3 mins) R2 ← D1 (ETA 3 mins) R3 ← D3 (ETA 4 mins) Total ETA: 3 + 3 + 4 = 10 minutes ← 50% better! Uber refers to this problem as Global Optimization — finding an assignment strategy that minimizes the total ETA of the entire system , rather than optimizing just for individual pairs. Bipartite Graph Matching: The Mathematical Foundation (Lyft) Before diving into the systems, it helps to understand the mathematical model that all ride-hailing matching engines share at their core. Lyft formalizes dispatch as a bipartite graph matching problem : Bipartite Graph: Set A (Riders): { R1, R2, R3, R4 } Set B (Drivers): { D1, D2, D3, D4, D5 } Edges: every possible Rider ↔ Driver pair Edge Weight: cost of that match (e.g., ETA, driver rating, distance) Goal: Find a set of edges (a "matching") where: - No rider is matched to more than one driver - No driver is matched to more than one rider - The total cost of all selected edges is minimized This is known as the Minimum Weight Bipartite Matching problem. The classical algorithm for solving it is the Hungarian Algorithm (

Network Protocols Network protocols define how computers communicate over a network. Whether you're opening Instagram, sending a WhatsApp message, watching Netflix, or transferring money through a banking app, some protocol is working behind the scenes to make communication possible. Client-Server Model What is it? The Client-Server model is a communication architecture where: Client requests a service or data. Server processes the request and returns a response. Most modern applications follow this architecture. How Does It Work? Client ---------- Request ----------> Server Client <--------- Response ---------- Server The client always initiates communication, and the server listens for incoming requests. Real World Example Instagram When you open Instagram: Mobile app sends a request. Instagram servers process the request. Feed data is fetched from databases. Posts are returned to your phone. Instagram App | V Instagram Server | V Database Advantages Centralized control Easier security management Easy maintenance Easier data consistency Disadvantages Server can become a bottleneck Single point of failure if not replicated Interview One-Liner Client-Server architecture is a centralized model where clients request resources and servers provide them. Peer-to-Peer (P2P) Model What is it? In a Peer-to-Peer network, every machine can act as both: Client Server There is no central server controlling communication. How Does It Work? Peer A <------> Peer B ^ ^ | | V V Peer C <------> Peer D Each peer can directly share resources with others. Real World Example BitTorrent Instead of downloading a file from one server: User | +--> Peer 1 | +--> Peer 2 | +--> Peer 3 Different parts of the file are downloaded from multiple peers simultaneously. Blockchain Bitcoin and Ethereum networks operate using Peer-to-Peer communication. Advantages Highly scalable No central server cost Better fault tolerance Disadvantages Harder to manage Security challenges Data consistency issues Inter

I Built a Consistent Hashing Ring in Pure Python and Finally Understood How Cassandra Distributes Data I've been using Cassandra and Redis Cluster for years. I knew consistent hashing was "how they work." But I never truly got it until I built one myself from scratch, in pure Python, with zero dependencies. This post is about what I learned doing that. The Problem Consistent Hashing Solves Imagine you have 3 servers and 1 million keys. The naive approach: server = hash(key) % 3 . It works great until you add or remove a server. Change 3 to 4, and almost every key remaps to a different server. In a caching layer, that means near 100% cache miss. In a database, it means massive data movement. That's the problem consistent hashing solves. When you add or remove a node, only a fraction of keys move. Specifically, 1/n of the keys, where n is the number of nodes. Building the Ring The core idea: place both nodes and keys on a circular number line from 0 to 2^32 (or any large integer). To find which node owns a key, walk clockwise until you hit a node. Here's the minimal version: import hashlib import bisect class ConsistentHashRing : def __init__ ( self , replicas = 150 ): self . replicas = replicas self . ring = {} # hash -> node name self . sorted_keys = [] # sorted hash positions def _hash ( self , key : str ) -> int : return int ( hashlib . md5 ( key . encode ()). hexdigest (), 16 ) def add_node ( self , node : str ): for i in range ( self . replicas ): virtual_key = f " { node } :vnode: { i } " h = self . _hash ( virtual_key ) self . ring [ h ] = node bisect . insort ( self . sorted_keys , h ) def remove_node ( self , node : str ): for i in range ( self . replicas ): virtual_key = f " { node } :vnode: { i } " h = self . _hash ( virtual_key ) del self . ring [ h ] idx = bisect . bisect_left ( self . sorted_keys , h ) self . sorted_keys . pop ( idx ) def get_node ( self , key : str ) -> str : if not self . ring : raise ValueError ( " Ring is empty " ) h = self . _hash

If you're building a modern data stack that requires either high-throughput transaction processing or large-scale analytical workloads, you've likely come across both Vertica and VoltDB (now rebranded as Volt Active Data). While both are distributed relational database management systems (RDBMS), they are architected for completely opposite use cases — choosing the wrong one can lead to 10x higher costs, missed latency SLAs, and poor application performance. In this guide, we break down every key difference between OpenText Vertica and Volt Active Data, with practical examples, real-world use cases, and best practices to help you make the right choice for your team. Table of Contents What is OpenText Vertica? What is Volt Active Data (Formerly VoltDB)? Core Differences Between Vertica and VoltDB Real-World Use Cases: When to Pick Which Best Practices & Common Mistakes Conclusion & Key Takeaways References What is OpenText Vertica? OpenText Vertica (formerly Micro Focus Vertica) is a columnar relational DBMS built exclusively for analytical (OLAP) workloads, first launched in 2005. As of 2026, the latest stable version is 26.1, with native lakehouse and Apache Iceberg export support for modern data ecosystems. Core Vertica Architecture Vertica's design is optimized for fast queries across massive datasets: Columnar storage : Data is stored by column instead of row, enabling significantly higher compression ratios and faster aggregation queries that only access a small subset of columns Massively Parallel Processing (MPP) : Query execution and data are distributed across hundreds of nodes for parallel processing Dual deployment modes : Enterprise Mode : Shared-nothing architecture with data stored locally on nodes for maximum performance Eon Mode : Compute and storage separated, using shared object storage (S3, GCS, ADLS) to scale compute independently of storage for cloud workloads Projections : Physical, sorted copies of data optimized for common query patterns (ins

As my backend engineering internship wraps up, I’ve been reflecting on the tasks that pushed me the hardest. Building minimum viable products is one thing, but making them resilient, scalable, and fault-tolerant is an entirely different beast. Here are two of the most memorable tasks from my time here—one solo dive into system scaling, and one team effort tackling asynchronous voice processing. Task 1: Scaling the Insighta Labs+ Query Engine (Individual) What it was Insighta Labs+ is a demographic intelligence platform where analysts and engineers run structured queries on user profiles via a CLI and a Web Portal (backed by GitHub OAuth and RBAC). My task was to take a functional MVP and evolve it into a robust query engine capable of handling tens of millions of records and hundreds of concurrent queries per minute. The problem it was solving The initial architecture worked flawlessly for a few thousand records, but under scale, it started showing cracks. Latency : Without indexing, every filter query triggered a full-table scan. Redundancy : Identical queries from different users wasted CPU and DB cycles. Write-Pressure : Users needed to bulk-upload CSVs containing up to 500,000 rows. Processing these synchronously locked the database, bringing read operations to a halt. How I approached it Instead of blindly throwing more server power at the problem, I focused on doing less work. Targeted Indexing : I added indexes only to frequently filtered columns. Caching & Normalization : I introduced Redis for TTL-based caching. To maximize cache hits, I built a query normalization layer. Whether a user queried "young males" or "men under 30" , the parser normalized the filter object into a canonical form before hashing the cache key. Connection Pooling : I set up PgBouncer to manage database connections and prevent exhaustion under high concurrency. Chunked Ingestion : For the massive CSV uploads, I implemented chunked streaming. Rows were validated individually; valid row

なぜ“特別な決済システム”の話は危険なのか？ 近年、SNSやメッセージアプリを通じて、「特別なPOS端末」や「秘密の決済システム」に関する話を目にすることがあります。 「通常の銀行システムを経由しない」 「オフラインでも大金を受け取れる」 「特別なカードと専用POSがあれば送金できる」 こうした説明は一見すると高度な金融技術のように聞こえます。 しかし、実際の決済システムを理解すると、多くの主張が現実的ではないことが分かります。 まず、POS端末とは何か？ POS（Point of Sale）端末は、店舗でクレジットカードやデビットカードによる支払いを処理するための装置です。 一般的な決済は以下のような流れで行われます。 顧客 ↓ POS端末 ↓ 加盟店契約銀行 ↓ カードブランド ↓ カード発行銀行 ↓ 承認または拒否 重要なのは、最終的な資金の確認を行うのはカード発行銀行であるという点です。 POS端末そのものが資金を生み出すことはありません。 「オフライン決済だから大丈夫」は本当か？ 一部の詐欺では、 「この端末はオフラインで動作する」 という説明が行われます。 確かに、現実の決済システムにはオフライン処理が存在します。 しかし、それは通信障害時の一時的な仕組みであり、最終的には銀行側との照合が行われます。 つまり、 オフライン処理 ≠ 資金の創造 です。 銀行が承認していない資金は、後の精算時に拒否される可能性があります。 なぜ人は信じてしまうのか？ 理由は単純です。 専門用語が多いからです。 例えば、 決済ネットワーク 国際ブランド オフライン認証 ISO規格 特殊プロトコル こうした言葉が並ぶと、本物らしく見えます。 しかし、本当に重要なのは技術用語ではありません。 重要なのは、 「お金はどこから来るのか？」 という一点です。 詐欺を見抜くための3つの質問 1. お金の出所はどこか？ 利益や送金の原資を説明できない場合は要注意です。 2. 誰が監督しているのか？ 銀行、決済事業者、規制当局など、責任主体が明確か確認しましょう。 3. 第三者による検証は可能か？ 説明が内部関係者の証言だけに依存している場合は危険です。 テクノロジーと金融リテラシー 新しい技術は私たちの生活を便利にします。 しかし、技術的な言葉が使われているからといって、その仕組みが正しいとは限りません。 本当に優れた金融サービスほど、 透明性が高い 説明が分かりやすい リスクが明示されている という特徴があります。 逆に、 「秘密」 「特別」 「限定」 「誰にも教えないでほしい」 といった言葉が頻繁に出てくる場合は、一度立ち止まって考えるべきです。 まとめ 金融詐欺の多くは、技術ではなく心理を利用します。 人々はお金を失うから騙されるのではありません。 「理解したつもりになる」から騙されるのです。 だからこそ、最も重要な防御策は、 「そのお金はどこから来るのか？」 というシンプルな質問を忘れないことです。 金融の世界に魔法はありません。 あるのは、透明な仕組みと説明可能な資金の流れだけです。

Removing expf() from a fire detector: one header, 1.95x faster, zero accuracy loss A smoke detector is not a demo project. When it fires, someone either evacuates in time or doesn't. The firmware running on that microcontroller has one job, and it needs to do it without hesitation, without bloat, and without dependencies that can fail in unexpected ways. Last May 28th I published a bare-metal fire detection system built with Hasaki 刃先 — a neural network trainer that exports standalone C headers with no runtime, no Python, no TensorFlow. The model is a 12-8-4-1 MLP trained on 28,596 sensor readings. It fits in 3.8 kB of Flash and achieves 99.93% accuracy on held-out data, with a single missed fire event out of 3,599. But there was something in that header that bothered me. static inline float sigmoid ( float x ) { return 1 . 0 f / ( 1 . 0 f + expf ( - x )); } expf() . Right there in a life-safety application. On a microcontroller that may not have a hardware FPU. The problem with expf() on bare metal On processors with a hardware FPU — like the ESP32-C3 — expf() is fast. But the moment you deploy to an ATmega328P, an ATtiny85, or any Cortex-M0 target, that call becomes software floating-point. The CPU has to simulate the operation in firmware, cycle by cycle. It works. But it carries hidden cost: unpredictable latency, dependency on math.h , and a transcendental function sitting in the critical path of every single inference. For a smoke detector running at 1 Hz this might seem irrelevant. But inference latency compounds with sensor reads, normalization, and communication overhead. And more importantly — if you're deploying to a truly constrained target, expf() might be the difference between fitting in Flash or not. The fix: one header from kigu-quant kigu-quant(comming soon) is a new tool in the Rosito Bench ecosystem. It generates ready-to-include C headers for evaluating mathematical functions on microcontrollers — no FPU, no libm, no dependencies. One command: k