标签：#data

I run a small self-hosted observability tool on the cheapest VPS I could find on purpose: 2 cores, 2 GB RAM, 20 GB SATA SSD . It ingests errors, traces and metrics from two low-traffic sites of mine. The stack is three containers — a Go app, PostgreSQL, and ClickHouse. One evening docker stats showed ClickHouse sitting on 880 MB of its 1 GB limit and the box swapping, with basically zero events coming in. So I went looking for where the memory and disk had gone. The answer turned out to be a good lesson in how a database can spend almost all of its I/O talking to itself. 543 KB of my data, 579 MB of ClickHouse talking about ClickHouse First thing I checked: how much data had my app actually stored versus how much ClickHouse had stored about itself . My application database: 543 KB, 16k rows The system database: 579 MB, 46.3M rows Roughly a thousand to one. Disk was 12 GB used out of 20 — on a tool that had recorded half a megabyte of real telemetry. The culprit was ClickHouse's own system logs, several of which have no TTL by default and therefore grow forever: trace_log — 404 MB, 26M rows (the query profiler writes here; it's on by default, sampling once per second) asynchronous_metric_log — 16.6M rows text_log — 132 MB plus query_log , latency_log Only metric_log , processors_profile_log and part_log ship with a TTL. Everything else just accumulates. Then I looked at the insert rate over 30 seconds: trace_log — 227 rows/s asynchronous_metric_log — 157 rows/s text_log — 44 rows/s my application — about 5 rows/s 98.8% of all inserts were ClickHouse narrating its own internals. The part that's expensive beyond disk Here's the number that made me stop. Over the same 30 seconds: rows inserted : 16,222 rows merged : 11,007,643 That's a 1 : 678 ratio. For every row written, the engine rewrote 678 already-sitting rows. The mechanics: MergeTree drops every insert into its own data part, then merges parts into bigger ones so reads stay fast. When the table is small this is

A normal secondary index can answer: status = "active" It cannot efficiently answer: documents containing "distributed database" LioranDB therefore has a dedicated text-segment architecture. Tokenization Text is split on non-alphanumeric characters. Depending on index options, tokens can be normalized to lowercase and filtered through stopwords. "Building Distributed Databases" becomes ["building", "distributed", "databases"] Segment contents A LioranDB text segment can contain several files and structures: Term dictionary Posting lists Document map Document-length norms Optional term positions Bloom filter Segment metadata A posting connects a term to the local documents containing it. "database" → [doc 2, doc 8, doc 19] Positions can record where the term appears inside each document. That enables more advanced query behaviour and phrase-aware features. Global and local document IDs Each segment assigns compact local IDs to its documents. A separate document map translates them back to global document IDs. This keeps postings smaller while preserving the external identity of the record. Bloom filters Each segment also maintains a Bloom filter for terms. Before reading a segment's postings, the query path can test whether the term might exist there. A negative answer is definitive. A positive answer means the segment may contain the term and should be checked. Query modes The text query layer supports modes such as: AND OR It also emits scored documents and metrics including: Query time Postings read Candidate documents Segments searched Full-text search is essentially a specialized database living beside the document database. Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake. Built by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

Disk structures are durable, but updating them for every write is expensive. LioranDB uses memtables to absorb writes before flushing them to the on-disk B+ tree. What is a memtable? A memtable is an ordered in-memory map. In LioranDB, each entry contains either: Value ( bytes ) or: Tombstone A tombstone represents a deletion. The memtable also tracks: Approximate memory usage Minimum LSN Maximum LSN Entry count Put count Delete count The write path A simplified write path looks like this: Application write ↓ WAL durability ↓ Mutable memtable ↓ Immutable memtable queue ↓ Background flush ↓ Disk B+ tree The active mutable memtable accepts new writes. When it crosses a size limit, the engine rotates it into an immutable memtable. That immutable table is no longer modified and can safely be flushed in the background. Why ordered maps? LioranDB uses an ordered map for memtable entries. This helps because the flush process can emit keys in sorted order, which is friendly to the B+ tree and bulk-write paths. It also simplifies range merging between: Mutable data Immutable data On-disk pages Backpressure Background flushing cannot be allowed to fall behind forever. LioranDB therefore tracks limits such as: Maximum immutable memtables Maximum immutable bytes Partition-wide queue limits Maximum writer stall duration If the disk cannot drain the backlog quickly enough, the foreground write path slows down. That may sound undesirable, but controlled backpressure is much safer than consuming memory until the process dies. A memtable is not merely a cache. It is a pressure valve between CPU-speed writes and disk-speed persistence. Without that valve, the engine would either become slow on every commit or dangerously accumulate unbounded work. Built by Swaraj Puppalwar under Lioran Group . Links: LioranDB Lioran Developer Solutions Lioran Group

Most developers think of LioranDB as a document database. Internally, however, its storage engine does not understand documents, objects, fields, or JSON. It understands only: table + key bytes + value bytes That separation is intentional. The architecture LioranDB is split into two major layers: Application ↓ Document DBMS ↓ Transactional key-value engine ↓ WAL, memtables, B+ tree, pager and disk The engine exposes operations such as: get ( table , key ) put ( table , key , value ) delete ( table , key ) scan ( table , range ) The DBMS layer then adds document-oriented features: Collections JSON encoding Queries Updates Secondary indexes Text indexes Transactions For example, a secondary index can be represented as: idx:status:active → document_id A text index can be represented as: inv:database → posting_list The storage engine does not need to know what status , active , or database means. It only stores ordered bytes. Why this matters This architecture keeps the core engine small and reusable. The engine focuses on difficult low-level concerns: Durability Page management Transactions Recovery Ordering Concurrency Range scans The DBMS focuses on application-level semantics. This also makes it possible to build different data models over the same engine in the future. A document database is therefore not one giant component. It is a collection of carefully separated layers. That separation is one of the most important architectural decisions inside LioranDB. LioranDB is being developed by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

Rule would allow states to decide how much—if any—public input there can be.

Jörg Schad explains how to tame the complex "data management hairball" to build scalable, safe architectures for AI. He shares how autonomous data products act like containers for data, encapsulating pipelines, schemas, and metadata. Discover how progressive tool discovery via protocols like MCP limits context rot, enforces governance policies, and ensures reliable, multi-modal access. By Jörg Schad

The headline going around is that Spark 4.2 can retire your vector database. That's half true, which is the most dangerous kind of true. Spark did add real vector search, and for some workloads it genuinely removes a whole system from your stack. For others, you'd regret dropping your vector DB. Here's the honest version. The short answer If your vectors already live in your data platform and your searches are batch or analytical, Spark 4.2 can absolutely replace a separate vector database. If you're serving live, low-latency retrieval for a chatbot or search box, you probably still want a dedicated one. It's a "depends on the workload" answer, and the details matter. What Spark 4.2 actually added Spark 4.2 brought vector search into plain SQL. No bolt-on library, no separate engine. The new primitives include vector distance and similarity functions, vector normalization, vector aggregation like sum and average, and the headline one, NEAREST BY, a top-K ranking join that finds the closest matches by distance. In practice that means you can store embeddings in a Spark table and run a similarity search with SQL you already know. Those operations cover the real use cases: retrieval, recommendations, entity resolution, and candidate generation. Databricks is openly framing this as Spark becoming an AI serving layer, not just a batch engine. Why this is a big deal The value isn't that Spark invented vector search. Plenty of tools do it. The value is that you can keep your retrieval pipeline on one platform. Think about the normal setup today. Your data sits in a lakehouse or warehouse. To do vector search, you spin up a separate vector database, then build a pipeline to copy and sync embeddings into it, and keep the two in step forever. That's a second system to run, secure, pay for, and debug at 2 a.m. Spark 4.2 lets you skip that for a lot of cases. The embeddings stay where your data already is, and the search runs right there. Fewer moving parts is a real win, and i

Redis gets reached for reflexively, "just add Redis," as if it were a single fix for slowness. It's genuinely one of the most useful tools in a backend engineer's kit, but using it well starts with understanding what it actually is: an in-memory data structure store, not just a cache. Once you see it as a fast, versatile store of real data structures, the range of problems it solves cleanly (caching, rate limiting, queues, sessions, leaderboards, locks) stops looking like a grab bag and starts looking like one idea applied many ways. This is the opening article of the Redis Masterclass, and it builds on the PostgreSQL series : Redis usually sits alongside a primary database like Postgres, not instead of it. In-memory is the whole point Redis keeps its data in RAM. That single fact explains most of its character. Reading from memory is orders of magnitude faster than reading from disk, so Redis operations typically complete in well under a millisecond, and a single instance handles a very high request rate. That speed is why it's the default choice for anything on the hot path, where a database round trip would be too slow. The tradeoff is that RAM is smaller and more expensive than disk, and volatile. Redis addresses durability with persistence options we'll cover later, but the mental model to start with is: Redis is fast because it's in memory, and you use it for data that benefits from being fast to access, not as the permanent home for everything. It's a data structure store, not a key-value blob The common misconception is that Redis is a simple key-value store, strings in and strings out. It's much more. Redis stores real data structures as values, each with its own commands: Strings for simple values, counters, and cached blobs. Hashes for objects with fields, like a user record. Lists for ordered sequences and simple queues. Sets for unique collections and membership checks. Sorted sets for ranked data like leaderboards and priority queues. Plus streams, bit

Many databases use B+tree indexes, but they all differ. It's a sorted structure. The leaf pages are logically sorted so that a specific key value belongs to one page. A lookup by value reaches a single leaf page and either directly finds an entry for that value or immediately knows there's no entry with that key. When a page becomes full, it is split into two pages, each covering its own dedicated range. To find the right page, an internal page holds the range of values for the pages below. This internal page can become full, and a new level is added above it. Finally, at the highest level, there's a single internal page that is the root. A lookup always starts at the root and goes down to the leaves, following the branches of internal pages. In a traditional B+tree lookup, the cost is proportional to the height of the tree because the search starts at the root and descends to a leaf: 1 page to read when all fits in one leaf that is also the root (0 levels of internal pages, total height is 1). With small keys, this level can typically index hundreds of rows. 2 pages to read when there's one root that can list all leaf pages (1 level of internal page, total height is 2). With small keys, this level can typically index tens or hundreds of thousands of rows. 3 pages to read when there's one level of branches under the root (so 2 levels of internal pages, total height is 3). With small keys, this level can typically index millions of rows. This means that finding one key within ten million rows may require traversing 3 index pages, where most of them are probably in cache given the small number of branches compared to the leaves. For a given index size, whatever the value you are looking for, it's always the same number of pages to read because the index is balanced (the commonly accepted meaning of the B in B+tree). This property is maintained because any page can split, but only splitting the root adds another level. I've described how the height of an index can incr

You have a row in Postgres. You want that same row in MongoDB — not tonight in a batch job, but the instant it changes. And next month you'll want it in Redis too, and maybe POSTed to some webhook. Today that's a Kafka cluster, a Debezium connector, a sink connector, a schema registry, and a weekend. For a job that is, at its heart, one sentence: a source → one or more destinations → kept in sync. Syncle is an open-source tool that does exactly that sentence, and nothing you didn't ask for. Connect your databases, draw a bridge from a source to one or more destinations, and the moment a row changes in the source it's written to every destination you linked. Any engine to any engine — PostgreSQL · MySQL/MariaDB · SQLite · MongoDB · Redis — plus HTTP endpoints when you need them. This post is a tour of what it does and, for the curious, how it's built. The core idea: a bridge A bridge reads rows from a source and writes each one to its destinations . A destination is either: another database — the headline feature. Postgres → MongoDB, MySQL → SQLite, MongoDB → Redis. One bridge can fan out to several databases at once, and bridges can chain (A → B → C). an HTTP endpoint — POST/PUT/PATCH each row to a URL with a payload you design, for feeding a service instead of a database. The interesting part isn't that it copies data — plenty of things copy data. It's the guarantees around how . No duplicates, ever Every database write is an idempotent upsert , keyed by columns you choose. So replays, retries, and at-least-once redeliveries never double-write. Under the hood each engine does it with its own native atomic operation: Engine Upsert PostgreSQL / SQLite INSERT ... ON CONFLICT MySQL INSERT ... ON DUPLICATE KEY UPDATE MongoDB updateOne(filter, ..., { upsert: true }) Inserts, updates, and deletes all propagate — a delete routes to a keyed delete on each target. Missing table? It builds it If the destination table or collection doesn't exist, Syncle creates it from the sou

Jeremy Edberg & Qian Li discuss why external orchestrators decrease reliability and how to use your existing database for durable execution. They share how DBOS Transact uses standard tables, SKIP LOCKED queues, and unique primary keys to manage complex, fault-tolerant AI workflows with minimal latency, all without the operational overhead of separate distributed systems. By Jeremy Edberg, Qian Li

Every morning, an online store receives the previous day’s orders from a marketplace partner. The file comes in JSON format. The company needs to add those orders to its main MongoDB orders collection. The sales manager also needs a CSV report that can be opened in Excel. That sounds like a small task. Import the file, copy the documents, export the report. But in practice, a few things can break the process. A date can be imported as a string. A field can have the wrong name. One batch may use total , while the main collection uses totalAmount . A temporary collection can keep old records and trigger duplicate key errors. A CSV export can create null values because the mapping points to fields that do not exist. And then there is customer data. The manager may need the sales numbers, but they probably do not need real customer names or internal customer IDs. This article walks through a real daily workflow: Import marketplace JSON ↓ Store the batch in a temporary MongoDB collection ↓ Copy the orders into the main orders collection ↓ Mask customer fields during export ↓ Create a CSV report The goal is not just to move data from JSON to CSV. The goal is to make the process repeatable, easier to check, and safer to share. The workflow The workflow has three jobs: Import Yesterday Orders ↓ Add Orders to Main ↓ Export Daily Sales Report The important part is the parent relationship between the jobs. Add Orders to Main depends on Import Yesterday Orders , so it only runs after the JSON file is imported successfully. Export Daily Sales Report depends on Add Orders to Main , so the CSV is created only after the main orders collection has been updated. This prevents the report from being generated when data is missing or incomplete. The incoming JSON file The partner sends a file with yesterday’s completed orders. A single order looks like this: { "orderId" : "ORD-2026-07-201" , "customerId" : "CUST-1003" , "customerName" : "Sofia Rossi" , "orderDate" : "2026-07-21T08:20:00

The bottleneck in a mature SOC is rarely analyst triage; rather, it is the detection-engineering team's ability to keep the rule base aligned with a threat landscape that evolves faster than rules can be written. Learn how multi-agent system for production security operations has reduced mean times to detect and to respond by 40% and compressed the human work required by 12x. By Willem Berroubache

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n. Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried. I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical , every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway: config 0: 47/100 = 47.0% config 1: 50/100 = 50.0% config 2: 52/100 = 52.0% config 3: 52/100 = 52.0% config 4: 50/100 = 50.0% config 5: 50/100 = 50.0% config 6: 54/100 = 54.0% <- selected winner (argmax) config 7: 44/100 = 44.0% config 6: 54.0% (k=54 n=100 SE=4.98) config 2: 52.0% (k=52 n=100 SE=5.00) RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed. Held-out the winner on a fresh 100 items: 48/100 = 48.0%. Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE . The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be. TL;DR Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strict

Using Firestore as "the app's primary database" is easy at first. Writes and reads complete in one place, and onSnapshot gives you realtime push for free. But as a service grows, keeping Firestore as the SoT (source of truth) exposes some fatal constraints. Where SoT-Firestore hurts Weak transaction boundaries, missing complex queries, the cost structure, the difficulty of migrating away. The harshest one: you cannot narrow related updates with a WHERE . For the use case "update a set of documents matching a condition, consistently, in one go," Firestore is structurally weak. Redefining it as a public cache In one project, I made Cloud SQL the SoT and treated Firestore as a "read-optimized projection." Writes go through the backend (Next.js / Cloud Run), and the SoT (Cloud SQL via Data Connect) and Firestore are both updated within the same write path . Separately, a Cloud Run reconciliation job runs and checks and repairs consistency against the SoT as authoritative. This reconciliation job is enqueued at the start of the request, before the DB updates. Because it's queued first, even if the write dies halfway, the consistency check always runs afterward and repairs the state. From the client's perspective, Firestore is a rebuildable cache — in the worst case it can be rebuilt from the SoT. // Writes go through the backend. The backend updates both the SoT // (Data Connect → Cloud SQL) and Firestore. Clients never write Firestore directly. await backend . updateEntity ({ id , title }); // Changes are pushed back in realtime via Firestore's onSnapshot unsubscribe = onSnapshot ( entityRef , ( snap ) => render ( snap . data ())); Benefits and costs The benefits are clear. The SoT side (SQL) brings transactional consistency and complex queries; the read side (Firestore) brings freedom in data modeling and realtime push; and consistent syncing with external systems (OpenSearch / BigQuery, etc.) coexists naturally. Since the backend updates Firestore at write time, the f

OpenAI made a mistake setting up what it called a “highly isolated” testing environment and sandbox. According to cybersecurity experts, that human mistake is what made the AI-powered attack on Hugging Face possible.

There's a romantic idea about Bitcoin's chain: that it's a wall of human messages. Proposals, memorials, "Vahe was here," pizza jokes, the occasional protest note pinned into the world's most expensive append-only log. I wanted to know if that was actually true. So I counted. Every OP_RETURN output, from the genesis block to block 958,893, no sampling. The answer is no, and it's not close. The one number All human-readable OP_RETURN text ever mined into Bitcoin: 3,827,227 outputs. Runes, one token protocol, in its own era: 171,114,058 OP_RETURN outputs. That's a ratio of 44.7 to 1 . One machine protocol, in a single two-year stretch, wrote about 45 times more to the chain than every human-readable message in Bitcoin's entire history combined. The evidence, per era I split the chain into four eras by block height, not by any label stored in my database. Height boundaries are canonical and anyone can check them against a node, so the result doesn't depend on trusting my extractor's tags. era height range boundary event pre-ordinals 0 – 767,429 before the first inscription ordinals 767,430 – 779,831 inscription #0 to BRC-20 deploy boom-brc20 779,832 – 839,999 BRC-20 ordi deploy to Runes runes 840,000 – 958,893 Runes launch at the halving Then I counted the full population of OP_RETURN outputs in each era. Human-readable text, Runes token messages, and binary blobs (Veriblock and OMNI proof-of-proof timestamping, mostly). era total OP_RETURN human text human % Runes Runes % binary binary % pre-ordinals 51,965,944 861,532 1.66% 7 0.00% 51,103,723 98.34% ordinals 261,767 32,189 12.30% 3 0.00% 229,544 87.69% boom-brc20 2,980,954 402,161 13.49% 40,251 1.35% 2,538,248 85.15% runes 177,474,762 2,531,345 1.43% 171,114,058 96.42% 3,828,604 2.16% Here's the honest twist When I started, I expected to find a fall. A golden human era that machines later ate. That's the clean story, and it's wrong. Look at the human % column again. Human text was never the majority of OP_RETURN. Not

OpenAI will spend the equivalent of Sweden's GDP on infrastructure through 2030.

-- looks routine in review, locks your table in production: CREATE INDEX idx_orders_customer ON orders ( customer_id ); ALTER TABLE users ALTER COLUMN email SET NOT NULL ; Both of those statements hold a lock that blocks traffic while they scan or build against the table. On a small table you never notice. On a busy production table it is downtime, and a diff review does not catch it because the SQL looks fine. pg-migration-guard is a free tool that reads the migration and flags these before they reach production: $ pg-migration-guard V42__orders.sql WARN IDX01 CREATE INDEX without CONCURRENTLY [becomes a blocker on a large or busy table] Why: Plain CREATE INDEX takes a lock that blocks all writes until the build finishes. Safe: CREATE INDEX CONCURRENTLY idx ON tbl (col); -- outside a transaction WARN NN01 SET NOT NULL scans the whole table under ACCESS EXCLUSIVE Safe: ADD CONSTRAINT c CHECK (col IS NOT NULL) NOT VALID; VALIDATE CONSTRAINT c; then SET NOT NULL; Summary: 0 blocker, 2 warn, 0 advisory Why migrations bite Whether a Postgres migration is safe has almost nothing to do with intent and everything to do with locks. A migration is dangerous when it holds a strong lock while it scans or rewrites a large table. ACCESS EXCLUSIVE blocks reads and writes; SHARE blocks writes. The change that reads fine in review is the one that quietly takes that lock. Put it in CI The most useful place for this is the pull request, so a risky migration fails the build before anyone merges it. There is a GitHub Action: name : migration-guard on : pull_request : paths : [ " db/migration/**.sql" ] jobs : guard : runs-on : ubuntu-latest steps : - uses : actions/checkout@v4 - uses : actions/setup-python@v5 with : python-version : " 3.x" - uses : technical-turtle/pg-migration-guard@v1 with : path : db/migration fail-on : blocker Each finding shows up as an inline annotation on the diff, so the author sees exactly which line is the problem and what the safe rewrite is. Or run it locall

Jake Mannix discusses moving AI agents past chaotic "1970s BASIC" architectures. He shares how implementing an intermediate protocol layer allows engineering leaders to build versioned, encapsulated "virtual tools." This design enables interface mapping, dynamic schema projection, and runtime taint tracking to proactively eliminate data exfiltration risks without slowing velocity. By Jake Mannix