标签：#Data

Jörg Schad explains how to tame the complex "data management hairball" to build scalable, safe architectures for AI. He shares how autonomous data products act like containers for data, encapsulating pipelines, schemas, and metadata. Discover how progressive tool discovery via protocols like MCP limits context rot, enforces governance policies, and ensures reliable, multi-modal access. By Jörg Schad

The headline going around is that Spark 4.2 can retire your vector database. That's half true, which is the most dangerous kind of true. Spark did add real vector search, and for some workloads it genuinely removes a whole system from your stack. For others, you'd regret dropping your vector DB. Here's the honest version. The short answer If your vectors already live in your data platform and your searches are batch or analytical, Spark 4.2 can absolutely replace a separate vector database. If you're serving live, low-latency retrieval for a chatbot or search box, you probably still want a dedicated one. It's a "depends on the workload" answer, and the details matter. What Spark 4.2 actually added Spark 4.2 brought vector search into plain SQL. No bolt-on library, no separate engine. The new primitives include vector distance and similarity functions, vector normalization, vector aggregation like sum and average, and the headline one, NEAREST BY, a top-K ranking join that finds the closest matches by distance. In practice that means you can store embeddings in a Spark table and run a similarity search with SQL you already know. Those operations cover the real use cases: retrieval, recommendations, entity resolution, and candidate generation. Databricks is openly framing this as Spark becoming an AI serving layer, not just a batch engine. Why this is a big deal The value isn't that Spark invented vector search. Plenty of tools do it. The value is that you can keep your retrieval pipeline on one platform. Think about the normal setup today. Your data sits in a lakehouse or warehouse. To do vector search, you spin up a separate vector database, then build a pipeline to copy and sync embeddings into it, and keep the two in step forever. That's a second system to run, secure, pay for, and debug at 2 a.m. Spark 4.2 lets you skip that for a lot of cases. The embeddings stay where your data already is, and the search runs right there. Fewer moving parts is a real win, and i

Redis gets reached for reflexively, "just add Redis," as if it were a single fix for slowness. It's genuinely one of the most useful tools in a backend engineer's kit, but using it well starts with understanding what it actually is: an in-memory data structure store, not just a cache. Once you see it as a fast, versatile store of real data structures, the range of problems it solves cleanly (caching, rate limiting, queues, sessions, leaderboards, locks) stops looking like a grab bag and starts looking like one idea applied many ways. This is the opening article of the Redis Masterclass, and it builds on the PostgreSQL series : Redis usually sits alongside a primary database like Postgres, not instead of it. In-memory is the whole point Redis keeps its data in RAM. That single fact explains most of its character. Reading from memory is orders of magnitude faster than reading from disk, so Redis operations typically complete in well under a millisecond, and a single instance handles a very high request rate. That speed is why it's the default choice for anything on the hot path, where a database round trip would be too slow. The tradeoff is that RAM is smaller and more expensive than disk, and volatile. Redis addresses durability with persistence options we'll cover later, but the mental model to start with is: Redis is fast because it's in memory, and you use it for data that benefits from being fast to access, not as the permanent home for everything. It's a data structure store, not a key-value blob The common misconception is that Redis is a simple key-value store, strings in and strings out. It's much more. Redis stores real data structures as values, each with its own commands: Strings for simple values, counters, and cached blobs. Hashes for objects with fields, like a user record. Lists for ordered sequences and simple queues. Sets for unique collections and membership checks. Sorted sets for ranked data like leaderboards and priority queues. Plus streams, bit

Many databases use B+tree indexes, but they all differ. It's a sorted structure. The leaf pages are logically sorted so that a specific key value belongs to one page. A lookup by value reaches a single leaf page and either directly finds an entry for that value or immediately knows there's no entry with that key. When a page becomes full, it is split into two pages, each covering its own dedicated range. To find the right page, an internal page holds the range of values for the pages below. This internal page can become full, and a new level is added above it. Finally, at the highest level, there's a single internal page that is the root. A lookup always starts at the root and goes down to the leaves, following the branches of internal pages. In a traditional B+tree lookup, the cost is proportional to the height of the tree because the search starts at the root and descends to a leaf: 1 page to read when all fits in one leaf that is also the root (0 levels of internal pages, total height is 1). With small keys, this level can typically index hundreds of rows. 2 pages to read when there's one root that can list all leaf pages (1 level of internal page, total height is 2). With small keys, this level can typically index tens or hundreds of thousands of rows. 3 pages to read when there's one level of branches under the root (so 2 levels of internal pages, total height is 3). With small keys, this level can typically index millions of rows. This means that finding one key within ten million rows may require traversing 3 index pages, where most of them are probably in cache given the small number of branches compared to the leaves. For a given index size, whatever the value you are looking for, it's always the same number of pages to read because the index is balanced (the commonly accepted meaning of the B in B+tree). This property is maintained because any page can split, but only splitting the root adds another level. I've described how the height of an index can incr

You have a row in Postgres. You want that same row in MongoDB — not tonight in a batch job, but the instant it changes. And next month you'll want it in Redis too, and maybe POSTed to some webhook. Today that's a Kafka cluster, a Debezium connector, a sink connector, a schema registry, and a weekend. For a job that is, at its heart, one sentence: a source → one or more destinations → kept in sync. Syncle is an open-source tool that does exactly that sentence, and nothing you didn't ask for. Connect your databases, draw a bridge from a source to one or more destinations, and the moment a row changes in the source it's written to every destination you linked. Any engine to any engine — PostgreSQL · MySQL/MariaDB · SQLite · MongoDB · Redis — plus HTTP endpoints when you need them. This post is a tour of what it does and, for the curious, how it's built. The core idea: a bridge A bridge reads rows from a source and writes each one to its destinations . A destination is either: another database — the headline feature. Postgres → MongoDB, MySQL → SQLite, MongoDB → Redis. One bridge can fan out to several databases at once, and bridges can chain (A → B → C). an HTTP endpoint — POST/PUT/PATCH each row to a URL with a payload you design, for feeding a service instead of a database. The interesting part isn't that it copies data — plenty of things copy data. It's the guarantees around how . No duplicates, ever Every database write is an idempotent upsert , keyed by columns you choose. So replays, retries, and at-least-once redeliveries never double-write. Under the hood each engine does it with its own native atomic operation: Engine Upsert PostgreSQL / SQLite INSERT ... ON CONFLICT MySQL INSERT ... ON DUPLICATE KEY UPDATE MongoDB updateOne(filter, ..., { upsert: true }) Inserts, updates, and deletes all propagate — a delete routes to a keyed delete on each target. Missing table? It builds it If the destination table or collection doesn't exist, Syncle creates it from the sou

Jeremy Edberg & Qian Li discuss why external orchestrators decrease reliability and how to use your existing database for durable execution. They share how DBOS Transact uses standard tables, SKIP LOCKED queues, and unique primary keys to manage complex, fault-tolerant AI workflows with minimal latency, all without the operational overhead of separate distributed systems. By Jeremy Edberg, Qian Li

Every morning, an online store receives the previous day’s orders from a marketplace partner. The file comes in JSON format. The company needs to add those orders to its main MongoDB orders collection. The sales manager also needs a CSV report that can be opened in Excel. That sounds like a small task. Import the file, copy the documents, export the report. But in practice, a few things can break the process. A date can be imported as a string. A field can have the wrong name. One batch may use total , while the main collection uses totalAmount . A temporary collection can keep old records and trigger duplicate key errors. A CSV export can create null values because the mapping points to fields that do not exist. And then there is customer data. The manager may need the sales numbers, but they probably do not need real customer names or internal customer IDs. This article walks through a real daily workflow: Import marketplace JSON ↓ Store the batch in a temporary MongoDB collection ↓ Copy the orders into the main orders collection ↓ Mask customer fields during export ↓ Create a CSV report The goal is not just to move data from JSON to CSV. The goal is to make the process repeatable, easier to check, and safer to share. The workflow The workflow has three jobs: Import Yesterday Orders ↓ Add Orders to Main ↓ Export Daily Sales Report The important part is the parent relationship between the jobs. Add Orders to Main depends on Import Yesterday Orders , so it only runs after the JSON file is imported successfully. Export Daily Sales Report depends on Add Orders to Main , so the CSV is created only after the main orders collection has been updated. This prevents the report from being generated when data is missing or incomplete. The incoming JSON file The partner sends a file with yesterday’s completed orders. A single order looks like this: { "orderId" : "ORD-2026-07-201" , "customerId" : "CUST-1003" , "customerName" : "Sofia Rossi" , "orderDate" : "2026-07-21T08:20:00

The bottleneck in a mature SOC is rarely analyst triage; rather, it is the detection-engineering team's ability to keep the rule base aligned with a threat landscape that evolves faster than rules can be written. Learn how multi-agent system for production security operations has reduced mean times to detect and to respond by 40% and compressed the human work required by 12x. By Willem Berroubache

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n. Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried. I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical , every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway: config 0: 47/100 = 47.0% config 1: 50/100 = 50.0% config 2: 52/100 = 52.0% config 3: 52/100 = 52.0% config 4: 50/100 = 50.0% config 5: 50/100 = 50.0% config 6: 54/100 = 54.0% <- selected winner (argmax) config 7: 44/100 = 44.0% config 6: 54.0% (k=54 n=100 SE=4.98) config 2: 52.0% (k=52 n=100 SE=5.00) RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed. Held-out the winner on a fresh 100 items: 48/100 = 48.0%. Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE . The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be. TL;DR Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strict

Using Firestore as "the app's primary database" is easy at first. Writes and reads complete in one place, and onSnapshot gives you realtime push for free. But as a service grows, keeping Firestore as the SoT (source of truth) exposes some fatal constraints. Where SoT-Firestore hurts Weak transaction boundaries, missing complex queries, the cost structure, the difficulty of migrating away. The harshest one: you cannot narrow related updates with a WHERE . For the use case "update a set of documents matching a condition, consistently, in one go," Firestore is structurally weak. Redefining it as a public cache In one project, I made Cloud SQL the SoT and treated Firestore as a "read-optimized projection." Writes go through the backend (Next.js / Cloud Run), and the SoT (Cloud SQL via Data Connect) and Firestore are both updated within the same write path . Separately, a Cloud Run reconciliation job runs and checks and repairs consistency against the SoT as authoritative. This reconciliation job is enqueued at the start of the request, before the DB updates. Because it's queued first, even if the write dies halfway, the consistency check always runs afterward and repairs the state. From the client's perspective, Firestore is a rebuildable cache — in the worst case it can be rebuilt from the SoT. // Writes go through the backend. The backend updates both the SoT // (Data Connect → Cloud SQL) and Firestore. Clients never write Firestore directly. await backend . updateEntity ({ id , title }); // Changes are pushed back in realtime via Firestore's onSnapshot unsubscribe = onSnapshot ( entityRef , ( snap ) => render ( snap . data ())); Benefits and costs The benefits are clear. The SoT side (SQL) brings transactional consistency and complex queries; the read side (Firestore) brings freedom in data modeling and realtime push; and consistent syncing with external systems (OpenSearch / BigQuery, etc.) coexists naturally. Since the backend updates Firestore at write time, the f

OpenAI made a mistake setting up what it called a “highly isolated” testing environment and sandbox. According to cybersecurity experts, that human mistake is what made the AI-powered attack on Hugging Face possible.

There's a romantic idea about Bitcoin's chain: that it's a wall of human messages. Proposals, memorials, "Vahe was here," pizza jokes, the occasional protest note pinned into the world's most expensive append-only log. I wanted to know if that was actually true. So I counted. Every OP_RETURN output, from the genesis block to block 958,893, no sampling. The answer is no, and it's not close. The one number All human-readable OP_RETURN text ever mined into Bitcoin: 3,827,227 outputs. Runes, one token protocol, in its own era: 171,114,058 OP_RETURN outputs. That's a ratio of 44.7 to 1 . One machine protocol, in a single two-year stretch, wrote about 45 times more to the chain than every human-readable message in Bitcoin's entire history combined. The evidence, per era I split the chain into four eras by block height, not by any label stored in my database. Height boundaries are canonical and anyone can check them against a node, so the result doesn't depend on trusting my extractor's tags. era height range boundary event pre-ordinals 0 – 767,429 before the first inscription ordinals 767,430 – 779,831 inscription #0 to BRC-20 deploy boom-brc20 779,832 – 839,999 BRC-20 ordi deploy to Runes runes 840,000 – 958,893 Runes launch at the halving Then I counted the full population of OP_RETURN outputs in each era. Human-readable text, Runes token messages, and binary blobs (Veriblock and OMNI proof-of-proof timestamping, mostly). era total OP_RETURN human text human % Runes Runes % binary binary % pre-ordinals 51,965,944 861,532 1.66% 7 0.00% 51,103,723 98.34% ordinals 261,767 32,189 12.30% 3 0.00% 229,544 87.69% boom-brc20 2,980,954 402,161 13.49% 40,251 1.35% 2,538,248 85.15% runes 177,474,762 2,531,345 1.43% 171,114,058 96.42% 3,828,604 2.16% Here's the honest twist When I started, I expected to find a fall. A golden human era that machines later ate. That's the clean story, and it's wrong. Look at the human % column again. Human text was never the majority of OP_RETURN. Not

OpenAI will spend the equivalent of Sweden's GDP on infrastructure through 2030.

-- looks routine in review, locks your table in production: CREATE INDEX idx_orders_customer ON orders ( customer_id ); ALTER TABLE users ALTER COLUMN email SET NOT NULL ; Both of those statements hold a lock that blocks traffic while they scan or build against the table. On a small table you never notice. On a busy production table it is downtime, and a diff review does not catch it because the SQL looks fine. pg-migration-guard is a free tool that reads the migration and flags these before they reach production: $ pg-migration-guard V42__orders.sql WARN IDX01 CREATE INDEX without CONCURRENTLY [becomes a blocker on a large or busy table] Why: Plain CREATE INDEX takes a lock that blocks all writes until the build finishes. Safe: CREATE INDEX CONCURRENTLY idx ON tbl (col); -- outside a transaction WARN NN01 SET NOT NULL scans the whole table under ACCESS EXCLUSIVE Safe: ADD CONSTRAINT c CHECK (col IS NOT NULL) NOT VALID; VALIDATE CONSTRAINT c; then SET NOT NULL; Summary: 0 blocker, 2 warn, 0 advisory Why migrations bite Whether a Postgres migration is safe has almost nothing to do with intent and everything to do with locks. A migration is dangerous when it holds a strong lock while it scans or rewrites a large table. ACCESS EXCLUSIVE blocks reads and writes; SHARE blocks writes. The change that reads fine in review is the one that quietly takes that lock. Put it in CI The most useful place for this is the pull request, so a risky migration fails the build before anyone merges it. There is a GitHub Action: name : migration-guard on : pull_request : paths : [ " db/migration/**.sql" ] jobs : guard : runs-on : ubuntu-latest steps : - uses : actions/checkout@v4 - uses : actions/setup-python@v5 with : python-version : " 3.x" - uses : technical-turtle/pg-migration-guard@v1 with : path : db/migration fail-on : blocker Each finding shows up as an inline annotation on the diff, so the author sees exactly which line is the problem and what the safe rewrite is. Or run it locall

Jake Mannix discusses moving AI agents past chaotic "1970s BASIC" architectures. He shares how implementing an intermediate protocol layer allows engineering leaders to build versioned, encapsulated "virtual tools." This design enables interface mapping, dynamic schema projection, and runtime taint tracking to proactively eliminate data exfiltration risks without slowing velocity. By Jake Mannix

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N) , which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend<=budget over 200 deterministic agent runs observed failures: 0 (distinct scenarios: 200) naive point rate : 0.00% binomial SE: 0.00 pp naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200) , which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%] . A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much. TL;DR "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't. The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N) . The rule of three, 3/N , approximates it and rounds the risk slightly up. At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate. To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "

New data centers built through 2033 could consume as much electricity as India uses today.

When a gbase database cluster experiences an unclean shutdown — a sudden power loss or kernel panic — some metadata that was still in memory may never make it to disk. This can leave the critical gbase.table_distribution table with missing rows, causing queries against certain tables to fail with the error: ERROR 1149 (42000): (GBA-02SC-1001) Can't get distribution attribute of table `testdb`.`t2_test`, please check your gbase.table_distribution. The Symptom The affected table enters a contradictory state: Dropping it returns Unknown table . Re‑creating it returns Table already exists . The data files exist on disk, but the metadata record that describes how the table is distributed is gone. Root Cause GBase 8a stores critical table metadata — distribution type (hash or random), replication status, hash column — in the system table gbase.table_distribution . After an abrupt shutdown, this table may have missing entries, causing the cluster to lose track of the table's distribution properties. Recovery Options Option 1: Re‑insert the Metadata Row (Keeps Data Intact) This is the recommended approach when the data must be preserved. Insert the missing row directly into gbase.table_distribution . -- Insert the lost metadata record INSERT INTO gbase . table_distribution VALUES ( 'testdb.t2_test' , 'testdb' , 't2_test' , 'NO' , NULL , NULL , NULL , 'NO' , 2 ); -- Verify the row is now present SELECT * FROM gbase . table_distribution WHERE index_name LIKE '%t2_test%' ; Key fields: isReplicate : YES if the table is replicated, otherwise NO . hash_column : The distribution key column name (if hash‑distributed); NULL for random distribution. data_distribution_id : A numeric ID. Copy the value from another healthy table in the same database to keep it consistent. After inserting, restart the gcluster service so the metadata takes effect (the table_distribution table is managed by gcluster, not gnodes): gcluster_services gcluster restart The table should now be queryable again.

Bhavuk Jain discusses translating foundational AI into scalable mobile products. He shares the engineering challenges behind AI Wallpapers and Circle to Search, detailing how to implement robust runtime guardrails, fine-tuning, and seamless OS integration. For engineering leaders, he explains balancing UX constraints with model latency and infrastructure cost to deliver safe, reliable AI. By Bhavuk Jain

Yelp has launched Training Orchestrator. This new internal framework replaces individual team Spark training scripts. Now, it uses a configuration-driven, DAG-based execution model. By Claudio Masolo