标签：#data

Most teams connecting AI agents to their data warehouse start with text-to-SQL. The agent generates SQL from natural language, runs it against the warehouse, and returns results. It works until it doesn't: hallucinated JOINs, inconsistent aggregations, no access control, no audit trail. There's a better approach. Define your business metrics in a semantic layer, expose them via MCP (Model Context Protocol), and let any AI agent query governed definitions instead of raw tables. Then add one tool so the agent can chart the result in Claude or ChatGPT. This tutorial shows how to set it up in under 30 minutes. Why does text-to-SQL break in production? The agent sees column names but not business logic. It doesn't know that your company excludes refunds from revenue. It doesn't know that status = 'completed' means something different in orders than in subscriptions . It doesn't know that marketing and finance defined "active user" differently three years ago and never reconciled. So the agent writes plausible SQL and returns plausible numbers. Ask the same question twice with different phrasing and you get different answers. Ask two different agents and you get two different numbers. Neither matches the number your finance team reports. Beyond consistency, there's no row-level security. No multi-tenancy. No audit trail showing which agent queried what, when, and for whom. In production, with real customers, that's a non-starter. Text-to-SQL gives you speed. It doesn't give you trust. What is the semantic layer approach? Instead of letting agents write arbitrary SQL, define your metrics once in YAML: cubes, measures, dimensions, access rules. Then expose those definitions via MCP so agents query governed metrics, not raw tables. The difference: every agent gets the same answer because the metric definition is fixed. total_revenue isn't a column the agent interprets. It's a pre-defined calculation with agreed-upon filters and aggregations. When your finance team updates th

A three-person agency received a $14,000 AWS bill in one day after attackers extracted static access keys and burned Claude invocations on Bedrock. Combined with May's DN42 incident, where an autonomous agent provisioned $6,531 of oversized infrastructure in 24 hours, practitioners warn that cloud billing lags roughly a day behind agent-speed spend. By Steef-Jan Wiggers

Gone are the hours of careful thought and planning that go into coding a new feature. Vibe coding is too risky though, so another Driven Development was created. I'm referring to SDD (Spec Driven Development) of course. The vibe coding approach is great for prototypes and throwaway code, but this way of working falls apart when teams realise that the code needs to be maintained. So the thing that helps fix this is SDD. Create a spec once from clear technical specs and then generate some high quality code. Sounds great, right. Reminds me of IaC, where you use a templating language to create infrastructure. Software as Code maybe. SaC anyone? Unfortunately, in practice it's not that straightforward. Thoughtworks have placed SDD into an "Assess" category and warned that it could be an anti-pattern for releasing software. Deterministic vs Probabilistic This article isn't about SDD. I'm more interested in discussing the output of SDD and how that is tested. Code can now be generated fast these days. So what better to test AI-written code than with AI itself. There are a lot of concepts and technical terms for the Quality Assurance part of AI generated code. One of these is the LLM-as-a-Judge idea. This idea is used to score the output of an LLM based on some explicit criteria. Traditionally, the way to evaluate an LLM was to judge its output on the helpfulness or faithfulness (using something called "exact-match" metrics). Sometimes it was usually down to a human to do this. It also changes the way that Quality is Assured when dealing with AI-written code. Traditional QA is built on deterministic checks; either something does or does not fail. Something like expect(x).toContainText(y); . A failing test means that something is wrong. Then the bug can be fixed in the code and the test will pass. However, the outputs of an LLM are probabilistic , so it breaks the traditional pass/fail model. This is where a judge comes in. Instead of pass/fail, it can assign a score based o

I made ClickHouse and Postgres Fight to Find Out 🥊 One Mac Mini. Two databases. 100 million rows of...

"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates." I've heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka's exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents. What Kafka's Exactly-Once Actually Covers Kafka's exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels: Producer idempotence ( enable.idempotence=true ): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count. Transactions ( transactional.id ): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions. Together, these give you exactly-once message delivery within the Kafka cluster. What Exactly-Once Does Not Cover Here's the boundary that engineers miss: Kafka's exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you're outside the transaction boundary. Consider a typical consumer: consumer . poll ( records ); for ( record : records ) { database . save ( process ( record )); // External write — outside Kafka transaction } consumer . commitSync (); If the application crashes after database.save() but before commitSync() , Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event. Enabling producer idempotence on the consumer's Kafka writes does not fix this. The Patterns That Actually Give You End-to-End Safety Idempotent Consumers Design consumer processing logic to be idempote

Every GEO ("generative engine optimization") tool, including ours until recently, sells some version of the same pitch: fix your robots.txt, add Schema.org markup, write FAQ schema, and AI engines will cite you more. We build one of these tools — Causabi scans sites for AI-crawler readiness and generates fix files (robots.txt, llms.txt, JSON-LD, FAQ blocks). As part of validating our own scoring weights, we ran the numbers on whether the score actually predicts getting cited. Short version: it mostly doesn't, once brand prominence is in the picture. What we measured We scored 44 domains on a 6-category on-site readiness algorithm: robots.txt (AI bots allowed or blocked) Schema.org (Organization/LocalBusiness JSON-LD completeness) FAQ schema (FAQPage markup, 3+ entries) content depth/structure brand/NAP signals freshness (dateModified, recency) Then we checked how often each domain actually got cited by an AI engine (Claude, via its web-search tool, one measurement window, a fixed prompt set per domain). What we found On-site score vs. citation rate: Pearson r ≈ -0.08, Spearman ρ ≈ -0.03. Functionally no correlation — if anything, a very slight negative one, which is more likely noise than a real inverse relationship at this sample size. 86% of the 44 domains got zero citations in the window, independent of their score. The domains that did get cited clustered almost entirely by brand prominence — well-known domains got cited at a noticeably higher rate (~0.16 of prompts) than everyone else (~0 for the rest of the sample), regardless of how well-optimized their markup was. Why I'm not overselling this n=44 is small. This is an internal validation exercise for our own product, not a peer-reviewed study, and I don't want it read as one. Specific caveats: Single engine (Claude) this round. Citation behavior differs meaningfully across ChatGPT, Gemini, Grok, and Perplexity — we haven't run the same check across all four yet. One time window, no longitudinal before/after.

DPDP compliance costs for Indian startups: what to budget before 13 May 2027 Summary. Full compliance with India's Digital Personal Data Protection Act is due on 13 May 2027. That is the date consent, notice, security safeguards, breach intimation and data-principal rights all become enforceable, and it is roughly 10 months away. The DPDP Rules 2025 were notified on 13 November 2025 and land in three phases: the Data Protection Board of India stood up immediately, penalties and Consent Manager registration begin on 13 November 2026, and everything else bites on 13 May 2027. The penalty schedule is not proportionate to your size: up to ₹250 crore for failing reasonable security safeguards, ₹200 crore for failing to notify a breach, and ₹150 crore for missing Significant Data Fiduciary obligations. There is no revenue or headcount exemption. The cost gap is where founders get hurt. Vendors routinely quote ₹15 lakh to ₹2 crore for DPDPA compliance, while a startup under 10,000 users can be substantively compliant for under ₹50,000 a year. India's privacy and data governance market is worth roughly $1 billion to $1.5 billion today, per IDfy founder Ashok Hariharan speaking to Inc42 in April 2026, and a lot of that revenue depends on you not reading the rules yourself. This is a budgeting guide, not a legal opinion. The rules are short. Read them, then price the work. The deadline that actually matters There are three dates, and only one of them is a real deadline for most startups. Phase Date What switches on Does it affect a typical startup? Phase 1 13 November 2025 Data Protection Board of India established, Rules notified No direct obligation, but the Board can already receive complaints Phase 2 13 November 2026 Consent Manager registration opens, penalty framework and enforcement powers begin Only if you intend to register as a Consent Manager Phase 3 13 May 2027 Notice, consent, security safeguards, breach intimation, retention, children's data, data-principal righ

Risk stratification sounds like a data-science buzzword until you have to build the thing. For a Medicare Advantage plan, it's a concrete pipeline: take a population of members, score each one's clinical and financial risk, and rank them so care management and documentation teams know who to touch first. Here's how I'd architect it. The core idea Population health risk stratification = scoring + segmentation. You compute a per-member risk signal, then bucket members into tiers (e.g., rising-risk, high-risk, catastrophic) so finite resources go where they move outcomes and revenue most. The mistake teams make is treating it as a single ML model. In practice you want a layered signal: a stable, explainable base (RAF + chronic conditions) plus optional predictive overlays. Explainability matters because care managers won't act on a black-box score, and auditors won't accept one. Step 1: Build the member feature record { "member_id" : "SYNTH-77310" , "age" : 73 , "hccs" : [ "HCC37_1" , "HCC85" , "HCC18" ], "raf" : 1.842 , "gaps" : [ "a1c_overdue" , "no_pcp_visit_180d" ], "utilization" : { "ed_visits_12m" : 3 , "inpatient_12m" : 1 } } The RAF here is your defensible, model-grounded risk anchor under CMS-HCC V28. Everything else is supplemental signal. Step 2: Score and tier def risk_tier ( member ): base = member [ " raf " ] util = 0.15 * member [ " utilization " ][ " ed_visits_12m " ] \ + 0.30 * member [ " utilization " ][ " inpatient_12m " ] score = base + util if score >= 3.0 : return " catastrophic " if score >= 1.8 : return " high " if score >= 1.0 : return " rising " return " stable " Keep the weights transparent and tunable. The point isn't a perfect model; it's a defensible, reproducible ranking your operational teams trust. Step 3: Make "rising-risk" actionable The tier that quietly drives the most ROI is rising-risk — members trending toward high cost who still have open documentation and care gaps. Surface their specific gaps (overdue labs, undocumented chroni

Database branching is one of the best ideas serverless Postgres brought to the mainstream. Fork the database at a point in time, get an isolated copy with all the data, run something risky against it, throw it away. It made preview databases and safe migrations feel routine. But a real application is not just a database. It is a database, plus the files it stores in object storage, plus the backend code that serves it, plus, increasingly, the model and gateway config it calls for AI. When you branch only the database, those other three stay shared. Your "branch" points at the same S3 bucket, the same deployed backend, and the same AI configuration as everything else. So it is half a copy, and the half it leaves out is where a lot of the interesting bugs and the scary migrations live. Neon's platform preview changes what a branch contains. A branch now forks the database and its data, the object storage and its files, the functions that run your backend, and the AI gateway config, all at the same point in time, all isolated. A branch stops being a database copy and becomes a whole environment. To make sure that is a real claim and not a diagram, I took a full-stack project, branched it, and checked every layer. Here is what happened. TL;DR Elsewhere, "branch" means the database only. Object storage, backend deploys, and AI config stay shared, so you bolt on scripts to fake per-branch versions of them. A Neon branch forks all four together: Postgres + data, object storage + files, functions (each branch gets its own URL), and the AI gateway. I proved it: branched a project with a DB, a bucket of files, a function, and the gateway. The branch came up with a copy of the rows, a copy of the files on its own storage endpoint, its own function URL, and the gateway. A write to the branch left main untouched, and deleting the branch removed all of it. That makes a branch a real environment: true preview stacks, whole-state bug reproduction, and disposable sandboxes for agent

"The ISS radiators are expensive and heavy. We're focused on making them cheap and light."

On 9 July 2026 the head of Windows published a post about AI-powered vulnerability discovery. One line in it was a warning to customers: "As AI helps defenders discover more issues, customers will see a higher volume of security updates included in each security release." It does not say how much higher. The post runs about 1400 words and contains no numbers at all. Five days later Microsoft shipped the July package: 1150 CVEs. The number Microsoft would not put in the blog post is sitting in Microsoft's own API. The Security Update Guide publishes every monthly package as machine-readable CVRF, acknowledgments included, no key required. So I pulled twelve months of it and did the arithmetic. What the data says I sampled eight months before the ramp and four after it. Month CVEs Month CVEs 2024-07 454 2026-04 737 2025-01 343 2026-05 991 2025-04 374 2026-06 1281 2025-07 527 2026-07 1150 2025-10 427 2026-01 310 2026-02 169 2026-03 460 The eight pre-ramp months average 383 CVEs. July 2026 is 1150, so the package is 3,0 times the old normal. The baseline broke in April and peaked in June at 1281. April to July inclusive is 4159 CVEs. At the old rate that is 10,9 months of output, delivered in four. The number I am not going to use February 2026 had 169 CVEs. It is the lowest month in two years, less than half the baseline. Divide July by February and you get 6,8 times, which is a much better number for a headline. I am not using it, because choosing your denominator is how honest people produce dishonest numbers. February is an outlier, and the only reason to anchor to it is that it flatters the story. The real multiplier is 3,0. It does not need help. It is not noise The obvious objection is that volume without quality is just a bigger pile. If AI were generating low-value findings that got patched anyway, the severity distribution would sag. It did the opposite. Measure 2025-07 2026-07 CVEs 527 1150 CVSS median 6,5 7,5 CVSS mean 6,47 7,26 CVSS 7,0 and above 48,0 % 71,

Twelve pricing tabs open. Neon, Hetzner, Supabase, Prisma, Scaleway, OVH. My database is half a gigabyte. I was comparing ten-terabyte price curves. At some point this week I typed the words "I am super lost here" about my own infrastructure. I advise companies on this exact class of decision. That sentence still came out of my hands. If you have ever spent an evening deep in provider pricing pages for a workload that fits on a USB stick from 2009, this one is for you. All numbers below come from the live pricing pages as of July 2026. Rates move, so verify before you commit. Three fears, all pointed at the wrong layers I went in worried about getting attacked, running out of space, and being locked in. All three dissolved under ten minutes of honest reading. DDoS lands on the website edge, not the database. My site already sits behind Cloudflare and Vercel, and a database is never publicly exposed. Only the app talks to it. Whichever provider I picked, that attack surface stayed identical. Here is the shape of the stack, and where each fear lives. MANAGED (what I run today) visitors ──> Cloudflare edge ──> Vercel app ──> managed Postgres [DDoS absorbed] [stateless] [never public, app-only access, provider patches, provider backups, provider on-call] SELF-HOSTED (the alternative I priced) visitors ──> Cloudflare edge ──> Vercel app ──> Hetzner CAX11 [DDoS absorbed] [stateless] [Postgres :5432 firewalled to app, SSH hardened, fail2ban + auto- patching = MINE] │ pg_dump every 6h ▼ encrypted ────> Cloudflare R2 [off-site copies] Same edge, same app, same attack surface. Everything in the right-hand box is what changes owners. Storage was a rounding error. My data is 0.5 GB. Even the cheapest self-hosted box includes 40 GB, eighty times headroom before the first extra cent. Lock-in was a phantom too. Managed Postgres is still stock Postgres. Exiting means a dump, a restore, and one connection string change in the deployment environment. Minutes of cutover, no rewrite an

Design + Product Thinking: NYC’s Path to Reliable AI AI delivers value when it’s useful, trusted, and operational. For city services that affect millions, those qualities don’t happen by accident — they come from applying design thinking (who the service is for, how it’s used) together with product thinking (what outcome we’re trying to achieve and how we operate over time). This article explains why hiring designers and product managers matters for NYC’s digital and AI initiatives, summarizes the city’s PIT Crew program, and outlines how Flamelit applies outcome-focused delivery in the public sector. Why design and product roles matter Designers and product managers have distinct but complementary responsibilities that reduce common AI delivery failures: Designers (Design Thinking): center human needs, prototype user flows, and validate that interfaces and decision workflows are understandable and accessible. They surface usability and trust issues early, preventing technically accurate models from becoming unusable in practice. Product managers (Product Thinking): define the measurable outcomes, prioritize use cases, align stakeholders, and manage the lifecycle from discovery to ongoing operations. They ensure work is evaluated against mission impact, not just technical metrics. Together they prevent common failures: building technically impressive models that nobody trusts, deploying brittle systems without human review, or shipping features with unclear ownership that decay in production. PIT Crew and NYC hiring context NYC’s PIT Crew program is a city initiative designed to attract and staff product, engineering, and design talent for public service projects. It’s a practical recognition that public-sector digital transformation needs people skilled in user research, product management, and delivery. Read more about the PIT Crew and how it works here: https://www.nyc.gov/content/pitcrew/pages/ (open in a new tab). Hiring programs like PIT Crew help create the c

SQLite Internals: lcd-ex vs hctree; PostgreSQL 19 SQL/PGQ Rewrites & pg_timetable Migration Today's Highlights This week's highlights feature a deep dive into SQLite's internal data structures, offering insights for advanced optimization. Also, PostgreSQL users gain practical guidance on migrating to pg_timetable for robust job scheduling and understanding how SQL/PGQ translates to efficient joins in PostgreSQL 19. Replacing pgAgent with pg_timetable: Installing as a Linux Service (Planet PostgreSQL) Source: https://postgr.es/p/9pE Regina Obe presents a crucial guide for PostgreSQL administrators looking to modernize their task automation by replacing pgAgent with pg_timetable . This second part of the series focuses specifically on the practical steps of installing and configuring pg_timetable as a systemd service on Linux, ensuring it runs reliably in a production environment. The article details the process from downloading binaries and creating dedicated user accounts to setting up service files and enabling autostart, providing a comprehensive walkthrough for seamless integration. pg_timetable offers significant advantages over pgAgent , including advanced scheduling capabilities, event-driven task execution, parallel job processing, and improved logging. This migration strategy is vital for enhancing the robustness and efficiency of database maintenance, data synchronization, and complex ETL pipelines within the PostgreSQL ecosystem. By following this guide, developers and DBAs can transition to a more powerful and flexible job scheduler, leading to greater control and reliability over their automated PostgreSQL operations. Comment: Migrating to pg_timetable from pgAgent is a significant step forward for job scheduling in PostgreSQL. This guide provides the hands-on steps needed to get it running as a service, which is essential for any production deployment. SQLite Forum Discusses lcd-ex vs hctree (SQLite Forum) Source: https://sqlite.org/forum/info/3494bff42

The Quest Begins (The "Why") I still remember the first time I stared at a LeetCode problem that asked me to count the number of islands in a grid. My initial instinct? Run a BFS/DFS from every unvisited land cell, mark everything reachable, and repeat. It worked, but each query felt like I was re‑exploring the same territory over and over again—like walking the same hallway in a dungeon every time I wanted to open a new door. Then a friend tossed me another problem: “Given a list of friendships, tell me if two people are in the same social circle.” Again, the naive solution was to rebuild the whole graph for every query. I felt like I was stuck in a grind‑fest, repeating the same low‑level work while the real challenge—understanding the structure of the connections—remain. That frustration sparked a question: Is there a way to remember what we’ve already discovered about connectivity, so future queries are instant? The answer, as many of you have guessed, lives in a humble but mighty data structure called Union‑Find (also known as Disjoint Set Union, DSU). The Revelation (The Insight) At its core, Union‑Find is about two simple ideas : Each element starts in its own set – think of every person as a lone adventurer. When we learn that two elements belong together, we merge their sets – we call that a union . The magic isn’t just in merging; it’s in how we find the representative (or “root”) of a set later on. If we naïvely walked up a chain of parents every time, we could end up with O(n) per find—still a grind. Two optimizations turn this into near‑constant time: Union by rank (or size) – always attach the smaller tree under the root of the larger one. This keeps the overall tree shallow, guaranteeing that the height never exceeds log n. Path compression – during a find operation, we make every node we pass point directly to the root. It’s like handing every traveler a map that instantly shows the shortest route to the campfire, so next time they don’t need to trek

RocheDB v0.5.0 has been released. Release: github.com/puffball1567/rochedb/releases/tag/v0.5.0 RocheDB is an open-source NoSQL document and vector store written in Nim. The project is built around one idea: Data locality should be part of the database model, not only an accidental result of indexes, caches, or application code. A lot of database performance discussions start with indexes, query syntax, or caches. Those are important. But layout often decides whether a system is working with the hardware or asking it to fight back. RocheDB v0.5.0 is a step toward making locality explicit at three levels: logical placement with rings; related-data retrieval with stellar locality; physical layout visibility with WAL locality reporting. Ring locality RocheDB uses a ring as a semantic and structural placement unit. An application, import rule, or operator chooses a ring when writing data: users/123/profile users/123/orders shops/1123/orders docs/japan/support tenant/acme/orders/2026 That ring is not only a directory-like label. It is a coordinate in the retrieval space. When a request already knows its natural locality, RocheDB can open that local region first instead of scanning unrelated records and filtering later. roche put --ring = docs/japan --payload = '{"title":"Refund guide","status":"draft"}' --codec = json roche get --ring = docs/japan --filter = '{"status":"draft"}' --selection = '{ title }' The important part is not the string syntax. The important part is that placement and retrieval scope are connected. Short version: a ring is both where data is placed and where retrieval can start. Not only point reads Point reads are important, but many real applications need related data. For example, a user page may need profile data, orders, billing information, and support metadata. In a relational database, that often becomes joins. In a document database, it often becomes manual denormalization or multiple application-side reads. RocheDB v0.5.0 adds a locality mec

JavaScript ships with Array , Set , and Map — but nothing that keeps its elements sorted as you insert. If you've ever built a leaderboard, an order book, or anything that answers "give me the items between X and Y", you know the workaround: push into an array and .sort() after every insertion. It works, until scale punishes you — you're paying O(n log n) over and over for data that was already 99.9% sorted. Python solved this years ago with sortedcontainers , built on an elegant "list of lists" design instead of balanced trees. I just published sorted-collections , which brings that idea to TypeScript — with full credit to the original as its inspiration. What you get SortedList, SortedSet, SortedMap — always sorted, no manual re-sorting, range queries built in. O(log n) insertions, O(√n) positional access via sqrt-decomposition into buckets. Zero runtime dependencies , ~2KB gzip, types included, dual ESM/CJS. Package quality gated in CI with publint , arethetypeswrong , and size-limit . The API in 30 seconds import { SortedList , SortedSet , SortedMap } from " sorted-collections " ; // SortedList: stays sorted on every insert const list = new SortedList ([ 5 , 1 , 4 , 2 , 3 ]); list . add ( 0 ); console . log ([... list ]); // [0, 1, 2, 3, 4, 5] console . log ( list . at ( 2 )); // 2 — positional access on sorted order // SortedSet: no duplicates, plus set algebra const a = new SortedSet ([ 1 , 2 , 3 , 4 ]); const b = new SortedSet ([ 3 , 4 , 5 ]); console . log ([... a . intersection ( b )]); // [3, 4] // SortedMap: keys always in order, range queries built in const prices = new SortedMap < number , string > ([ [ 104.5 , " order-3 " ], [ 99.2 , " order-1 " ], [ 101.0 , " order-2 " ], ]); for ( const [ price , id ] of prices . irange ( 100 , 105 )) { console . log ( price , id ); // 101.0 order-2, then 104.5 order-3 } Custom comparators are fully typed: number and string get natural ordering for free; for your own types, TypeScript requires a comparator at compile

New York has become the first state to temporarily halt approval of large data centers, as Gov. Kathy Hochul argues the AI-driven building boom shouldn’t come at the expense of higher electricity costs, water supplies, or local control.

New York’s data center moratorium may become the blueprint for anti-AI movement.

Meta recently open-sourced Brain2Qwerty v2, a noninvasive Brain–Computer Interface (BCI) that can decode sentences from thoughts using electroencephalography (EEG) or magnetoencephalography (MEG) signals from the brain. In evaluations, the system achieved a word accuracy rate 61% on average, compared to 8% for other non-invasive methods. By Anthony Alford