标签：#data

Everything from 1.4.1 through 1.4.9 happened inside a single function, standard_planner() . Building paths, costing them, searching for a join order, estimating cardinality from statistics: all of it runs inside that one function. Yet PostgreSQL does not call standard_planner() directly. It puts another function, planner() , one step ahead of it, and has planner() call standard_planner() . And planner() can be made to call some other function instead of standard_planner() . That replacement is what the planner hook enables. When pg_stat_statements measures per-query planning time, or pg_hint_plan rewrites a plan according to hints, it all goes through this hook. Let's look at how PostgreSQL provides a way to observe or change planning behavior without touching a single line of the core, and how external code plugs into it. All planner() does is check the hook The body of planner() is essentially this. if ( planner_hook ) result = ( * planner_hook ) ( parse , query_string , cursorOptions , boundParams ); else result = standard_planner ( parse , query_string , cursorOptions , boundParams ); planner_hook is a global function pointer. Its default value is NULL , in which case standard_planner() is called right away. A plain PostgreSQL build always takes this path: planner_hook is empty, so the incoming query goes straight to standard_planner() . The key here is the type of planner_hook . typedef PlannedStmt * ( * planner_hook_type ) ( Query * parse , const char * query_string , int cursorOptions , ParamListInfo boundParams ); This signature is identical, down to the character, to that of planner() and standard_planner() . It takes the same Query and returns the same PlannedStmt (the execution plan). So external code only has to write a planner function matching this type and store its address in planner_hook . Let's call this function, written by external code to register in planner_hook , a custom planner function. The moment its address is stored, every planning reque

In the earlier posts of this series, we looked at practical query tuning tips and how to read and interpret query plans . A recurring theme in both was: "add an index here." But "add an index" is a bit like saying "use the right tool" — the interesting part is which one. PostgreSQL ships with several index types, each tuned for a different kind of data and query. Picking the wrong one means PostgreSQL quietly ignores your index and goes back to a sequential scan. In this post, we'll walk through the main index types, when each shines, and the special index variations (composite, partial, covering, expression) that often matter more than the type itself. Setting the Scene: Schema and Sample Data We'll reuse the same schema from the previous posts, with one small addition — a metadata JSONB column and a tags array on orders , so we can explore the more exotic index types. CREATE TABLE customers ( id SERIAL PRIMARY KEY , customer_name VARCHAR ( 255 ), email VARCHAR ( 255 ), created_at TIMESTAMPTZ DEFAULT NOW () ); CREATE TABLE orders ( id SERIAL PRIMARY KEY , customer_id INT REFERENCES customers ( id ), order_date TIMESTAMPTZ DEFAULT NOW (), total_amount NUMERIC ( 10 , 2 ), status VARCHAR ( 20 ), tags TEXT [], metadata JSONB ); -- Insert sample customers INSERT INTO customers ( customer_name , email ) SELECT 'Customer ' || i , 'customer' || i || '@example.com' FROM generate_series ( 1 , 1000000 ) AS s ( i ); -- Insert sample orders INSERT INTO orders ( customer_id , order_date , total_amount , status , tags , metadata ) SELECT ( RANDOM () * 1000000 ):: INT , NOW () - interval '1 day' * ( RANDOM () * 365 ):: int , ( RANDOM () * 500 + 20 ), ( ARRAY [ 'pending' , 'shipped' , 'delivered' , 'cancelled' ])[ FLOOR ( RANDOM () * 4 + 1 )], ARRAY [( ARRAY [ 'gift' , 'priority' , 'fragile' , 'bulk' ])[ FLOOR ( RANDOM () * 4 + 1 )]], jsonb_build_object ( 'channel' , ( ARRAY [ 'web' , 'mobile' , 'store' ])[ FLOOR ( RANDOM () * 3 + 1 )]) FROM generate_series ( 1 , 1000000 ) AS s ( i

I've been working on ctrodb — a client-side database for TypeScript that runs in the browser (IndexedDB) and Node.js (in-memory). Zero runtime dependencies. It started as a personal project to stop rewriting IndexedDB wrappers. Every new client-side app needed the same boilerplate: open a connection, create object stores, handle version upgrades, write CRUD helpers. After the sixth time, I wrote it once and got it right. What it does ctrodb gives you MongoDB-like CRUD with schema validation at write time: import { Database } from " ctrodb " const db = new Database ({ name : " my-app " , schema : { version : 1 , collections : { notes : { fields : { title : { type : " string " , required : true }, body : { type : " string " }, pinned : { type : " boolean " , default : false }, tags : { type : " array " , items : { type : " string " } }, createdAt : { type : " string " , default : () => new Date (). toISOString () }, }, indexes : [{ field : " createdAt " }], }, }, }, }) await db . connect () const notes = db . collection ( " notes " ) const note = await notes . create ({ title : " Hello " , body : " World " }) const results = await notes . query () . where ( " pinned " , true ) . sort ({ createdAt : " desc " }) . limit ( 10 ) . fetch () Every record is a Model — a Proxy wrapper with typed field access. note.title works. note.update() handles writes. Direct property assignment logs a warning telling you to use .update() instead. What's included The core package ships with three plugins: Full-text search — inverted index, stop word removal, auto-indexed on create/update/delete Relations — has_many, belongs_to, has_one with lazy accessors built into every Model and eager loading via .with() Custom validation — extendable rules beyond the built-in validators (email, URL, regex) Plus React hooks (separate import, same package): import { DatabaseProvider , useQuery , useMutation } from " ctrodb/react " Signal-based reactivity. When data changes, useQuery re-fetches and your

This guide covers three core areas of daily GBase 8a operations: tracking data loads and collecting error details, configuring audit logs and analysing slow queries, and hierarchically tuning memory parameters. It also provides a standard daily and weekly inspection checklist for your gbase database . 1. Data Load Monitoring 1.1 Load Methods GBase 8a supports two main load methods: gload for large‑scale offline imports (recommended), and LOAD DATA INFILE for single‑file loads with MySQL‑like syntax. 1.2 Checking Load Progress Monitor running and historical loads through system tables: -- Currently executing load tasks SELECT task_id , table_name , status , start_time , loaded_rows , error_rows , TIMESTAMPDIFF ( SECOND , start_time , NOW ()) AS elapsed_sec FROM gclusterdb . load_task WHERE status IN ( 'RUNNING' , 'PENDING' ) ORDER BY start_time DESC ; -- Last 50 load history records SELECT task_id , table_name , status , start_time , end_time , loaded_rows , error_rows , TIMESTAMPDIFF ( SECOND , start_time , end_time ) AS duration_sec FROM gclusterdb . load_task ORDER BY start_time DESC LIMIT 50 ; 1.3 Retrieving the Last Load Task ID SELECT @@ gbase_loader_last_task_id ; Then query error details with that ID: SELECT * FROM gclusterdb . load_error_log WHERE task_id = 'your_task_id' LIMIT 100 ; 1.4 Error Data Collection Enable error collection in the gcluster configuration file ( gbase.cnf ) for production: gbase_loader_logs_collect = ON 1.5 Load Performance Parameters Parameter Scope Description Recommended gcluster_loader_max_data_processors gcluster Max concurrent load processing threads CPU cores / 2 gcluster_loader_min_chunk_size gcluster Chunk size sent to gnode (bytes) 64 MB gbase_loader_parallel_degree gnode Parallel write threads on gnode 4 – 8 gbase_loader_buffer_count gnode Number of load buffers 4 2. Audit Log Configuration and Analysis 2.1 Enabling Audit Logs Configure in both gcluster and gnode gbase.cnf files: audit_log = ON log_output = FILE # or TABLE

Dalla libreria embedded che ha invaso ogni dispositivo a un’implementazione moderna con concorrenza, async I/O e vector search: cosa cambia davvero per chi sviluppa app. Nel frontend e nel full‑stack capita spesso di parlare di database come servizi: Postgres gestito, cluster, repliche, connessioni, pooling, credenziali e una lunga lista di “cose che possono rompersi”. Ma esiste un’altra filosofia, più vicina all’idea di “dipendenza” che di “infrastruttura”: un motore SQL che vive dentro l’applicazione. Questa è la ragione per cui SQLite è ovunque. È una libreria, non un server. Legge e scrive su un singolo file su disco. Riduce drasticamente configurazione, porte, processi separati e complessità operativa. Ed è proprio questa semplicità a renderla una delle fondamenta silenziose dell’informatica moderna: la usi in browser, smartphone, desktop app, tool CLI, IoT… spesso senza nemmeno accorgertene. Ora immagina di riscrivere tutto da capo, in Rust, cercando di essere compatibile al 100% e allo stesso tempo più “moderna”. Sembra un’idea folle per definizione—finché non inizi a guardare ai limiti pratici che oggi emergono in molte applicazioni. Perché toccare SQLite, se funziona così bene? SQLite non è “il problema”. Anzi: è considerata estremamente robusta perché è conservativa, minimalista, e custodita con un rigore quasi maniacale. Il punto è un altro: il suo modello di sviluppo e manutenzione è atipico rispetto a quello che molti intendono per open source collaborativo . Il codice è disponibile e utilizzabile liberamente, ma l’evoluzione è guidata da pochissime persone e—di fatto—non segue la dinamica classica delle contribution esterne. Questa scelta ha un effetto collaterale positivo: riduce il rischio di regressioni introdotte da cambiamenti non coerenti con la visione del progetto. Ma ha anche un costo: se la tua azienda o il tuo prodotto hanno esigenze nuove (concorrenza più spinta, I/O non bloccante, funzionalità specifiche), “aspettare che arrivi upstream” n

Using Claude Fable 5 or Mythos 5 on Amazon Bedrock requires opting into provider_data_share, sending prompts and outputs to Anthropic for 30-day retention with human review. Previous Bedrock models kept inference data inside the AWS boundary. Three days after launch, Anthropic asked AWS to revoke access to both models citing US export control compliance. By Steef-Jan Wiggers

The AI agent had one job: pay approved vendor invoices, so the finance team could stop doing it by hand. On a Tuesday morning, it picked up invoice #4471 from a freight vendor Ksh48,000, stamped Approved in the company's ERP, cleanly matched to a valid purchase order. The agent checked the things it was told to check. They all passed. It paid the invoice. The invoice had already been paid. The previous Thursday. By a member of the finance team. Here is what the company's systems believed that morning and none of them was wrong. The ERP said: Approved. Unpaid. The reconciliation job that pulls in bank activity runs overnight, and last night it had failed silently. So the ERP's picture of the world was simply four days stale. The bank feed said: Paid. Last Thursday. It was right. Nobody had told the ERP. A Slack thread said: "hold everything to this vendor they double-billed us last quarter, I'm sorting it out with their AP team." Posted by the accounts-payable lead. Three days earlier. Resolved in her head, and nowhere else. The vendor's own email said: "Payment well received, thank you!" referring, of course, to Thursday's payment. The agent's inbox reader had seen it that morning, then set it aside, because email ranked below the ERP and the two disagreed. Every system was internally consistent. Every system was the authority on something . And there was no system anywhere not one that could answer the only question that actually mattered: has invoice #4471 been paid? A human clerk would almost certainly have caught it. Not because a clerk is smarter than the model they're not. Because a clerk would have felt the friction. They'd have half-remembered cutting the check. Or scrolled past the Slack message that morning and hesitated. Or simply had the reflex to ping someone before sending $48,000 out the door. Reconciling systems that quietly disagree is most of what operations people actually do all day so much of it that nobody files it under "work." It's just judgm

Stop Fighting Scikit-Learn Pipelines: How Metadata Routing Fixes Sample Weights & Groups A couple of months ago, I stumbled upon this video by Vincent D. Warmerdam about metadata routing in scikit-learn. I'll be honest, I had no idea what "metadata routing" even meant, but Vincent's explanation completely changed how I think about building ML pipelines. The video showed me that one of the most frustrating problems in scikit-learn; passing sample weights and groups through complex pipelines finally had an elegant solution. It piqued my curiosity enough that I dove deep into the feature, tested it extensively, and honestly, I was surprised by how little coverage this gets in technical blogs and articles. So I figured, why not write about it myself and share what I learned? If you've ever struggled with imbalanced datasets, grouped cross-validation, or just wanted to pass custom information through your pipelines, this article is for you. Let's start from the very beginning. What is "Metadata" in Machine Learning? Let's start with a concrete example. You're building a credit card fraud detection model with this data: # Your training data X = transaction_features # Amount, merchant, time, location, etc. y = is_fraud # 0 = legitimate, 1 = fraud # But you also have additional information: sample_weights = [ 1.0 , 1.0 , 10.0 , 1.0 , ...] # Fraud transactions weighted 10x customer_ids = [ 101 , 102 , 101 , 103 , ...] # Which customer made each transaction Metadata is the "extra information" beyond your features (X) and labels (y): sample_weight : How important is each transaction? (Fraud = 10x more important) groups : Which customer does each transaction belong to? (For proper cross-validation) Custom metadata : Transaction timestamps, confidence scores, data quality flags, etc. Why Metadata Matters: The Credit Card Fraud Problem Imagine you're building a fraud detection system for a financial company. You have: Imbalanced data : 99% legitimate transactions, 1% fraudulent T

OpenAI’s Bonnie Xu discusses Kepler, an internal AI data analyst agent built to query 600+ petabytes of data. She explains how they overcome context window limits using MCP, automated code crawling, and RAG. Xu also shares how their team leverages scoped semantic memory for self-learning and utilizes AST-based LLM grading to build a robust, regression-free evaluation pipeline. By Bonnie Xu

CircleCI has launched Chunk Sidecars, a new capability designed to bring CI-style validation directly into an AI coding agent's inner development loop By Craig Risi

Most cloud systems make one authorization decision at login. Everything after runs on trust established at authentication time. For systems handling regulated data, that gap is where breaches happen. This article presents a continuous authorization architecture covering risk-tiered evaluation, behavioral baselines, privacy-preserving audit trails, and a phased and incremental rollout. By Venkata Nedunoori

In a new Windows Developer Blog post titled "Windows platform security for AI agents", Microsoft positions Windows as the trustworthy operating system for autonomous agents and introduces the Microsoft Execution Containers (MXC) SDK as the core of that strategy. The post argues that containment, identity and manageability must be built into the operating system. By Matt Saunders

ClickHouse HTTP API: A Complete Beginner's Guide Introduction When most people think about interacting with a database, they usually imagine connecting through a database client or application. However, ClickHouse also provides a simple and powerful HTTP API that allows you to query and manage your database using standard HTTP requests. The ClickHouse HTTP API provides a universal interface for communicating with your ClickHouse server. Since almost every programming language and automation tool supports HTTP, it becomes an excellent choice for integrations, monitoring, scripting, and lightweight applications. In this guide, you'll learn what the ClickHouse HTTP API is, why it's useful, and how to perform common database operations using simple HTTP requests. What Is the ClickHouse HTTP API? The ClickHouse HTTP API is a built-in interface that enables clients to communicate with a ClickHouse server using the HTTP protocol. Instead of connecting through the native TCP protocol, you simply send HTTP requests and receive responses in formats such as JSON, CSV, TSV, XML, or plain text. The HTTP interface is: Language agnostic Easy to integrate with web applications Firewall friendly Simple to test using tools like cURL, Postman, or a web browser Because of its simplicity, the HTTP API is widely used for automation, dashboards, data pipelines, and monitoring systems. Why Use the HTTP API? The ClickHouse HTTP API offers several advantages: No dedicated database driver is required. Works with virtually every programming language. Easy integration with REST-based applications. Supports multiple output formats. Ideal for automation and scripting. Perfect for cloud-native applications and microservices. Common Operations Using the HTTP API, you can: Execute SQL queries Insert data Create and modify tables Retrieve query results Export data in different formats Automate database operations Authentication Options ClickHouse supports multiple authentication methods when using th

DuckDB 1.4.5 LTS, pgEdge ColdFront Beta, and SQLite's FCNTL_PDB Internals Today's Highlights This week's highlights feature the latest DuckDB 1.4.5 LTS release, a new open-source beta for PostgreSQL data tiering, and a deep dive into an obscure SQLite internal file control operation. These updates offer performance, architectural flexibility, and internal insights across the SQLite ecosystem. Announcing DuckDB 1.4.5 LTS (Andium) (DuckDB Blog) Source: https://duckdb.org/2026/06/17/announcing-duckdb-145.html The latest Long Term Support (LTS) release of DuckDB, version 1.4.5 named "Andium", has been announced, primarily focusing on bugfixes and performance enhancements. DuckDB, an in-process analytical processing database, continues to refine its engine for enhanced stability and efficiency in embedded and edge computing environments. While the announcement is concise, LTS releases are crucial for developers and organizations relying on a stable and well-tested version for their data pipelines and analytical workloads, ensuring long-term compatibility and reliability. This update is vital for maintaining the robustness of applications that utilize DuckDB for local data transformations, complex analytical queries, and other high-performance data operations. Users of previous 1.4.x versions are encouraged to upgrade to benefit from the accumulated stability improvements and minor speedups, all without introducing major breaking changes. This commitment to incremental improvements and stable releases solidifies DuckDB's position as a premier solution for embedded analytical database needs, making it a reliable choice for critical projects. Comment: An LTS release, even with bugfixes, is always welcome from DuckDB. It reinforces their commitment to a stable and performant analytical database that I frequently use for local data processing and reporting. Introducing ColdFront: Seamlessly Uniting OLTP, Analytics and AI Workloads on PostgreSQL (Planet PostgreSQL) Source: htt

FERC told grid operators to give data centers a fast lane for interconnections, but it failed to address electricity supply shortages.

A data breach involving government-issued ID documents affects over three million people in Texas.

The software engineers filed a complaint with Seattle’s civil rights office accusing Amazon of illegally retaliating against them for expressing their personal political beliefs.

Vinay Chella and Akshat Goel discuss the challenges of running traditional CDC across heterogeneous databases during peak order traffic. They explain how Debezium hit limits under high load and share how they built Write-Ahead Intent Log (WAIL) - a custom architecture that utilizes a dumb producer proxy and a smart consumer pattern to cleanly separate the intent from the state payload. By Vinay Chella, Akshat Goel

pgrac is an open attempt to rebuild Oracle RAC's core machinery (shared-everything storage, multiple active nodes all writing one database, a cluster-wide change number) on top of PostgreSQL 16. Build log #1 laid out the four problems that fight back. This one is about the problem that turns a node failure into silent data corruption, and the first, deliberately modest, layer pgrac ships against it. The failure mode In a shared-nothing cluster an evicted node is mostly harmless: it owns its own disks, so the cluster routes around it. In a shared-everything cluster the same event is dangerous, because every node writes the same storage. Picture the classic split: node 2 misses heartbeats, the cluster declares it dead and remasters its work elsewhere, but node 2 is not actually dead. It is frozen on a long GC pause, or its interconnect NIC flaked, and it is about to wake up and finish the write it started. Now two nodes believe they own the same blocks, and shared storage will accept both writes. That is not a crash. It is corruption you find three days later. Oracle RAC's answer is I/O fencing: before remastering a dead node's resources, you make certain it can no longer touch the storage, with STONITH, SCSI-3 persistent reservations, or a hardware watchdog. The node is fenced at a layer below its own software, because the whole point is that you cannot trust the dead node's software to behave. That hardware layer is real work, and it is not what pgrac built first. What it built first is the layer above it: an in-process cooperative write-fence, now default-ON. The rest of this is precise about what that does and does not buy you, because "we have fencing" is the kind of claim that is worth less than nothing if it is overstated. A fence needs an authority everyone can agree on You cannot fence on local opinion, because the whole problem is that the dead node disagrees about being dead. Authority has to live on durable, shared, quorum-backed storage. pgrac writes a sm

Microsoft recently introduced at Build 2026 Microsoft Scout, an always-on agent. Scout belongs to a new category of agents Microsoft called Autopilots: always-on agents that work autonomously on a user’s behalf with their own identity, without needing to be prompted each time. Microsoft Scout integrates with Work IQ and is based on the open-source agent framework OpenClaw. By Bruno Couriol