标签：#data

Opinion: Your Tests Can't See What a Migration Destroys — Dry-Run It on a Clone A green test suite is the wrong tool for judging an AI-generated migration, because tests run against the post-migration schema and never observe the intermediate states where data disappears. The up migration is the visible artifact that gets reviewed, while the down migration is treated as an afterthought even though it is the only safety net when the deployment goes wrong. Free model access makes the problem structural: generation cost drops to zero, so migration volume rises, and every additional migration multiplies the surface for unreviewed data loss. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Tests validate the destination, not the journey When a test suite runs against a migrated database, it confirms that the application can read the new schema, but it cannot confirm that the migration preserved the data it was supposed to preserve. The test runner connects after the migration has executed, so it never sees the moment when a column is dropped, a table is renamed, or a constraint is silently relaxed. A migration that passes every test can still destroy production data, because the tests were designed to validate application behavior, not migration safety. The standard mitigation is a staging database, but staging is a poor substitute for a dry run because it has different data, different volume, and different usage patterns. The dry run I recommend uses a clone of the production schema with a representative data sample, and it exercises both directions of the migration with data integrity checks at every step. The clone does not need to be large; a few thousand rows per table is enough to expose most destructive patterns. The dry-run workflow in five steps The workflow is deliberately mechanical, because the goal is to remove judgment from the verification process and reserve human attention for the migration's intent: Clone the schema and lo

I thought that when i join Lux Dev i would jump straight into building complex data pipelines and getting to understand kafka, kafka sounds like a really cool name, but if there's one thing I'm realizing quickly, it's that before you can orchestrate complex data pipelines or deploy web scrapers, you have to master the absolute basics of version control. This week, I was working on setting up a new local project, a health records analysis and pushing it to GitHub entirely through the command line. If you're just starting out with version control, here is exactly how I took a project from a completely blank folder on my desktop to a live repository on GitHub, including testing SSH keys. Setting Up the Local Project First, I needed a place for my project to live. I opened my bash terminal, navigated to my Desktop using he cd command, and created the main project folder along with a sub-folder for the data named Data. cd Desktop mkdir -p Kenya_Hospital_Health_Records_Project/Data cd Kenya_Hospital_Health_Records_Project With the directories created, I copied and pasted my Kenya_Hospital_Health_Records_Project.csv data set we were given in class into the Data folder. Writing the README via Terminal Instead of opening a text editor, I decided to build out my README.md right from the command line using echo command. The > operator adds new text the file, while >> adds text to the already creaed line. echo "# KENYA HEALTH RECORDS ANALYSIS" > README.md echo "## Project Overview" >> README.md echo "This project analyses health records of a hospital" >> README.md I also added a quick list of tools and challenges using the same method and used the cat README.md command to print the contents of the file directly in the terminal to confirm that everything looked right. Initializing and Staging Now it was time to turned this folder into a tracked Git repository. git init Running git status showed that my Data/ folder and README.md were untracked. To stage them for my first commit,

Short version of a post on gex.live/research ; the full write-up, definitions and reproduce block live there. Most published dealer-gamma numbers are built from open interest : yesterday's outstanding contracts, multiplied by a convention about who holds which side. Whether the convention is right is a separate question. The prior question is simpler: how much of what trades today was already in that book this morning — and how much of tomorrow's book is being created today? Open interest and volume are enough to answer it, with no assumption about who bought. Sample: SPX and SPXW, 2022-04-14 to 2026-08-14, 1,081 trading days, every expiry within about a month (0DTE plus the 21 nearest), 8.6 million contract-days, 4.3 million with volume. Definitions Per contract (expiry, strike, right) and session D: OI(D) is open interest at the start of D, OI(D+1) at the start of the next session, ΔOI = OI(D+1) − OI(D) , vol the day's volume in that contract. |ΔOI| / vol is a lower bound on how one-sided the day's trading in that contract was — 1.0 means every lot opened (or every lot closed), 0 means opens and closes cancelled. Contracts expiring on D have no next-day OI and drop out of the ΔOI statistics; 4.1% of rows (3.8% of volume) show |ΔOI| > vol, which is impossible (OI snapshot timing) and are excluded. The book grows by 40% of what trades, every day days to expiry on D net ΔOI / volume |ΔOI| / volume (lower bound on one-sidedness) share of volume in contracts whose OI rose contract-days 1–5 37.8% 41.8% 90.3% 813,013 6–21 42.7% 53.3% 81.1% 2,206,446 22+ 42.8% 57.6% 76.5% 831,896 Across the whole book, net ΔOI is 39.9% of the day's volume on the median session (IQR 36.2–44.0%), positive in every year and every expiry bucket: the SPX book is always being built faster than it is unwound, until expiry does the unwinding. Far expiries are open-and-hold (a day's trading in a 22+ DTE contract is at least 58% one-sided); the nearest expiries churn (42% at 1–5 DTE). Per contract-

An AI assistant will write you a query in ten seconds, the query will run, and the number that comes back will look completely reasonable. This page gives you the five checks that tell you whether that number is right. They take about two minutes, they need no tools beyond the database you already have, and they catch the four mistakes AI-written SQL actually makes. The order matters. The checks are arranged cheapest first, so the first one costs a single row count and the last one costs a short conversation. Most wrong queries fall to the first two. The short version. A query that runs has only passed a grammar check. The number is right when the rows, the filters and the denominator match the question you asked. The database only takes a query as far as the first gate. Why a query that runs can still be wrong Before the list: what do you think the database actually checks when it accepts a query? Grammar. That is the whole list. Spell a table name wrong and you get an error. Sum the wrong column, join in a way that doubles rows, or filter after grouping when the question needed it before, and you get a clean result set with a wrong number in it. Every mistake on this page is valid SQL. AI assistants add one specific difficulty: their queries are fluent. The aliases are tidy, the formatting is clean, and the shape looks like something a careful person wrote. Fluency reads as correctness, and it is not the same thing. Treat an AI query the way you would treat a first draft from a new colleague: with respect, and with the row counts open. The table the examples run on Everything below runs on one small shop dataset, so every number can be checked by hand. Thirteen orders in July, five customers, and a refunds table where two orders were refunded in two parts. Eleven of the thirteen orders are completed; one is refunded, one is pending. There is also a staff_accounts table listing internal accounts, and it contains one NULL row, because real lookup tables usually do.

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you will be running real SQL against a real database with nothing installed, and you will know which of the free browser tools suits which job. You will also know the thing none of them puts on the front page: some of them run entirely inside your browser, and some upload whatever you paste to a stranger's server. That difference decides what you are allowed to practise on. Here is what to actually do today. If you want a database already loaded and questions already written, open sql-practice.com . If you want to create your own tables and share the result with someone, open DB Fiddle . Both start working immediately with no account. The short version: browser-only tools keep your data on your machine, server-backed tools do not, and neither kind is the right place for anything from work. Where the data goes is the one idea that should drive your choice, so it gets the picture. The original carries a diagram here. In words: Two panels side by side, each drawn as a laptop outline containing a browser window. In the left panel a small data box sits inside the browser window, with a short circular arrow looping back into itself, showing the data never leaves the laptop. In the right panel the same data box has a long arrow leading out of the laptop, across a gap, and into a separate server rack drawn beyond the laptop's edge, with a copy of the data box now sitting in the rack as well. The original box remains, showing the data has been copied out rather than moved. Every tool below was opened and checked on 8 August 2026. These sites change often, so the descriptions describe what was actually on screen, and anything I could not confirm by looking is not claimed here. 1. Run your first query, right now Before the explanation: what do you think has to exist on your computer for a SELECT statement to return rows? The honest answer is nothing at all, and that surprises people who have sp

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you will have a real database sitting on your own computer, with 11 tables, 3,503 tracks and 412 customer invoices in it, and you will have run a query that proves every table arrived intact. Then you will run a join across two of those tables, which is the thing a single spreadsheet can never teach you. It takes about five minutes and costs nothing. Here is what to actually do today. Download the Chinook database file, open it in DB Browser for SQLite, and run one query that counts the rows in every table. If the counts match the ones printed below, you have a working practice environment and you can stop shopping for one. The short version: get Chinook_Sqlite.sqlite , open it, count the rows, then join two tables. Northwind and Sakila are the other two names you will see, and there is a table further down saying when each is the right pick. The reason a sample database beats the CSV you already have is one idea, so it gets the picture. The original carries a diagram here. In words: Two panels side by side. The left panel holds a single grid of rows and columns, standing alone with nothing attached to it. The right panel holds four smaller grids arranged around each other. A highlighted column at the edge of each small grid is joined by a solid line to a matching highlighted column on a neighbouring grid, so all four grids are wired together into a connected shape. The left panel has no lines at all, because there is nothing for a line to reach. Every number on this page is real. I downloaded Chinook v1.4.5 and Northwind on 8 August 2026 and ran each query with SQLite 3.51.1. The counts, the outputs and the row multiplication are what came back, not what should have come back. If you have no database software at all yet, how to set up a SQL database is the fifteen-minute version of that step, and this page picks up right after it. 1. Why one CSV is not enough Before the explanation:

Cheap energy, abundant land, and proximity to Beijing have turned a city in Inner Mongolia into a crucial hub for data centers.

Nvidia continues to pour money into data center development — just as AI data centers bring lots of money into Nvidia.

There is a comforting story about how AI models handle the enormous quantities of text and images they are trained on: they do not store any of it, they merely learn general patterns, and once training is done the original data is gone in any meaningful sense. It is a reassuring account, and it is not quite true. Large models memorise fragments of their training data — verbatim, recoverable fragments — and a decade of research has produced reliable ways to detect and extract them. The answer-first version: if your data was in a model’s training set, the model may have memorised identifiable pieces of it, and those pieces can leak. Two families of attack make this concrete. Membership inference works out whether a specific record was in the training data at all. Data extraction pulls memorised content back out word-for-word. Neither is exotic; both are well documented against production systems. This is the mechanism underneath both the newspaper lawsuits alleging near-verbatim reproduction of their articles and the quieter privacy research showing that models leak the people in their training sets. Understanding it is the difference between trusting the comforting story and knowing its limits. Memorisation is a feature of the maths, not a bug Start with why models memorise at all. A large neural network has an enormous number of parameters — enough capacity to do more than compress general patterns. During training it is rewarded for predicting its training data accurately, and one very effective way to predict a specific example accurately is to memorise it. For data that appears once in an unusual form, or many times in an identical form, memorisation is often the path of least resistance for the optimiser. This is measurable. Researchers can show that a model assigns systematically higher confidence, and lower prediction error, to examples it was trained on than to otherwise-similar examples it has never seen. The size of that gap grows with the size of the model

Some Americans fear the FTC may be thinking about personalized pricing all wrong.

There's about to be a big fight to secure access to space.

Mark (aka Winsaucerer) here to show you how you can test your PostgreSQL database like a sorcerer. We are going to be using Spawn, a SQL build system supporting migrations and testing. You do not need to be using Spawn for migrations in order to use it for testing. Spawn does not require any extension installed. All you need is the spawn CLI and a psql connection to the database for Spawn to connect through. Spawn was built to solve some migration pains I've experienced, but I happily discovered that when used for testing, it is very powerful. To show you some of that power, we're going to use a contrived database example. It uses golden file testing to determine success. When the test runs, we capture the stdout and stderr output from psql, and compare that to expected output. Testing with Spawn involves these steps: Create a new test with spawn test new <name> and fill out the test steps Check test outputs with spawn test run <name> (or view the SQL that will be sent to psql via spawn test build <name> ) When outputs are as expected, create the golden file with spawn test expect <name> Run the test and compare to expected output with spawn test compare <name> For now, Spawn only supports connecting via psql, which means that you have access to all the features that psql provides. To get started, follow the Spawn install instructions: Install Spawn And then create a new folder on your system, and initialise a new project with a docker compose config ready for us to play with: # inside your new folder: spawn init --docker docker compose up -d You now have a running docker based PostgreSQL database and a spawn.toml file configured to connect to it. We are not assuming that you are using Spawn or any other tool for migrations, so you can manually create and update the database by connecting directly using psql: docker exec -ti postgres-db psql -U postgres Create the database ⚠️ Caution This post is not intended as an example of how to build an orders database. The des

Five states, one query language, and an "active licenses only" checkbox that only actually filters two of them. That's the trap in scraping US childcare-license open-data registries: Socrata SODA makes every state's API look identical, but "active" is defined — or not defined at all — differently in every dataset. Quick answer New York, Connecticut, Colorado, Delaware, and Texas all publish their childcare-facility registries through Socrata, and all five accept the same $where query syntax. But only NY and CT ship a server-side status filter this Actor can apply. Colorado and Delaware have no status column in the dataset at all — there's nothing to filter on. Texas does have a status column ( operation_status ), it's just not wired into the active-only filter, so toggling activeOnly doesn't touch Texas rows either way. Treating "active only" as a global switch that behaves the same everywhere will silently hand you closed and revoked facilities in three of the five states while you believe you filtered them out. STATE_CONFIGS : dict [ str , StateConfig ] = { " NY " : StateConfig (..., col_status = " facility_status " , active_where = " facility_status= ' Active '" ), " CT " : StateConfig (..., col_status = " status " , active_where = " status= ' ACTIVE '" ), " CO " : StateConfig (..., col_status = None ), # no status column to filter on " DE " : StateConfig (..., col_status = None ), # no status column to filter on " TX " : StateConfig (..., col_status = " operation_status " ), # status exists, filter isn't wired } Why does "active only" do nothing in three states? Because the filter is applied per-state, not globally, and only two states have both a status column and a configured $where fragment for it: async def _fetch_page (...): params = { " $limit " : str ( page_limit ), " $offset " : str ( offset ), " $order " : config . order_key } if active_only and config . active_where : params [ " $where " ] = config . active_where return await _get_with_retry ( session

"Just query the Socrata API" is true and also useless advice. Socrata SODA is a real open standard — New York, Connecticut, Colorado, and Texas all expose their professional-license registries through the same $limit / $offset / $where query language. The standard ends there. What each state puts inside that standard is four unrelated data models wearing the same protocol. Quick answer Every state's cosmetology/barber/salon registry is one giant multi-profession table with its own column names, its own beauty-credential filter, and its own idea of what "active" means — and one state (Texas) doesn't expose a status column at all, so an activeOnly toggle is a silent no-op there. A generic Socrata client that assumes one schema will either miss most of the data or crash on the first state whose columns don't match. The fix is a per-state config object that maps each state's real column names to one canonical output row, with the active-license filter applied only where the underlying data supports it. @dataclass ( frozen = True ) class StateConfig : state : str endpoint : str order_key : str col_business_name : str | None col_licensee_name : str | None col_status : str | None base_where : str | None = None active_where : str | None = None Why does the same query return different professions per state? Cosmetology licenses don't get their own dataset — they're rows buried inside each state's entire professional-licensing table, next to electricians, dentists, and notaries. Filtering has to happen server-side, in SoQL, before pagination even starts, or you're downloading (and paying to store) irrelevant rows. Texas needs a starts_with() match across three license-type prefixes plus an Establishment wildcard; Connecticut needs an exact in() list of six credential names; Colorado needs a four-code in() list: TX_BEAUTY_WHERE = ( " starts_with(license_type, ' Cosmetology ' ) " " OR starts_with(license_type, ' Class A Barber ' ) " " OR starts_with(license_type, ' Barber ' ) "

Top Vector Databases for AI Agents in 2026: Qdrant vs Pinecone vs Weaviate vs PgVector vs Milvus Persistent memory is the foundation that turns a stateless LLM into a continuously improving, autonomous agent. In 2026, selecting a vector database is no longer just about raw Approximate Nearest Neighbor (ANN) speed. For AI agents, the critical requirements have shifted to: Payload & Metadata Filtering : Can you filter by tenant_id , user_id , and timestamp during vector graph traversal without sacrificing recall? Hybrid Search (BM25 + Dense Vectors + Sparse SPLADE) : Combining exact keyword matching (for code symbols and error codes) with semantic understanding. Multi-Tenancy & Memory Namespacing : Safely isolating memory blocks across thousands of users and sessions. Billion-Scale Quantization (Product Quantization & Scalar Quantization) : Slashing RAM costs by 75–90% in production. This guide provides a comprehensive architectural comparison of the top 5 vector databases for AI agents in 2026. Head-to-Head Comparison Matrix Feature / Metric Qdrant Pinecone (Serverless) Weaviate PgVector (PostgreSQL) Milvus Primary Architecture Rust-native, disk-backed Fully managed serverless Go-native, modular RAG PostgreSQL extension Distributed cloud-native Open Source Yes (Apache 2.0) Proprietary SaaS Yes (BSD-3) Yes (Open Source) Yes (Apache 2.0) Payload Filtering Exceptional (HNSW custom payload indexing) Good (Metadata filtering) Strong (Inverted index + HNSW) SQL WHERE clause Strong (Partition keys) Hybrid Search Native (Dense + Sparse vectors) Native hybrid Native BM25 + Vector SQL text search + pgvector Native multi-vector Quantization Scalar & Product Quantization (Binary) Automatic serverless compression PQ, BQ, SQ Halfvec, Binary Quantization Scalar / Product Quantization Best Fit High-performance agent memory & self-hosted RAG Zero-maintenance cloud SaaS GraphQL & multi-modal search Unified relational + vector apps Ultra-large enterprise (100M+ vectors) 1. Qdrant: The

Disclosure: I build Schemity , a desktop ERD tool - this post is from our blog and uses it for the examples. TL;DR: The database has a built-in place to document a column - COMMENT ON COLUMN in PostgreSQL, the COMMENT attribute in MySQL - and almost nobody fills it in, because a sentence of prose has to travel the same path as a schema change: a migration file, a review, a deploy. Schemity keeps field descriptions in the diagram instead, where editing one generates no SQL, reads existing database comments in on import, and exports the result as a data dictionary. You can document a database column without touching the database: write the description in the model rather than in the schema. That sounds like a dodge until you price the alternative. The database's own mechanism for column documentation, COMMENT ON COLUMN in PostgreSQL and the COMMENT attribute in MySQL, sends a sentence of prose down exactly the same path as a change to how data is stored - a migration file, a code review, an approval, a deploy window - and on MySQL it does something worse than that. Schemity keeps field descriptions in the diagram, where editing one produces no SQL at all. This is why so many production schemas have thousands of columns and almost no comments. Not because nobody wanted to write them. Because writing one costs a deploy. How do I document a database column without running a migration? Keep the description in the model rather than in the storage engine. A field description is a fact about what the column means to your team; it changes no type, no constraint, no index, and nothing about what the database will accept. When it lives in the diagram, editing it is like editing a comment in a code file: you change it, review it in the same pull request as everything else, and nothing has to run against production for it to take effect. The moment that description is a column comment, it stops being prose and becomes DDL. Now it needs a migration file, and the migration needs a

Surging demand for AI training data is driving rapid growth for the startup and its rivals.

In the current epoch of Artificial Intelligence, the industry remains singularly preoccupied with the "Model" — obsessing over the raw parameter scales of the latest LLMs or the specific benchmark performance of a new transformer variant. However, at Informatiqs, we shift the lens. We recognize that sustainable enterprise value is rarely derived from the model in isolation; instead, it emerges from the high-stakes architectural decisions and systemic orchestration that define its environment. As we launch our inaugural edition, we dissect a critical technological nexus: the convergence of Graph Neural Networks (GNNs), Generative AI, and the industrial-grade infrastructure of Google Cloud Platform (GCP). We argue that for complex enterprise datasets, the transition from flat vector embeddings in latent space toward non-Euclidean, graph-based relational intelligence is the primary differentiator for the next generation of resilient AI applications. 1. The Scientific Foundation: Exploiting Relational Inductive Bias Traditional Deep Learning architectures, such as Convolutional Neural Networks (CNNs) for images or Transformers for text, primarily operate on data structured as sequences (Euclidean space). While exceptionally powerful, these structures often fail to capture the topological nuances of real-world systems like supply chains, molecular structures, or fraudulent transaction webs where data is inherently non-Euclidean. Graph Neural Networks (GNNs) provide a framework for learning from data represented as nodes and edges. Unlike standard neural networks that process inputs in isolation, GNNs utilize a Message Passing paradigm. In this process, a node's internal representation is iteratively updated by aggregating information from its immediate neighbors. Instead of looking at a data point as a single row in a database, the GNN looks at who that data point "talks to" and how those connections define its identity. By utilizing Graph Attention mechanisms, we can fu

Jason Kelce joked that people should cool data centers with their pee, rather than potable water -- but his suggestion is not completely ludicrous.

I Gave Five Graph Databases 256MB of RAM Each. Here's What Broke. CognoDB Cloud's free tier gives you a graph database instance with half a CPU core and 256MB of RAM. That's not a lot. It's also, honestly, a pretty realistic starting point a lot of real side projects and early-stage products live exactly there, on whatever the free tier happens to give them, and find out the hard way what their database does under pressure. So I decided to actually find out. I took CognoDB and lined it up against four other graph databases Neo4j AuraDB, FalkorDB, and ArangoDB gave every single one of them the same tiny resource budget, threw the same 198,050-edge dataset at all of them, and ran the same queries. No cherry-picking, no "best case" numbers. Just: here's a small VM's worth of resources, go. One of the databases I originally planned to include never even made it into the results. It crashed on startup. Not "slow to start" a full segfault, reproducibly, across two different versions, with nothing I threw at it fixing it. More on that below, because it's honestly one of the more interesting parts of this whole thing. The setup, quickly Five candidates going in: CognoDB (mandatory, since that's the actual point of this), Neo4j AuraDB Free, Memgraph, FalkorDB, and ArangoDB. Same dataset for all of them a real social-graph-shaped dataset from Stanford's SNAP collection, ~18.7k nodes and ~198k edges, sized specifically to fit inside every platform's free tier without anyone getting an unfair advantage. Same queries too: I wrote every single query 1-hop, 2-hop, 3-hop traversals, point lookups, filtered lookups, aggregations exactly once, then translated each one into whatever query language a given platform actually speaks. No platform ever got a "friendlier" version of a query than another. And everyone ran under the same 0.5 vCPU / 256MB RAM ceiling, whether that was their real cloud free tier or a Docker container I capped by hand to match. The one that didn't survive Memgra