标签：#Data

Most "data lineage" you've seen answers a schema question: table B comes from table A , or column B.total comes from columns A.price and A.qty . That's genuinely useful, and tools like OpenLineage do it well. But notice what it doesn't tell you: it says which columns can influence an output. It never says which values actually did . That gap is the whole subject of this post. I built a small, self-contained reference pipeline that captures provenance at the row and field level — "the value in this destination row, this field, was computed from these specific source (row, field) pairs" — and I want to walk through two things: why you'd ever want provenance at that granularity, and why it's genuinely hard once you commit to it. Repo (dbt-core + DuckDB, no server, no cloud, runs on a clean checkout): https://github.com/stevenblough/row-level-prov The one distinction everything follows from Here's the sentence the entire project turns on: Column-level lineage is a schema-sized, static fact you can derive from code. Value-level provenance is a data-sized, dynamic fact you must capture at execution. Put it in complexity terms and the consequences become obvious: Column lineage is O(schema) . It scales with how many columns you have. You can compute it by parsing SQL, offline, without ever looking at a single row. Value provenance is O(rows × fan-in) . It scales with your data volume times how many source values feed each output value. It does not exist anywhere until the query runs, and it can only be captured there , piggybacked on the query that actually produced the values. You cannot "reconstruct" value provenance later by re-querying the sources — the moment the source changes, you'd reconstruct a different answer than what really happened. That single exponent change ( schema → rows × fan-in ) is why value-level provenance has an entire class of problems that column lineage never faces. Why bother? The reasons for this level of granularity Granularity is expensive,

TimescaleDB 2.27, released May 12 2026, extends bloom-filter batch pruning from reads to writes. UPDATE, DELETE, and UPSERT against compressed columnstore data can now skip decompressing batches that provably cannot contain the target rows. The reported gains are real: up to 160x for selective UPDATE/DELETE, and over 2x for UPSERT. The feature is automatic. Whether it is actually firing on your workload is not something you can assume, and the only way to confirm it is to read new EXPLAIN counters that the release notes mention but do not explain. Worse, the counter names are inconsistent between the write paths, so even a careful reader ends up guessing. This post is about reading those counters correctly, and about the two things in this release that will silently break a query if you upgrade without noticing them. What is actually being skipped A quick model of the mechanism, because the counters only make sense against it. Hypercore stores compressed data in batches, roughly a thousand rows each. For columns that are not the segmentby key, TimescaleDB maintains a sparse bloom filter per batch: a small probabilistic summary that answers one question, "could this batch contain column = X ?", without touching the compressed payload. A bloom filter has a useful asymmetry. A negative is certain: if the filter says no, the value is definitely absent, and the batch can be skipped whole. A positive is not: the filter says "maybe", you decompress, and sometimes the value is not there after all. That last case is a false positive, and it is the number that tells you whether the whole scheme is paying off. Before 2.27, a DELETE ... WHERE sensor_id = 'x' against compressed data decompressed every candidate batch to check. Now the bloom filter is consulted first, and batches that cannot match are never decompressed. The work you save is the decompression of the batches that get pruned. The work you waste, when the filter is poorly matched to your data, is the bloom check on

SpaceX is building a new power plant for xAI's Colossus data centers, but it won't remove existing, unpermitted turbines for many more months.

Metabase is an open-source BI tool for building charts and dashboards over MySQL, PostgreSQL, MongoDB, Redshift, and more. This guide deploys Metabase on Kubernetes, loads the Sakila sample dataset into MySQL, builds a dashboard, and secures it behind Nginx Ingress with cert-manager TLS. Prerequisites: a Kubernetes cluster with kubectl / helm configured, a Linux workstation, a reachable MySQL server, and a domain name. Load the Sakila Sample Database Sakila models a DVD rental store — films, actors, inventory, rentals. $ sudo apt install zip -y $ wget https://downloads.mysql.com/docs/sakila-db.zip $ unzip sakila-db.zip Connect to your MySQL server (replace host/port/user): $ mysql -h <HOST_ENDPOINT> -P <DATABASE_PORT> -u <ADMIN_USER> -p mysql > CREATE DATABASE sakila ; mysql > SOURCE sakila - db / sakila - schema . sql ; mysql > SOURCE sakila - db / sakila - data . sql ; Deploy Metabase $ nano metabase.yaml apiVersion : apps/v1 kind : Deployment metadata : name : metabase spec : selector : matchLabels : app : metabase replicas : 1 template : metadata : labels : app : metabase spec : containers : - name : metabase image : metabase/metabase:latest ports : - containerPort : 3000 protocol : TCP --- apiVersion : v1 kind : Service metadata : name : metabase-svc spec : type : LoadBalancer selector : app : metabase ports : - name : http port : 8080 targetPort : 3000 Your cloud provider may need a provider-specific LoadBalancer annotation here (e.g. to set the listener protocol) — check its Kubernetes docs if the default doesn't work. $ kubectl apply -f metabase.yaml $ kubectl get deployments $ kubectl get services Wait for metabase-svc to get an EXTERNAL-IP (can take a few minutes), then visit http://<external-ip>:8080 to confirm the Metabase welcome page loads. Connect Metabase to the Database Let's get started → pick language. Enter your name, email, company, and a password. Select your use case. Database engine: MySQL . Set a display name, then host/port/database/user/pa

A Patroni cluster needs an odd number of nodes to maintain quorum — with 3 nodes, losing 1 still leaves a majority, so the cluster keeps running. This guide builds a 3-node PostgreSQL cluster on Ubuntu 24.04 with Patroni handling replication and automatic failover, etcd as the coordination store, and HAProxy load-balancing client connections — all secured with TLS. Prerequisites: three Ubuntu 24.04 servers (2 vCPU / 4GB RAM minimum) with PostgreSQL installed, non-root sudo access, and a domain with three A records: node1.example.com , node2.example.com , node3.example.com . Replace these placeholders with your actual subdomains throughout. Install Dependencies Run on all three nodes unless noted otherwise. 1. Install packages: $ sudo apt update $ sudo apt install haproxy certbot pipx -y $ sudo pip3 install --break-system-packages 'patroni[etcd3]' psycopg2-binary psycopg 2. Install etcd: $ wget https://github.com/etcd-io/etcd/releases/download/v3.6.4/etcd-v3.6.4-linux-amd64.tar.gz $ tar -xvf etcd-v3.6.4-linux-amd64.tar.gz $ sudo mv etcd-v3.6.4-linux-amd64/etcd etcd-v3.6.4-linux-amd64/etcdctl /usr/local/bin/ 3. Open firewall ports — 80 (Certbot), 2379/2380 (etcd), 5432/5433 (PostgreSQL + Patroni-managed PostgreSQL), 8008/8009 (Patroni REST API): $ sudo ufw allow 80,2379,2380,5432,5433,8008,8009/tcp $ sudo ufw reload $ sudo ufw status Configure SSL Certificates 1. Request a certificate per node (run on each node for its own subdomain): $ sudo certbot certonly --standalone -d node1.example.com -m admin@example.com --agree-tos --no-eff 2. Create a cert-prep script on each node (set HOSTNAME to that node's subdomain): $ sudo nano /usr/local/bin/prepare-ssl-certs.sh #!/bin/bash HOSTNAME = "node1.example.com" # Update for each node CERT_DIR = "/etc/letsencrypt/live/ $HOSTNAME " ARCHIVE_DIR = "/etc/letsencrypt/archive/ $HOSTNAME " getent group ssl-users > /dev/null || sudo groupadd ssl-users for user in etcd patroni haproxy postgres ; do if ! id " $user " > /dev/null 2>&1 &&

Manticore Search 28.6.6 has been released. The headline additions are UUID document IDs for real-time tables and ordered, limited GROUP_CONCAT() for grouped queries. The release also includes 16 fixes for backups, replication, query processing, SQL compatibility, and secondary indexes. This post covers everything shipped from 28.4.5 through 28.6.6 . Upgrade notes There are no new mandatory data migrations in this release. UUID IDs are an opt-in table definition: existing numeric-ID tables keep working as they are. If you want UUID identifiers, create a real-time table with id uuid ; ALTER TABLE cannot convert an existing table between numeric and UUID IDs. Two fixes are particularly useful for production installations. Successful backups now always unfreeze real-time tables when they finish (previously in rare cases they didn't), rather than leaving writes blocked. And authenticated replication can again add an existing populated RT table with ALTER CLUSTER ... ADD . UUID document IDs for real-time tables Applications often already have UUID identifiers from the system of record. Until now, using them with Manticore Search meant maintaining a separate numeric ID mapping. Real-time tables can now use UUID document IDs directly: CREATE TABLE products_uuid ( id uuid , title text , price int ); Manticore accepts an explicit UUID string, or generates one when id is omitted from an insert or replace. UUID equality and IN filters work in queries, and UUID IDs can be used with REPLACE , UPDATE , and DELETE . This is currently a real-time-table capability, including columnar and replicated RT tables. Plain, percolate, and sharded tables continue to use their existing ID models. Ordered and limited GROUP_CONCAT() Grouped results often need a compact preview of the most relevant values in each group. GROUP_CONCAT() can now sort values and retain only the requested number of them in explicit SQL GROUP BY queries: SELECT category , GROUP_CONCAT ( title ORDER BY price DESC SEPARA

TL;DR I designed a proposed AWS Transform Custom community skill that prepares Glue 2.0, 3.0, and 4.0 repositories for Glue 5.0. It separates safe mechanical transformations from changes that require human evidence, generates a migration report, and preserves already-compatible files unchanged. Because I didn't have live atx access, the benchmarks in this post are explicitly labeled manually simulated, not agent-executed. The proposal is open as issue #75 — not yet merged, not yet a pull request. The missing data-engineering transformation AWS Transform Custom can apply agent-driven code transformations across a single repository — or thousands of them at once, via AWS Batch and Fargate. As of July 30, 2026, its public sample repository, aws-samples/aws-transform-custom-samples , contained three community-contributed transformations: an EKS version-upgrade-readiness skill, a JBoss-to-Spring-Boot migration, and a Kubernetes readiness migration. None of them touched data engineering. Given that most of my day-to-day work sits across AWS data engineering, Databricks, and Delta Lake, that gap was the obvious thing to fill. What an AWS Transform Custom "skill" looks like Before writing anything, I studied the deepest existing example, jboss-to-springboot , since the pattern it establishes is effectively an unwritten spec for the other two skills as well: README.md — the problem, what the skill does, and how to invoke it via the atx CLI. This is also where the repo draws a clear line: these are readiness transformations. They modify repository artifacts — code and infrastructure-as-code — but they don't deploy jobs, call AWS APIs to change running resources, or claim data-level equivalence. That distinction matters throughout everything below. SKILL.md — the agent-facing definition: YAML frontmatter with trigger keywords, an Objective, explicit Non-Goals , Constraints, worked before/after examples, a "signal in source code → reference file" routing table, and a numbered V

Satellites show burn scars and fires at AWS data centers and Saudi oil refinery.

Whether you are using off-the-shelf AI coding tools like Claude Code and Cursor or building custom autonomous AI agents with TypeScript and LLM APIs, you hit the exact same fundamental wall: AI agent amnesia . As an agent user , you spend forty-five minutes explaining your architecture, deployment quirks, and database rules. The agent writes brilliant code. You close the CLI or tab, open a new session the next morning, and the agent suggests the exact legacy library you rejected yesterday. As an agent builder , you struggle to keep your custom agentic loops focused. As multi-step agent trajectories expand, LLM token limits force context compaction, wiping out subtle rules and past decisions while escalating API costs. The intelligence is real. The amnesia is structural. Context Windows Are Working Memory, Not Long-Term Memory The AI industry’s standard reflex to agent amnesia has been pushing context windows to 1M+ tokens. But a context window is working memory (RAM), not long-term storage (disk). Relying on massive context windows introduces three critical engineering bottlenecks for both users and builders: Context Compaction Destroys Rationale : When a session reaches token limits, agents automatically compact their context history. Compaction summarizes conversations into short summaries, quietly wiping out subtle architectural constraints, edge cases, and past decisions. Context Drift & Attention Loss : LLMs struggle with needle-in-a-haystack attention degradation when context windows are stuffed with 100k+ lines of raw conversation history. Escalating API Costs & Latency : Re-sending full project transcripts on every prompt burns tokens rapidly and adds seconds of input processing delay for users while skyrocketing LLM bills for agent builders. Agents do not need larger transcripts. They need a durable, inspectable, versioned memory layer . Why Vector Databases Fall Short for Local & Workspace Agent Workflows When developers and AI engineers realize raw contex

Add another member to the fast-and-furious AI unicorn club: Simile

Databricks is expanding its Lakebridge toolkit by introducing an agentic code conversion feature designed to help organizations migrate from legacy data warehouses. This new capability uses Genie Code to rewrite complex SQL scripts, allowing customers to transition their workloads to the Databricks lakehouse environment with higher efficiency and less manual intervention. Advanced Automation for Complex Code Translation The core of this update is the agentic code converter, a system that utilizes AI subagents to manage the heavy lifting of migration projects. These agents perform a variety of tasks including deep analysis of source code and the parallel conversion of multiple files. They also validate translated SQL and can autonomously retry sections that fail during the initial pass. This iterative approach is a significant step forward from traditional methods that often require human developers to step in when software hits a wall. By allowing developers to set specific migration rules for unique enterprise SQL structures, the tool provides a level of customization that previous automated systems lacked. The Lakebridge suite already offers several transpilation engines, such as the pattern-based BladeBridge technology and the compiler-based Morpheus engine. However, the addition of agentic AI introduces a reasoning layer that these older technologies do not possess. This reasoning is vital for moving beyond simple syntax mapping and into the realm of complex logic. Traditional transpilers like Morpheus are excellent at handling standard syntax mapping. They easily manage date functions and basic join commands. Problems arise when these tools encounter control-flow reasoning, cursors, or dynamic SQL that is generated at runtime. These complex elements often differ significantly across platforms like Oracle or Teradata. Industry experts note that these difficult sections usually represent about 15 percent of a codebase but consume the vast majority of manual labor

British AI neocloud Nscale is buying software startup Anyscale, which helps companies scale their AI workloads across data centers and servers.

Cybersecurity experts told TechCrunch that one of the biggest lessons to be taken from the OpenAI hack against Hugging Face has nothing to do with AI, but traditional cybersecurity defense.

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

Despite the ongoing AI boom, LinkedIn is holding the line on compute spending. Instead, it’s challenging engineers to make every GPU count.

How I applied Exploratory Data Analysis, Feature Engineering, Pipelines, and Ensemble Models to solve a real-world machine learning problem—and the lessons I learned along the way. Introduction There comes a point in every machine learning learner's journey when watching tutorials and completing small practice exercises are no longer enough. After spending weeks understanding statistics, exploratory data analysis (EDA), feature engineering, preprocessing techniques, and classical machine learning algorithms, I wanted to answer one question: Can I apply everything I've learned to a real machine learning competition? That's when I decided to participate in a Kaggle Playground competition. Unlike classroom datasets, Kaggle competitions force you to think like a machine learning engineer. You're responsible for understanding messy data, building preprocessing pipelines, selecting models, evaluating performance, debugging errors, and finally creating a submission that competes with thousands of participants. This article documents my complete journey—from loading the dataset to building production-style preprocessing pipelines and training multiple ensemble models. Along the way, I'll also share the challenges I faced, what worked well, and the lessons I'll carry into future competitions. Why Kaggle? Learning machine learning isn't just about knowing algorithms. Real-world ML requires answering questions like: Which features are useful? How should missing values be handled? Should categorical variables be one-hot encoded or ordinal encoded? Which preprocessing steps belong inside a pipeline? How do different ensemble models compare? Kaggle provides an environment where all of these questions matter. Instead of building a model that works only inside a notebook, you're solving a problem under realistic constraints and evaluating your solution on unseen data. Competition Goal The objective of this Playground competition was to predict the target class based on a combinatio

Introduction When building a Flutter application, you'll often need to store data on the user's device. For example: Saving user preferences Storing login information Caching API responses Creating offline applications Building note-taking or to-do apps While there are several local storage solutions available, Hive is one of the fastest and easiest local storage for Flutter developers. In this tutorial, you'll learn Hive from scratch by building a simple example. No prior database knowledge is required. What is Hive? Hive is a lightweight, NoSQL database written entirely in Dart. It stores data directly on the device, making it perfect for Flutter applications. Why use Hive? Extremely fast Works offline No native platform code required Simple API Easy to learn Great for small and medium-sized applications Think of Hive as a collection of boxes where each box stores your application's data. Hive ├── User Box ├── Settings Box ├── Notes Box └── Products Box Each Box is similar to a table in traditional databases. Step 1: Create a Flutter Project Create a new Flutter project. flutter create hive_demo Open the project. cd hive_demo Step 2: Install Hive Open pubspec.yaml and add the following packages. dependencies : flutter : sdk : flutter hive : ^2.2.3 hive_flutter : ^1.1.0 Then install them. flutter pub get Step 3: Initialize Hive Before using Hive, initialize it inside main() . import 'package:flutter/material.dart' ; import 'package:hive_flutter/hive_flutter.dart' ; void main () async { WidgetsFlutterBinding . ensureInitialized (); await Hive . initFlutter (); await Hive . openBox ( 'settings' ); runApp ( const MyApp ()); } Here we open a box called settings . Step 4: Understanding Boxes A Box is where Hive stores data. Imagine this box: Settings Box theme -> dark username -> Alex loggedIn -> true Keys are on the left. Values are on the right. Step 5: Save Data Saving data is incredibly simple. var box = Hive . box ( 'settings' ); box . put ( 'username' , 'John' );

At TechCrunch Disrupt 2026, the AI Stage is back to dig into the single hottest topic in the community for the past few years, presented by Google for Startups.

On May 20, 2026, AWS ended support for DMS Fleet Advisor. Fleet Advisor answered a question every migration team asks first: what is actually in my database estate, and how hard will it be to move? It was free. It was fully managed. It was backed by the largest cloud provider on earth. It still lost. AWS's official notice says only: "After careful consideration, we decided to end support for AWS DMS Fleet Advisor." No reason given. But you don't need one — the documentation tells you. Here is what Fleet Advisor required before it would tell you a single thing about your databases: Install a standalone data collector in your local environment Create an Amazon S3 bucket Create IAM policies, roles, and users — via CloudFormation, which was the recommended path Create database users with the minimum required permissions on every source Establish network access from the collector to each database server Then you'd meet the ceilings: recommendations for up to 100 databases at a time, one-to-one target mapping only, no multitenant server support. Now picture running that gauntlet inside a bank. You are a Business Solution Architect. You have been asked to scope a migration. You do not yet have approval for the migration — that approval is what the assessment is for . And to produce the assessment, you must first request production database credentials, get an agent binary through software approval, provision an S3 bucket, and get an IAM stack past a security review. That is a six-week procurement conversation to answer a question you were hoping to answer this week. AWS's replacement recommendation is Migration Evaluator — a consulting-led engagement. Read that as the finding it is: AWS looked at self-serve migration assessment, and concluded that humans and services do it better than a product. I think they were half right. And the half they got wrong is the interesting part. The lesson: friction is a competitor, and it usually wins We talk about developer tools as if the

Missing data is not the same thing as a convenient default. That sounds obvious, yet temporal software regularly converts an empty time field into midnight, noon, the current time, or the start of a day. The interface may look complete after that conversion, but the program has silently changed an unknown fact into a known one. This matters anywhere an hour can change the result: medical timelines, transport schedules, legal deadlines, astronomical calculations, historical records, and calendrical systems. I encountered the problem while working with a BaZi calculation pipeline. A BaZi chart can use year, month, day, and hour components. If the birth time is absent, the honest result is a three-component analysis with hour-dependent conclusions withheld. Inserting noon would make the output look richer while making its provenance weaker. The useful engineering question is not “Which fallback time should we choose?” It is “How do we keep uncertainty visible through every layer of the system?” The public calculation evidence repository provides the concrete calendar-domain fixtures referenced below. The rest of this article focuses on the reusable software boundary behind them. Model knowledge, not just a string A common input model makes absence too easy to erase: const birthTime = form . time || " 12:00 " ; After this line runs, downstream code cannot tell whether noon came from the user or the fallback. Validation, analytics, caching, and the result renderer all see the same string. The information loss happens before the calculation begins. A small discriminated union keeps the two states separate: /** * @typedef {{ kind: "known", localTime: string, source: "user" }} * KnownTime * @typedef {{ kind: "unknown" }} UnknownTime * @typedef {KnownTime | UnknownTime} BirthTime */ function parseBirthTime ( value ) { const normalized = value ?. trim (); return normalized ? { kind : " known " , localTime : normalized , source : " user " } : { kind : " unknown " }; } This typ