标签：#Data

User segmentation requirement Imagine you need to send a push notification to users who satisfy all of the following conditions: Push notification is enabled User is a VIP Active within the last 30 days Following the Voucher Hot category The traditional approach is to query multiple tables: SELECT DISTINCT u . id FROM users u JOIN user_configs c ON c . user_id = u . id JOIN devices d ON d . user_id = u . id JOIN follows f ON f . user_id = u . id WHERE c . push_optin = 1 AND c . mute = 0 AND d . fcm_token IS NOT NULL AND f . category = 'voucher_hot' AND u . last_active >= NOW () - INTERVAL 30 DAY ; As your user base grows into the millions, every campaign requires joining multiple large tables, filtering millions of records, and repeatedly computing the same audience. Query latency increases significantly, making real-time segmentation increasingly difficult. A Different Approach Instead of querying the database every time, we precompute each boolean attribute as a bitmap. Think of a bitmap as a huge array containing only 0 and 1, where the index corresponds to the user ID. For example, a bitmap representing whether a user has enabled push notifications: User ID : 0 1 2 3 4 5 6 7 Bitmap : 1 0 1 1 0 0 1 1 To check whether user 123 has enabled notifications, simply read bit 123. 1 → enabled 0 → disabled Each bitmap represents exactly one boolean property: bitmap:push_optin bitmap:vip bitmap:active30 bitmap:follow:voucher_hot Memory Usage Bitmap is extremely memory efficient. Each user requires only one bit. For 1 million users: 1.000.000 bits ~ 125.000 bytes ~ 122 KB That means every segment only consumes about 122 KB of Redis memory. Even 100 different segments require only around 12 MB . Finding Intersections Suppose you want all users that are: VIP AND Push Opt-in AND Active30 AND Following Voucher Hot Redis can calculate the result with a single command: BITOP AND result vip push_optin active30 follow_voucher_hot Need the number of matched users? BITCOUNT result No

If you've ever tried to buy a portable power station, you know the problem: every brand claims to be the best, the spec sheets are a wall of numbers, and the forums are full of confident but contradictory advice. "What size do I actually need?" is the most-asked question and the least-clearly-answered. So I did the boring thing. I built a structured database of 102 portable power stations from 24 brands — capacity, output, chemistry, cycle life, solar input, weight, price — and started running the numbers. A few findings were genuinely surprising. 1. Prices quietly collapsed The median portable power station now sits at $0.61 per watt-hour . The cheapest in the dataset is $0.39/Wh (the GRECELL T1000). A few years ago, ~$1/Wh was normal and anything under $0.70 felt like a deal. The practical takeaway: if you're paying much more than ~$0.70/Wh in 2026, you're mostly paying a brand premium. That premium sometimes buys you a better app, ecosystem, or support — but it's worth knowing you're paying it. 2. LiFePO4 basically won 94% of the models I track now use LiFePO4 (lithium iron phosphate) instead of the older lithium-ion (NMC) chemistry. This matters more than any marketing bullet point: LiFePO4: ~3,000–4,000+ charge cycles Older Li-ion (NMC): ~500 cycles At daily use — say you cycle it every day in a van or for backup — that's roughly 8 years vs 18 months before the battery is meaningfully degraded. If a listing still uses NMC to hit a lower price, that "deal" can cost you far more over its life. 3. The fridge myth costs people hundreds of dollars This is the single most common sizing mistake. People size a giant, expensive battery to run a fridge because they do the math like this: Fridge nameplate (150W) × 24 hours = 3,600 Wh/day But a fridge's compressor only runs about 40% of the time . Its real average draw is closer to 60W, so: 150W × 40% × 24h ≈ 1,440 Wh/day In runtime terms: a 1,000Wh power station runs a full-size fridge for about 14 hours , not the ~6 hour

Arun Joseph shares real-world insights on scaling enterprise agentic platforms like Deutsche Telekom’s LMOS. He discusses bridging organizational fault lines, replacing tool sprawl with core platform abstractions, and moving beyond basic chatbots to operational intelligence systems through ephemeral agents and an Agent Definition Language (ADL). By Arun Joseph

Embabel has reached its 1.0 release, providing a framework for AI agents on Java It allows Java and Kotlin developers to define agents as typed domain objects. Built on Spring AI, Embabel supports multiple model providers and combines planning with predefined state machines, offering flexibility for agent workflows. By Erik Costlow

In Parts 1-4, we built a transactional key-value store. It has WAL for durability, memtables and SSTables for storage, compaction to control file growth, and transactions for atomic multi-key writes. Now we move to the query layer where users can actually fire SQL queries like: CREATE TABLE payments ( amount INT , id STRING , status STRING , captured BOOL , PRIMARY KEY ( id )) INSERT INTO payments VALUES ( 500 , payment_1 , pending , 1 ) SELECT * FROM payments WHERE id = payment_1 In this blog and the next one we answer: How do we translate SQL strings into operations our key-value store already understands? This post focuses on the first half of that bridge, which is parsing SQL into structured commands. In the next post, we will take those commands and turn CREATE TABLE and INSERT into bytes on disk. The Core Idea: SQL Becomes Structured Data The storage engine does not understand SQL. It understands keys, values, WAL entries, memtables, SSTables, and transactions. So the SQL layer has two jobs: Parse a human-readable SQL string into a structured object. Translate that structured object into key-value operations. For example: CREATE TABLE payments (...) -> CreateTable{TableName: "payments", ColumnDetails: ...} INSERT INTO payments VALUES (...) -> InsertIntoTable{TableName: "payments", ColumnValues: ...} SELECT * FROM payments WHERE id = payment_1 -> SelectFromTable{TableName: "payments", QueryConditions: ...} Once we have these structs, the rest of the database can stop dealing with raw strings. Why Not Parse SQL Directly in the DB Layer? Imagine if db.CreateTable() directly walked through the SQL string and also updated storage. That would mix two very different responsibilities: parsing grammar, executing database operations. Keeping them separate makes the system easier to reason about. The parser validates syntax and builds an AST. The DB layer receives that AST and decides what to store. An AST, or Abstract Syntax Tree, is just a structured representation of

The lost-update bug that quietly corrupts homegrown wallet balances — and the five disciplines we used across PayWithToken to make money movement correct under concurrency. There is a bug that lives in a large share of the world's homegrown wallet systems. It doesn't throw an error. It doesn't show up in tests. It surfaces months later as a balance that is quietly, inexplicably wrong — and in a payments system, a wrong balance is either a customer who has lost money or a company that has given it away. This is the story of that bug, why the "obvious" wallet code causes it, and the handful of disciplines we used across PayWithToken to make money movement correct under concurrency. The bug: lost updates Here is wallet code almost everyone writes first. Credit a user's balance: // DON'T do this $row = $db->query("SELECT balance FROM users WHERE id = $id")->fetch(); $new = $row['balance'] + $amount; $db->exec("UPDATE users SET balance = $new WHERE id = $id"); Read the balance, add to it in PHP, write it back. It works perfectly — until two things happen at the same time. Picture a wallet at ₦1,000. Two credits of ₦500 arrive simultaneously — say a bank webhook and the user tapping "confirm" on their phone: Request A reads balance = 1000. Request B reads balance = 1000 (A hasn't written yet). A computes 1500, writes 1500. B computes 1500, writes 1500. Two credits landed; the balance rose by ₦500. ₦500 vanished. This is a lost update, and it is a race condition, which means it is invisible until you have real concurrent traffic — exactly when you can least afford it. The debit version of the same bug lets a balance go negative or double-spends a token. Fix #1: let the database do the arithmetic The read-modify-write happened in PHP, across three round trips, with a gap where another request could interleave. The fix is to make the update a single atomic statement and let the database's row lock serialise it: // DO this — one atomic statement $db->prepare("UPDATE users SET

That's a rebuild I shipped. The system: a RAG assistant for fraud analysts — ask it "how do we handle card testing followed by a successful auth?" and it should answer from the team's own SOPs and case history. The complaint: the answers were wrong, therefore the model must be dumb, therefore procurement should buy a bigger model. The model was fine. It was answering perfectly — from garbage context. Walk the forensic trail with me, because every step is checkable on your own system this week. Exhibit A: the chunking was destroying meaning before anything was embedded The ingestion split SOP documents every N characters, mid-sentence. Which means half the vectors in the index encoded fragments like this: chunk_147 = " ...ing to a freight forwarder. In these cases, do NOT " chunk_148 = " cancel the order immediately. First verify the customer via " The policy — don't cancel, verify first — exists in no single chunk. An embedding can't encode a meaning that isn't in its input. Retrieval was being asked to find semantics the pipeline had already shredded. Fix one: chunk on structure (sections, paragraphs), never on character counts, with enough overlap that no rule straddles a boundary. Exhibit B: dense-only retrieval, bimodal queries Fraud analyst queries split into two populations: pattern questions ("high-value order, new account, rushed shipping") and identifier questions ("what's the SOP for decline code 4863?", "rule VEL-013 rationale"). The system was dense-only — and embeddings treat a rare token like 4863 as noise, so identifier queries retrieved similar-feeling chunks instead of the literal match. Half the query population was structurally doomed regardless of model quality. Fix two: hybrid retrieval — BM25 for the identifiers, embeddings for the patterns, reciprocal rank fusion to merge. Exhibit C: nobody could see any of this, because quality was a rumor No golden dataset. No retrieval metric. The system's accuracy was whatever the loudest anecdote said it

We live in an era where our wrists track every heartbeat, step, and sleep cycle. Yet, most of this "Quantified Self" data sits rotting in massive .xml or .json export files that are impossible to read. What if you could simply ask your AI, "How did my resting heart rate trend during the week I was stressed about the product launch?" In this tutorial, we are building a Quantified Self RAG (Retrieval-Augmented Generation) pipeline . We will take fragmented health data from Apple HealthKit and Google Health Connect, process it using DuckDB , and vectorize it into Pinecone using LangChain . By the end of this guide, you’ll have a production-grade Health Data RAG system capable of high-performance natural language queries over your personal biometrics. The Architecture: From Raw Logs to Vector Insights Handling health data at scale requires a robust ETL (Extract, Transform, Load) process. Vectorizing every single heart rate measurement (which can occur every few seconds) is inefficient and expensive. We need to downsample and summarize before embedding. graph TD A[Apple Health/Google Health] -->|Export XML/JSON| B[Raw Data Storage] B --> C{DuckDB Processing} C -->|Cleaning & Downsampling| D[Structured Parquet/JSON] D --> E[LangChain Document Loader] E --> F[OpenAI Embeddings] F --> G[Pinecone Vector Database] H[User: 'Why was my sleep poor last Tuesday?'] --> I[LangChain RAG Chain] G --> I I --> J[LLM Contextual Answer] Prerequisites 🛠️ To follow along, you'll need: Python 3.10+ Tech Stack : Pinecone , LangChain , DuckDB , OpenAI , and Pandas . An export of your health data (Apple Health export.xml or Google Takeout). Step 1: Efficient Data Crunching with DuckDB Apple Health exports are notoriously large XML files. Loading them directly into memory with standard Python is a recipe for a crash. We use DuckDB for its blazing-fast analytical capabilities to filter and downsample our data. import duckdb # Load and parse the XML (simplified logic) # Note: In a real scenario,

From the outside, social features seem like something straightforward: follow a user, like a post, see a feed, but when you attempt to implement them at any real scale you find that every single one is something you've got to be aware of, a pattern, a well-documented query pattern with all its failure modes. Here's a step-by-step look at four of them: adjacency list, fan-out feed, mutual-friends join, and some graph traversal that SQL wasn't really designed for. The adjacency list and why "who do I follow" isn't free A follow relationship is usually modeled as a plain adjacency table: follows(follower_id, followee_id, created_at) . That's the whole schema, and it's deceptively adequate for a long time. The first place it breaks is when you need "who do I follow that also follows them", mutual connections, because that's a self-join against the same table: SELECT f2 . followee_id FROM follows f1 JOIN follows f2 ON f1 . followee_id = f2 . follower_id WHERE f1 . follower_id = : user_id AND f2 . followee_id != : user_id ; This query is ok if the fan-out is low. It feels like it's no longer fine when a few accounts have a few hundred thousand joins, because join now needs to walk a correspondingly large intermediate result set per request. The typical solution isn't a better query, it's a better index and a cap. composite indexes on (follower_id, followee_id) and (followee_id, follower_id) so both directions of the join hit an index-only scan, plus a LIMIT applied early so the planner doesn't materialize more rows than the response will ever use. The fan-out feed problem The “social systems” hard problem is the timeline: display to me my feed of what people I follow posted recently, in order. There are two ways to construct it and they are both right - anything else and outages happen. Fan-out on write means that when a user posts, you insert a row into every follower's feed table immediately. Reads are then a trivial SELECT * FROM feed WHERE user_id = :id ORDER BY creat

Olá, comunidade DEV! Meu nome é Beth Rodrigues e este é o meu primeiro post por aqui. Durante muito tempo, minha relação com tecnologia esteve ligada ao uso das ferramentas no dia a dia de trabalho. Planilhas, sistemas, relatórios e processos faziam parte da minha rotina, mas em algum momento comecei a enxergar algo diferente: por trás dos dados existiam histórias, oportunidades de melhoria e decisões que poderiam ser mais inteligentes. Foi assim que comecei minha transição para a área de Dados. Minha jornada está sendo construída passo a passo: estudando SQL, Excel, Power BI, Python e conceitos de análise de dados. Também tenho explorado ferramentas de inteligência artificial, automações e formas de transformar problemas reais em soluções. Uma das coisas que mais me motivam é perceber que dados não são apenas números em uma tabela. Eles podem ajudar uma empresa a entender seus clientes, melhorar processos e encontrar caminhos mais eficientes. Atualmente estou criando pequenos projetos para praticar, aprender e construir meu portfólio. Quero compartilhar por aqui meus aprendizados, erros, descobertas e experimentos nessa caminhada. Acredito muito que aprender tecnologia é como montar um grande quebra-cabeça: no começo algumas peças parecem não fazer sentido, mas aos poucos a imagem começa a aparecer. Espero trocar experiências com pessoas da comunidade, aprender com quem já está nessa estrada e também contribuir compartilhando minha evolução. Que venha essa nova fase! 🚀

Originally published at nlqdb.com/blog The standard agent-memory build is an afternoon of work: embed every fact worth keeping, upsert it into a vector store, and before each reply pull the top-k most similar memories back into context. And for what it's built for, it works. Ask "what did this user say about the Berlin migration" and the right snippets come back, ranked by cosine distance. Recall is solved enough that it feels like memory is solved. Then the agent has been running for a month, and you ask its memory a different kind of question: "how many users asked about pricing this month?" "Average deal size per stage?" "Top 10 topics I logged, ranked by count?" The store dutifully returns the twenty memories most similar to the question text , the LLM eyeballs them, and you get a confident, specific, wrong number. Recall is similarity. Reporting is aggregation. Nothing malfunctioned — the two questions want different machines. A vector store's primitive is nearest-neighbour search: embed the query, rank stored vectors by distance, return the top-k, optionally narrowed by a metadata filter. That is the whole contract. There is no COUNT , no GROUP BY , no JOIN , no HAVING — a similarity engine ships no query planner, and even the metadata filter only narrows candidates around the approximate search, so what comes back is still a ranking of similar items, never a computed result set. "How many" has to touch every matching row . If the agent logged 4,000 memories and top-k is 20, the context the LLM sees is structurally incapable of producing the count — and an LLM doing arithmetic over a retrieved sample is a hallucination generator, not a query engine. The failure is quiet, too: the answer arrives fluent and plausible, and nothing flags that it was computed from half a percent of the data. -- "top topics this month, ranked by count" is not a similarity query. -- It's this — and it must scan every matching row, not the top-k: SELECT topic , count ( * ) AS mentions

You've learned not to trust the first answer. So you read it carefully. You spot two problems. You fix them yourself, ship it, and move on. That's a reasonable way to work, and it's what separates an engineer who uses these tools well from one who copies and pastes. But there's a step you skipped. You never asked Claude to find the problems, and asking produces a different kind of output than reviewing does. 🔍 What actually happens Here's a specific case. You ask for an incremental load. You get something clean: a watermark column, a filter on records newer than the last run, an upsert into the target. You review it. You notice it assumes source records arrive in order, which yours don't, so you add a buffer window. You notice it doesn't handle the first run when the watermark is null, so you add a default. Two fixes, maybe fifteen minutes, and now it's correct. What you didn't find was the third problem: the upsert assumes a stable business key, and in your source system that key gets reassigned when records are merged. That one surfaces in production six weeks later as duplicate rows nobody can explain. You caught the problems you were looking for. You didn't catch the one you weren't. This is the normal outcome of self-review. You check against your own mental list of things that go wrong, and your list is good but finite. The problems that hurt are the ones outside it. 🧠 Why intermediates specifically miss this Beginners don't review AI output much at all, so this isn't their failure mode yet. Seniors have usually developed the habit after being caught out by something their own review missed. Intermediates sit in an awkward middle. They have learned, correctly, that AI output needs questioning. And they have concluded, understandably, that the questioning is entirely their job. That conclusion makes sense. Reviewing is what you do with a junior's pull request. It's what you do with your own code before you push. Review is a human activity performed on work some

How a single contiguous allocation — and a type system that won't let you feed strings to a scaler — is the real reason datarust fits in 2.3 megabytes. In the last post I showed you the whole datarust workflow: impute, scale, one-hot, train a logistic regression, evaluate, and save it as JSON — all without a Python runtime in sight. The Docker image shrank from ~900 MB to ~8 MB, and the binary was 2.3 MB. But I skimmed over something important. I kept saying "the flat memory layout" as if it were a detail. It isn't. It's the whole bet. Every scaler, every encoder, every model, every metric in datarust runs on top of one data structure. If you understand that structure — why it looks the way it does and what it refuses to let you do — the rest of the library stops being magic. So let's zoom in. Meet Matrix . Two containers, on purpose Real data is mixed. Numbers in one column, strings in the next. In Python, everything flows through one giant numpy.ndarray or a pandas.DataFrame , and the type system just... shrugs. A string column next to a float column gets coerced into object dtype. You'll find out at training time, in the form of an error message three frames deep. datarust does the opposite. It splits your data into two types at the source: use datarust :: Matrix ; use datarust :: matrix :: StrMatrix ; let numeric = Matrix :: new ( vec! [ vec! [ 3.0 , 85.0 , 24.0 ], vec! [ 12.0 , 70.0 , 31.0 ], vec! [ f64 :: NAN , 95.0 , 45.0 ], ]) ? ; let categorical = StrMatrix :: from_strings ( vec! [ vec! [ "MonthToMonth" ], vec! [ "OneYear" ], vec! [ "MonthToMonth" ], ]) ? ; Matrix is f64 only. StrMatrix is strings only. They are different types , and the compiler will refuse to compile a program that hands a string column to a scaler. Not at runtime — at compile time. In the last post I called this "putting on glasses for the first time." Let me show you what it actually buys you. The ColumnTransformer API is built on that split: ct .add_numeric ( "scaled" , vec! [ 0 , 1 ],

This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the Apache releases page before future installations. These instructions target Linux (Ubuntu/Debian) for development or testing. Production clusters need Kerberos, network controls, encryption, monitoring, backups, and an upgrade plan. Do not expose HDFS or YARN ports to the internet. Native single-node installation Prerequisites sudo apt-get update sudo apt-get install -y openjdk-17-jdk openssh-client openssh-server pdsh curl tar java -version Hadoop requires Java and SSH; pdsh is recommended by the current Apache single-node documentation. Find JAVA_HOME if needed: readlink -f "$(command -v java)" | sed 's:/bin/java::' Download and install Pin the version for repeatable installs and verify Apache's SHA-512 checksum: export HADOOP_VERSION=3.5.0 cd /tmp curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz" curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz.sha512" sha512sum -c "hadoop-${HADOOP_VERSION}.tar.gz.sha512" sudo tar -xzf "hadoop-${HADOOP_VERSION}.tar.gz" -C /opt sudo ln -sfn "/opt/hadoop-${HADOOP_VERSION}" /opt/hadoop sudo chown -R "$USER":"$USER" "/opt/hadoop-${HADOOP_VERSION}" Add this to ~/.bashrc, adjusting JAVA_HOME if necessary: export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 export HADOOP_HOME=/opt/hadoop export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop" export HADOOP_HDFS_HOME="$HADOOP_HOME" export HADOOP_YARN_HOME="$HADOOP_HOME" export HADOOP_MAPRED_HOME="$HADOOP_HOME" export PATH="$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin" Then load and verify it: source ~/.bashrc sed -i "s|^# export JAVA_HOME=.*|export JAVA_HOME=${JAVA_HOME}|" "$HADOOP_HOME/etc/hadoop/hadoop-env.sh" had

The answer arrives clean, well-formatted, and certain. You run it. The config flag doesn't exist. Nothing in the response told you the difference between the parts it was sure about and the parts it filled in. The syntax was right. The explanation was reasonable. The flag had a sensible name and a clear purpose. It just wasn't real. This is a calibration problem, not a trust problem. "Don't trust AI" is useless advice. It tells you to doubt everything, which is the same as doubting nothing, because you can't actually work that way. What you need instead is a way to tell which parts of an answer are grounded and which were filled in. There are patterns to look for, and once you know them, they are hard to miss. 🔍 Why this affects beginners specifically This isn't about intelligence. It's about experience. An engineer with a few years behind them has already lost time to a function that didn't exist. They have wasted an afternoon on a config parameter that turned out to belong to a different engine. They have built up a collection of "that looked right but wasn't" memories, and those memories now activate automatically. A small feeling that says check this one before they run it. Beginners haven't had those afternoons yet. The pattern recognition that comes from repeated failures doesn't exist, because the failures haven't happened. Everything arrives in the same confident tone, and without past mistakes to compare against, there is no internal signal separating a grounded answer from a made-up one. That's not a character flaw. It's missing experience, and it can be partly replaced with three concrete things to look for. ✅ The three signs These aren't perfect. They are a minimum standard, and they catch the common cases. I'll be clear about what they miss at the end. Sign 1: Very specific details with no source When the answer includes a precise detail (a specific config flag, a particular function signature, an exact version number) and doesn't say where that detail

Disclosure: I build Schemity , a desktop ERD tool - this post is from our blog and uses it for the examples. TL;DR: Database views carry real responsibilities - reporting layers, security boundaries, API surfaces - but ERD tools either leave them out entirely (DBML has no view support despite requests since 2022) or draw them as if they were ordinary tables. Schemity displays views and materialized views as read-only entities with italic names and a bold view or mview token in the entity footer, so derived relations are distinguishable from base tables at a glance, and they can be imported into context views like any other entity. A database view belongs in your ERD, but not disguised as a table: it is a derived, read-only relation, and the diagram should say so at a glance. Schemity draws views and materialized views as read-only entities with italic names - present on the canvas, visually distinct from the base tables they are built on. That sentence would be unremarkable if the rest of the tooling world agreed with it. Mostly, it does not. In most ERD tools your views are simply absent, and in the rest they are dressed up as something they are not. The reporting layer your diagram pretends does not exist Views are not decoration. They are where schemas put their public face: the reporting layer that joins five tables into one readable relation, the security boundary that exposes a subset of columns to an application role, the compatibility shim that survives a refactor. On Supabase , views are how you shape what PostgREST exposes as an API. A materialized view may be the single most performance-critical object in an analytics schema. Whoever reads your diagram to understand the system needs to see them. Yet the diagram usually cannot show them. DBML - the schema language behind dbdiagram.io - has no syntax for views at all: a user proposed designing views with join definitions in December 2022, others were still upvoting the request in July 2024, and there has be

Linear Regression: From Least Squares to Production-Ready Practice Tags : machinelearning , datascience , python , tutorial Linear regression is the first algorithm most people learn, and the one most people never study deeply. It is also the model you will still find in production after fancier algorithms fail, because it is fast, stable, and explainable. This article is not a "call .fit() and read the score" tutorial. We will cover the math, the statistical assumptions, the diagnostics, regularization, evaluation, production concerns, and the interview questions that separate beginners from engineers. Why Linear Regression Deserves a Second Look Linear regression is the foundation for understanding almost every other supervised model: Logistic regression is linear regression with a sigmoid on top. Ridge and Lasso are linear regression with constrained weights. Neural networks are stacked linear transformations with nonlinear activations. Tree models are judged against the same baseline: "can I beat a linear model?" More importantly, linear regression is still the right answer in many business problems. When you need to explain a prediction to a regulator, a client, or a finance team, a clean linear model with interpretable coefficients beats a black box. The Math: Least Squares and the Normal Equation Given features X and target y , a linear model assumes: y = X * beta + epsilon The goal is to minimize the residual sum of squares: L(beta) = ||y - X*beta||^2 Taking the derivative with respect to beta and setting it to zero gives the normal equation : beta = (X^T * X)^(-1) * X^T * y In practice, use the pseudoinverse ( pinv ) instead of the inverse, because X^T X may be singular or numerically unstable when features are collinear. import numpy as np def normal_equation ( X , y ): Xb = np . c_ [ np . ones ( X . shape [ 0 ]), X ] # add intercept beta = np . linalg . pinv ( Xb . T @ Xb ) @ Xb . T @ y return beta Three Equivalent Views of Least Squares 1. Geometric view

We've all been there: staring at a blood test report from three years ago, trying to remember if that "slightly elevated" glucose level was a one-time thing or a trend. Our health data is scattered across messy PDFs, fitness tracker exports, and physical medical folders. In the era of AI, why are we still manually digging through folders? 📂 Today, we are building the Ultimate Personal Health Knowledge Base . By leveraging Retrieval-Augmented Generation (RAG) , we will transform fragmented medical reports and logs into a searchable, private, and intelligent second brain. We’ll be using LlamaIndex for orchestration, Unstructured.io for parsing those pesky PDFs, and ChromaDB for local vector storage. If you're looking for advanced architectural patterns or production-grade data engineering strategies beyond this tutorial, I highly recommend checking out the deep dives over at WellAlly Tech Blog , which served as a major inspiration for this build. 🚀 The Architecture 🏗️ The goal is to create a pipeline that ingests raw data, vectorizes it, and allows for Hybrid Search —combining semantic meaning with keyword precision (crucial for medical terms!). graph TD A[Raw Health Data: PDFs, CSVs, MD] --> B(Unstructured.io Parser) B --> C{Chunking & Cleaning} C --> D[Sentence-Transformers] D --> E[(ChromaDB Vector Store)] F[User Query: Is my cholesterol improving?] --> G[LlamaIndex Query Engine] E <--> G G --> H[LLM: Local or OpenAI] H --> I[Actionable Health Insight] Prerequisites 🛠️ To follow along, you’ll need a Python environment with the following stack: Unstructured.io : To handle "dirty" PDF and image-based reports. ChromaDB : Our lightweight, open-source vector database. Sentence-Transformers : To generate local embeddings without sending data to the cloud. LlamaIndex : The glue that connects our data to the LLM. pip install llama-index chromadb unstructured sentence-transformers llama-index-vector-stores-chroma Step 1: Ingesting Messy Medical Reports 📄 Medical reports are

Most "data lineage" you've seen answers a schema question: table B comes from table A , or column B.total comes from columns A.price and A.qty . That's genuinely useful, and tools like OpenLineage do it well. But notice what it doesn't tell you: it says which columns can influence an output. It never says which values actually did . That gap is the whole subject of this post. I built a small, self-contained reference pipeline that captures provenance at the row and field level — "the value in this destination row, this field, was computed from these specific source (row, field) pairs" — and I want to walk through two things: why you'd ever want provenance at that granularity, and why it's genuinely hard once you commit to it. Repo (dbt-core + DuckDB, no server, no cloud, runs on a clean checkout): https://github.com/stevenblough/row-level-prov The one distinction everything follows from Here's the sentence the entire project turns on: Column-level lineage is a schema-sized, static fact you can derive from code. Value-level provenance is a data-sized, dynamic fact you must capture at execution. Put it in complexity terms and the consequences become obvious: Column lineage is O(schema) . It scales with how many columns you have. You can compute it by parsing SQL, offline, without ever looking at a single row. Value provenance is O(rows × fan-in) . It scales with your data volume times how many source values feed each output value. It does not exist anywhere until the query runs, and it can only be captured there , piggybacked on the query that actually produced the values. You cannot "reconstruct" value provenance later by re-querying the sources — the moment the source changes, you'd reconstruct a different answer than what really happened. That single exponent change ( schema → rows × fan-in ) is why value-level provenance has an entire class of problems that column lineage never faces. Why bother? The reasons for this level of granularity Granularity is expensive,

TimescaleDB 2.27, released May 12 2026, extends bloom-filter batch pruning from reads to writes. UPDATE, DELETE, and UPSERT against compressed columnstore data can now skip decompressing batches that provably cannot contain the target rows. The reported gains are real: up to 160x for selective UPDATE/DELETE, and over 2x for UPSERT. The feature is automatic. Whether it is actually firing on your workload is not something you can assume, and the only way to confirm it is to read new EXPLAIN counters that the release notes mention but do not explain. Worse, the counter names are inconsistent between the write paths, so even a careful reader ends up guessing. This post is about reading those counters correctly, and about the two things in this release that will silently break a query if you upgrade without noticing them. What is actually being skipped A quick model of the mechanism, because the counters only make sense against it. Hypercore stores compressed data in batches, roughly a thousand rows each. For columns that are not the segmentby key, TimescaleDB maintains a sparse bloom filter per batch: a small probabilistic summary that answers one question, "could this batch contain column = X ?", without touching the compressed payload. A bloom filter has a useful asymmetry. A negative is certain: if the filter says no, the value is definitely absent, and the batch can be skipped whole. A positive is not: the filter says "maybe", you decompress, and sometimes the value is not there after all. That last case is a false positive, and it is the number that tells you whether the whole scheme is paying off. Before 2.27, a DELETE ... WHERE sensor_id = 'x' against compressed data decompressed every candidate batch to check. Now the bloom filter is consulted first, and batches that cannot match are never decompressed. The work you save is the decompression of the batches that get pruned. The work you waste, when the filter is poorly matched to your data, is the bloom check on