标签：#Data

Most experiments that come back "no clear winner" were unwinnable on the day they launched. The data could not resolve an effect that size, and no amount of extra runtime was going to change that. You can find this out in about two minutes, before you spend anything, with one formula and a resampling pass over your own data. Here is the check, in three steps. Step 1. Compute the smallest lift your data can see For a two-arm test on a conversion rate, the smallest lift detectable at 95% confidence and 80% power is a one-liner: from math import sqrt Z_ALPHA = 1.96 # two-sided 95% Z_BETA = 0.84 # 80% power def mde ( baseline_cvr : float , n_per_arm : int ) -> tuple [ float , float ]: """ Minimum detectable effect: absolute (pp) and relative (%). """ se = sqrt ( 2 * baseline_cvr * ( 1 - baseline_cvr ) / n_per_arm ) abs_lift = ( Z_ALPHA + Z_BETA ) * se return abs_lift * 100 , abs_lift / baseline_cvr * 100 At a 3% conversion rate: clicks per arm smallest lift you can detect 5,000 +32% relative 20,000 +16% relative 100,000 +7% relative Read the middle row twice. Twenty thousand clicks per arm is a serious amount of traffic for a mid-market account, and a real 15% improvement still lands inside the confidence interval. The report will say "inconclusive," and the team will read that as a verdict on the idea. It is a verdict on the instrument. Invert the same formula and the planning question gets easier: at 3% baseline, detecting a 10% lift needs about 51,000 clicks per arm, and detecting a 5% lift needs about 203,000. If your account produces 8,000 clicks a month, you now know the honest answer to "how long should we run this." Step 2. Stop assuming your conversions are independent The formula above treats every click as an independent coin flip with the same probability. Account data does not behave that way, and the gap is not small. In a corpus of 31 advertiser accounts I maintain for diagnostic work (9.46 million search term rows, roughly $133M of spend, September 2024

Johannes Schickling explores how he moved beyond the traditional three-tier web stack to a local-first approach. He shares his experience transitioning from Prisma to developing Overtone—a music curation app leveraging client-side event sourcing and SQLite. The discussion highlights the trade-offs between event sourcing and CRDTs. By Johannes Schickling

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating. TL;DR Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting. Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction. The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control. Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint. This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model . Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one. Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait lon

The Confidence Tier Model: How to Decide When Your Data Isn't Enough Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule. TL;DR Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait. The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule. Underpowered tests systematically overestimate effect size (the "winner's curse" ). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise. The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge. This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence. A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world. The problem classic A/B testing doesn't solve Fixed-sample significance tes

Regression and regularization are both important concepts in machine learning and statistics, but they solve different problems. Regression is primarily used to model relationships and make predictions. Regularization is used to improve a model's ability to generalize by controlling its complexity. Regression This is a statistical and machine learning technique used to predict a continuous numerical outcome based on one or more input variables. For example, we might want to predict: A house's price based on its size and location A student's exam score based on study hours A company's sales based on advertising spending Simple Linear Regression In simple linear regression, we model the relationship between an input variable (x) and an output (y): $$ y = \beta_0 + \beta_1x + \epsilon $$ Where: (y) is the predicted outcome (\beta_0) is the intercept (\beta_1) is the coefficient or slope (x) is the input variable (\epsilon) represents the error The model learns values for (\beta_0) and (\beta_1) that make its predictions as close as possible to the actual values. Multiple Linear Regression In multiple linear regression, several predictors are used: $$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \cdots + \beta_px_p + \epsilon $$ The goal is typically to minimize the sum of squared errors (SSE) : $$ \text{SSE} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 $$ This approach is known as Ordinary Least Squares (OLS) . Regularization Regularization is a technique used to prevent a machine learning model from becoming too complex. A model can perform extremely well on training data but poorly on new, unseen data. This problem is called overfitting . Regularization addresses overfitting by adding a penalty for large model coefficients to the model's objective function. Instead of minimizing only the prediction error, the model minimizes: $$ \text{Prediction Error} + \text{Complexity Penalty} $$ The penalty discourages the model from relying too heavily on individual features. The Main Types o

The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew , and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformation Diagnosing and fixing data skew with salting and adaptive query execution (AQE) Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files Governing access to the whole pipeline with Unity Catalog Architecture overview Pipeline shape — a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage — this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1 — Set up governed tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics ; CREATE SCHEMA IF NOT EXISTS retail_analytics . events ; CREATE TABLE IF NOT EXISTS retail_analytics . events . raw_orders ( order_id STRING , customer_id STRING , product_id STRING , quantity INT , event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders' ; CREATE TABLE IF NOT EXISTS retail_analytics . events . dim_products ( product_id STRING , category STRING , unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products' ; GRANT SELECT ON TABLE retail_analytics . events . raw_orders TO `analysts` ; Step 2 — Read and force a broadcast join for the small dimension table dim_products is s

The moment that started it A colleague of mine was cleaning up some old records in a staging environment. Same query he'd run a dozen times before, except this time he was connected to production. He hit F5. No WHERE clause. 47,000 rows gone in milliseconds . We recovered from a backup, lost about two hours, and nobody got fired. But it stuck with me:** SSMS will let you delete an entire production table with the same amount of friction as running a SELECT 1**. No pause, no confirmation, nothing. The tool that DBAs and backend developers spend all day in has zero built-in protection against the single most common way people destroy data. So I built one. What SQL Guard does SQL Guard is a free SSMS extension (18 through 22) that inspects your query the moment you press F5, and pauses execution if it matches a known destructive pattern: DELETE without WHERE UPDATE without WHERE TRUNCATE TABLE DROP TABLE / DROP DATABASE / DROP PROCEDURE ALTER DATABASE Dangerous EXEC calls MERGE without a filter When one of these fires, you get a dialog showing exactly what object would be affected, with three options: cancel, run anyway, or run and ignore for the rest of the session. The point isn't to block you — it's to make the action conscious. Most accidental damage happens because muscle memory took over, not because someone genuinely meant to wipe a table. How the detection works Nothing exotic here, and honestly that's by design. SQL Guard runs a lightweight pattern-matching pass over the query text before it hits the connection. No parsing tree, no round-trip to the server, no measurable latency — it runs in under a millisecond, so you never notice it on normal queries. The core idea is simple: certain statements are only safe when scoped by a WHERE clause, and the extension checks for that clause's absence rather than trying to understand the full semantics of the query. That keeps false positives low and means it works the same way whether you're on SQL Server 2016 or the la

"What if satellite data could help protect the livelihoods of millions who depend on Africa's largest lake?" Our team JONAM had the privilege of participating in the Kijani Space Hackathon, where we proudly secured 3rd place while tackling Challenge 2: Sustainable Fisheries & Blue Economy. Rather than building another dashboard, we wanted to solve a real problem affecting millions of people around Lake Victoria: declining fish stocks caused by worsening water quality. Lake Victoria supports millions of people through fishing, transportation, agriculture, and tourism. However, over the years the lake has experienced: Increasing water pollution Poor water quality Frequent algal blooms Reduced fish breeding habitats Declining fish populations For fishing communities, these are not just environmental issues—they directly affect livelihoods, food security, and local economies. Our question became: Can Earth observation data help communities understand where water conditions are becoming unsuitable for fish before the problem becomes critical? Our Solution: JONAM JONAM is an AI-powered web application that combines satellite-derived environmental data with machine learning to monitor water quality and provide insights into conditions that may contribute to declining fish stocks. Instead of relying solely on manual sampling—which is expensive and only covers small areas—our platform continuously analyses satellite observations covering the entire lake. Why Copernicus? To build JONAM, we integrated the KijaniBox API, which provides access to environmental datasets from the Copernicus Programme. Copernicus is the European Union's Earth observation programme. It uses a constellation of Sentinel satellites together with in-situ observations to monitor Earth's atmosphere, land, and oceans. For our project, we focused specifically on live water telemetry variables available through the KijaniBox platform. Water Temperature Satellites measure the thermal radiation emitted from th

When I first heard the term ETL (Extract, Transform, Load), I thought it sounded like something only advanced data engineers dealt with. But as I’ve been learning, I realized ETL is the backbone of almost every data project. It’s the process that makes raw data usable, and without it, analysis can quickly fall apart. The first lesson was short but powerful. ETL is about moving data from one place to another, transforming it along the way so it’s clean and ready for analysis. I remember thinking: “So this is how companies make sense of the chaos in their databases.” It felt like peeking behind the curtain of how insights are really built. Then came the part about Excel macros. At first, I was intimidated, macros sounded complicated. But once I tried them, I realized they’re like little helpers that automate repetitive cleaning tasks. Instead of manually fixing hundreds of rows, I could write a macro and let Excel do the heavy lifting. It felt like discovering a secret shortcut. I even laughed at myself when I realized how much time I had wasted before, manually correcting data. This was a turning point: I started to see how automation can save not just minutes, but hours. Finally, I explored Power Query. If macros are shortcuts, Power Query is like a full toolkit. It lets you connect to different data sources, transform them, and keep everything organized. I loved how visual it was dragging, dropping, and shaping data felt almost creative. I remember thinking: “This is what makes data cleaning less of a chore and more of a craft.” It gave me confidence that even messy datasets could be tamed. Learning ETL, macros, and Power Query taught me that data cleaning isn’t just technical, it’s about mindset. It’s about respecting the data, being patient, and finding smarter ways to work. I used to think cleaning data was boring, but now I see it as the foundation of every meaningful insight. Without clean data, analysis is just noise. ✨ Takeaway: If you’re starting out in dat

A close call in Northern Virginia revealed just how poorly data centers respond to grid disruptions. Here's how to fix the problem.

Museums are embracing data-driven curation and a shifting technology landscape.

Engineers are increasingly arguing that modern LLMs can already reason through root cause analysis once given correctly prepared context, shifting the hard problem to the pipelines that correlate telemetry. A Coroot experiment across eleven models offers early evidence for the claim. By Mark Silvester

Most backend engineers implement multi-tenant quota checks using a standard "read-then-write" pattern. In production, this pattern is highly unsafe: SELECT grading_scans_remaining FROM profiles; If greater than 0, execute the application logic. UPDATE profiles SET grading_scans_remaining = grading_scans_remaining - 1; Under high volume or rapid concurrent requests, two independent processes will read the exact same balance before either one deducts usage. This race condition allows multi-tenant users to bypass your billing gates entirely. To solve this, you have to bypass the frontend and application-level checks, enforcing an atomic database operation that serializes the row update first. I have open-sourced a reference framework that outlines explicit subscription enums, core multi-tenant schemas, and a native VS Code / Cursor snippets configuration to speed up your local database modeling. 📂 Check out the repository on GitHub: { https://github.com/dollykm49/PostgreSQL-SaaS-Multi-Tenant-Subscription-Architecture-reference-framework- } What's inside the repository: Strictly Typed Enums: Centralized business rules handled natively by the database engine. Granular Balance Tracking: Optimized data-layer mapping for profiles and reset states. postgres-saas.code-snippets Engine: A local IDE configuration file that lets you deploy this core schema straight from your code editor by typing pg- shortcuts. For teams building commercial applications looking to skip weeks of writing custom migrations, testing concurrency edge-cases, and debugging row-locking security rules, the repository also includes a link to the extended 28-page production system bundle. Feedback on the multi-tier validation parameters is highly welcome!

Streamlit makes it remarkably fast to transform raw Python scripts into interactive, web-based data applications without needing any frontend knowledge in HTML, CSS, or JavaScript. In this tutorial, we will build a full-featured **Sales Analytics Dashboard** complete with real-time sidebar filtering, custom KPI metric cards, dynamic line/bar charts, and expandable data preview tables. --- ## Prerequisites To follow along, make sure you have Python 3.9+ installed along with the required libraries: bash pip install streamlit pandas numpy --- ## Step 1: Setting Up the Page & Mock Data with Caching First, we import the necessary libraries, set up the layout, and create a function to generate mock sales records. We use Streamlit’s `@st.cache_data` decorator so the data is only generated once per session, keeping the app snappy during user interactions. python import streamlit as st import pandas as pd import numpy as np Set layout configuration st.set_page_config(page_title="Sales Dashboard", layout="wide") Cache data loading for performance optimization @st .cache_data def load_data(): dates = pd.date_range("2025-01-01", periods=180) regions = ["North", "South", "East", "West"] df = pd.DataFrame({ "date": np.random.choice(dates, 500), "region": np.random.choice(regions, 500), "product": np.random.choice(["A", "B", "C"], 500), "sales": np.random.randint(100, 5000, 500), "units": np.random.randint(1, 50, 500), }) return df.sort_values("date") df = load_data() --- ## Step 2: Adding Interactive Sidebar Filters Next, we add controls inside the sidebar to let users filter the dataset by region, product type, and date range. A boolean mask applies those selections dynamically. python --- Sidebar filters --- st.sidebar.header("Filters") region_filter = st.sidebar.multiselect("Region", df["region"].unique(), default=df["region"].unique()) product_filter = st.sidebar.multiselect("Product", df["product"].unique(), default=df["product"].unique()) date_range = st.sidebar.date_input(

LSTMs are still the go-to architecture for a lot of time series work, but they're annoying to trust. You get a number out the other end and no real sense of why the model landed there. This tutorial walks through training an LSTM on daily temperature data, then pulling it apart with three explainability methods: permutation importance, SHAP, and Integrated Gradients. Who this is for: people who already know some Keras and want to add interpretability to a forecasting model, not a from-scratch intro to neural nets. 1. Getting the data into shape LSTMs want a 3D tensor — (samples, timesteps, features) — so before anything else we need to turn a flat column of temperatures into overlapping 7-day windows, each one paired with the value on day 8. import numpy as np import pandas as pd from sklearn.preprocessing import MinMaxScaler # 1. Load data df = pd . read_csv ( " weather_data.csv " ) data = df [ ' Temperature ' ]. values . reshape ( - 1 , 1 ) # 2. Scale the data for stable neural network training scaler = MinMaxScaler ( feature_range = ( 0 , 1 )) scaled_data = scaler . fit_transform ( data ) # 3. Create sequences: 7 days of lag to predict the 8th day X , y = [], [] for i in range ( 7 , len ( scaled_data )): X . append ( scaled_data [ i - 7 : i ]) y . append ( scaled_data [ i ]) X , y = np . array ( X ), np . array ( y ) print ( f " Input shape: { X . shape } " ) # Output: (Samples, 7, 1) Scaling matters more than it sounds like it should — LSTMs trained on unscaled temperature values are prone to exploding gradients, and training just falls apart. The windowing step is really the whole trick here: every prediction only ever sees the past seven days, nothing more. 2. Building the model Two stacked LSTM layers, dropout after each one, early stopping so we don't have to babysit the epoch count. from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM , Dense , Dropout , Input from tensorflow.keras.callbacks import EarlyStopping # 1. Build

I run a small self-hosted observability tool on the cheapest VPS I could find on purpose: 2 cores, 2 GB RAM, 20 GB SATA SSD . It ingests errors, traces and metrics from two low-traffic sites of mine. The stack is three containers — a Go app, PostgreSQL, and ClickHouse. One evening docker stats showed ClickHouse sitting on 880 MB of its 1 GB limit and the box swapping, with basically zero events coming in. So I went looking for where the memory and disk had gone. The answer turned out to be a good lesson in how a database can spend almost all of its I/O talking to itself. 543 KB of my data, 579 MB of ClickHouse talking about ClickHouse First thing I checked: how much data had my app actually stored versus how much ClickHouse had stored about itself . My application database: 543 KB, 16k rows The system database: 579 MB, 46.3M rows Roughly a thousand to one. Disk was 12 GB used out of 20 — on a tool that had recorded half a megabyte of real telemetry. The culprit was ClickHouse's own system logs, several of which have no TTL by default and therefore grow forever: trace_log — 404 MB, 26M rows (the query profiler writes here; it's on by default, sampling once per second) asynchronous_metric_log — 16.6M rows text_log — 132 MB plus query_log , latency_log Only metric_log , processors_profile_log and part_log ship with a TTL. Everything else just accumulates. Then I looked at the insert rate over 30 seconds: trace_log — 227 rows/s asynchronous_metric_log — 157 rows/s text_log — 44 rows/s my application — about 5 rows/s 98.8% of all inserts were ClickHouse narrating its own internals. The part that's expensive beyond disk Here's the number that made me stop. Over the same 30 seconds: rows inserted : 16,222 rows merged : 11,007,643 That's a 1 : 678 ratio. For every row written, the engine rewrote 678 already-sitting rows. The mechanics: MergeTree drops every insert into its own data part, then merges parts into bigger ones so reads stay fast. When the table is small this is

A normal secondary index can answer: status = "active" It cannot efficiently answer: documents containing "distributed database" LioranDB therefore has a dedicated text-segment architecture. Tokenization Text is split on non-alphanumeric characters. Depending on index options, tokens can be normalized to lowercase and filtered through stopwords. "Building Distributed Databases" becomes ["building", "distributed", "databases"] Segment contents A LioranDB text segment can contain several files and structures: Term dictionary Posting lists Document map Document-length norms Optional term positions Bloom filter Segment metadata A posting connects a term to the local documents containing it. "database" → [doc 2, doc 8, doc 19] Positions can record where the term appears inside each document. That enables more advanced query behaviour and phrase-aware features. Global and local document IDs Each segment assigns compact local IDs to its documents. A separate document map translates them back to global document IDs. This keeps postings smaller while preserving the external identity of the record. Bloom filters Each segment also maintains a Bloom filter for terms. Before reading a segment's postings, the query path can test whether the term might exist there. A negative answer is definitive. A positive answer means the segment may contain the term and should be checked. Query modes The text query layer supports modes such as: AND OR It also emits scored documents and metrics including: Query time Postings read Candidate documents Segments searched Full-text search is essentially a specialized database living beside the document database. Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake. Built by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

Disk structures are durable, but updating them for every write is expensive. LioranDB uses memtables to absorb writes before flushing them to the on-disk B+ tree. What is a memtable? A memtable is an ordered in-memory map. In LioranDB, each entry contains either: Value ( bytes ) or: Tombstone A tombstone represents a deletion. The memtable also tracks: Approximate memory usage Minimum LSN Maximum LSN Entry count Put count Delete count The write path A simplified write path looks like this: Application write ↓ WAL durability ↓ Mutable memtable ↓ Immutable memtable queue ↓ Background flush ↓ Disk B+ tree The active mutable memtable accepts new writes. When it crosses a size limit, the engine rotates it into an immutable memtable. That immutable table is no longer modified and can safely be flushed in the background. Why ordered maps? LioranDB uses an ordered map for memtable entries. This helps because the flush process can emit keys in sorted order, which is friendly to the B+ tree and bulk-write paths. It also simplifies range merging between: Mutable data Immutable data On-disk pages Backpressure Background flushing cannot be allowed to fall behind forever. LioranDB therefore tracks limits such as: Maximum immutable memtables Maximum immutable bytes Partition-wide queue limits Maximum writer stall duration If the disk cannot drain the backlog quickly enough, the foreground write path slows down. That may sound undesirable, but controlled backpressure is much safer than consuming memory until the process dies. A memtable is not merely a cache. It is a pressure valve between CPU-speed writes and disk-speed persistence. Without that valve, the engine would either become slow on every commit or dangerously accumulate unbounded work. Built by Swaraj Puppalwar under Lioran Group . Links: LioranDB Lioran Developer Solutions Lioran Group

Most developers think of LioranDB as a document database. Internally, however, its storage engine does not understand documents, objects, fields, or JSON. It understands only: table + key bytes + value bytes That separation is intentional. The architecture LioranDB is split into two major layers: Application ↓ Document DBMS ↓ Transactional key-value engine ↓ WAL, memtables, B+ tree, pager and disk The engine exposes operations such as: get ( table , key ) put ( table , key , value ) delete ( table , key ) scan ( table , range ) The DBMS layer then adds document-oriented features: Collections JSON encoding Queries Updates Secondary indexes Text indexes Transactions For example, a secondary index can be represented as: idx:status:active → document_id A text index can be represented as: inv:database → posting_list The storage engine does not need to know what status , active , or database means. It only stores ordered bytes. Why this matters This architecture keeps the core engine small and reusable. The engine focuses on difficult low-level concerns: Durability Page management Transactions Recovery Ordering Concurrency Range scans The DBMS focuses on application-level semantics. This also makes it possible to build different data models over the same engine in the future. A document database is therefore not one giant component. It is a collection of carefully separated layers. That separation is one of the most important architectural decisions inside LioranDB. LioranDB is being developed by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

Rule would allow states to decide how much—if any—public input there can be.