标签：#data

AI is expensive, Ali Ghodsi tells TechCrunch. With so many investors wanting into his latest round, he said yes to more than planned.

Part 3 of the Building the AI Memory Stack series After finishing Part 2, I noticed something. The...

Managed databases are the cloud cost line people quietly stop looking at. Compute gets rightsized, storage on the instances gets cleaned, but the RDS, Aurora, or Azure SQL bill just grows, because a database feels too load-bearing to touch. It is not. Here is how I have cut managed database spend without a maintenance window, in the order of least risk to most. The theme throughout: databases give you more no-downtime levers than people assume, and the biggest wins are usually storage and rightsizing, not some exotic re-architecture. Start with the free win: reclaim dead storage Storage is where the surprise terabytes hide, and most of it comes off with zero downtime. Drop what nobody reads. Old audit tables, soft-deleted rows that were never purged, expired sessions, staging data that got promoted to prod years ago. A DELETE in batches plus a purge job is the boring, safe first move. Reclaim space after deletes. On Postgres, deleted rows leave bloat until vacuumed. Run VACUUM (and check pg_stat_user_tables for dead tuples). On SQL Server / Azure SQL, rebuild or reorganize fragmented indexes to reclaim pages. This is where the "reclaimed terabytes" headlines actually come from. Kill redundant indexes. Unused and duplicate indexes cost storage and slow writes. Postgres pg_stat_user_indexes (look for idx_scan = 0 ) and SQL Server's missing/unused index DMVs tell you which ones earn their keep. Dropping an unused index is online. Right-size your storage type. On AWS, moving from gp2 to gp3 lets you provision IOPS and throughput independently and usually costs less for the same performance. The modify is applied without downtime. None of the above requires a window. It is pure hygiene, and on a neglected database it is often the single biggest line-item drop. Rightsize the instance (yes, without downtime) The reflex fear is that resizing a database means an outage. With a Multi-AZ deployment it usually does not. Check if you are oversized first. Pull 30 days of CPU, fre

I spent four months tuning a custom weather ensemble. It was worse than guessing. The fix was not a better ensemble. It was admitting someone already built the right thing and giving it away for free. What I built and why it failed The original weather bot counted forecast members. It pulled raw output from four systems: GFS, AIGEFS, ECMWF IFS, and AIFS. Up to 164 individual simulations per contract. The logic was simple. If at least three of four systems agreed on direction, the bot traded. If they disagreed, it sat out. That sounds reasonable. It was not. I ran 112 settled trades through the system and scored the model with a Brier score. The model scored 0.2858. Predicting the historical base rate, with no model at all, scores 0.2439. Lower is better. My model was worse than making no prediction. The problem was not direction. Direction was right about 60 percent of the time. The problem was confidence. The model spread its probabilities 2.1 to 4.0 times too narrow. It was certain when it should have been uncertain. In prediction markets, confidence sizes your bets. A model that is too confident trades too big on the wrong calls. The confident wrong calls cost more than the confident right ones made. There was also a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm in a way that was not in the data. It was in the model. What I should have done first Before building anything, I should have checked whether the thing I was building already existed in better form. NOAA publishes the National Blend of Models. It blends dozens of forecast systems and applies statistical post-processing no individual model can match. It produces calibrated, bias-corrected, station-level probabilistic temperature guidance. For exactly the stations Kalshi settles on. For free. The NBM already does what I was trying to do by hand. It corrects the biases I was measuring. It produces uncertainty ranges I was approximating with

You've seen the headlines by now. An agent in Cursor wiped a company's production database, backups and all, in about nine seconds. Replit's agent nuked another company's prod. Same shape every time: the agent was sure of itself, the SQL was valid, and nobody was in the loop to say wait. Those are the loud failures. The fix for them is boring and you already know it. Don't hand an agent write access to prod. Read-only by default, propose instead of apply, keep a human on the button. But there's a quieter version that a permissions policy won't catch, and that's the one I want to talk about. Your agent is probably doing it right now. It looks completely fine in the diff. The migration that passes review and still takes the site down Ask an agent to make an email column unique. It writes: ALTER TABLE users ADD CONSTRAINT users_email_unique UNIQUE ( email ); Correct SQL. Does exactly what you asked. It sails through review because there's nothing to see. Then on a users table with any real size, it grabs an ACCESS EXCLUSIVE lock and scans every row to build the unique index, and for the whole length of that scan nothing else can read or write the table. The API starts timing out. The connection pool fills. Now you're in an incident over a one-line migration that everybody approved. The agent didn't do anything a decent junior engineer wouldn't have done. That's the trap. The danger isn't the SQL, it's the lock the SQL takes, and you can't see a lock by reading a statement. You'd have to know Postgres locking cold: which DDL grabs which lock, and for how long, and what it shuts out while it holds. And you'll still miss one at 2am. I got tired of missing them. So I measured one. What the lock actually costs I ran the same schema change two ways against a real Postgres 18. Fifty million rows, twenty connections doing ordinary traffic. The unsafe version was a plain SET NOT NULL , which also scans under ACCESS EXCLUSIVE . The safe version was the NOT VALID then VALIDATE da

An extortion gang known for targeting transportation companies and private equity firms has taken credit for a breach at Uber Freight.

Spotify introduced external indexing architecture for Apache Parquet data lakes that enables low-latency point queries without replicating datasets into operational databases. The approach maps lookup keys to Parquet files and row locations, allowing targeted reads from cloud object storage while supporting analytics, machine learning, AI applications, and online services from the same datasets. By Leela Kumili

By the end of this page you can answer the question that stops most people the first week they write SQL: which row is the best one in each category. You will know OVER and PARTITION BY , the three ranking functions and how each treats a tie, a running total, and LAG for comparing a row to the one before it. It is about twenty-five minutes. Here is what to actually do with it. The next time you write GROUP BY genre and get back a best rating without the name attached to it, stop rewriting the GROUP BY . Add ROW_NUMBER() OVER (PARTITION BY genre ORDER BY rating DESC) to the plain query instead, then keep the rows numbered 1. That is the whole move, and it replaces a query most people never get working. The short version: a window function adds a calculated column to each row while leaving every row in place. Grouping collapses rows. A window looks at them. One idea decides everything else on this page, so it gets the picture. Both halves do the same arithmetic over the same four rows, and only one of them still has four rows at the end. The original carries a diagram here. In words: Two panels side by side, each starting from the same stack of four identical row shapes. The left panel is labelled GROUP BY. Its four rows funnel down through a single arrow into one row at the bottom, and the four original rows are shown faded to indicate they are gone from the result. Only one row remains. The right panel is labelled OVER. Its four rows stay exactly where they are, at full strength, and each one gains a small badge on its right hand side holding a number: one, two, three, four. Nothing funnels and nothing is faded. The contrast is the whole idea: the left panel ends with a single summary row and no way to say which original row it came from, while the right panel ends with all four rows still present, each carrying its own calculated value. The worked example is real. Every number on this page comes from a published portfolio project: finding the genuinely overlooked g

By the end of this page you can write a summary query and know its answer is real. You will know exactly what GROUP BY does to your rows, which columns you are allowed to select afterwards and why, where WHERE goes, where HAVING goes, and why swapping them is the difference between a finding and a number that means nothing. It is about twenty-five minutes. Here is what to actually do with it. On the next summary query you write, add one line setting a minimum group size before you read the ranking. One line, and it removes the most common way a summary query produces a confident wrong answer. The short version: WHERE filters rows before grouping. HAVING filters groups after. Without a HAVING floor, tiny groups float to the top of every ranking. One idea decides everything else here, so it gets the picture. Grouping happens in the middle of the query, and the two filters sit on opposite sides of it. The original carries a diagram here. In words: A left-to-right pipeline in four stages. Stage one is a column of eight individual row boxes. Stage two is a gate labelled WHERE, through which six rows pass and two are crossed out and stopped. Stage three shows the surviving six rows collapsing into three group boxes, one holding three rows, one holding two rows, and one holding a single row. Stage four is a second gate labelled HAVING, through which the group of three and the group of two pass, while the group holding only one row is crossed out and stopped. The result at the far right is two groups. The picture shows that WHERE acts on individual rows before any grouping exists, and HAVING acts on whole groups after they have been formed, which is why the two filters cannot be swapped. The worked example is real. Every number on this page comes from a published portfolio project: 82,956 games from the Steam catalogue, with review counts, ratings and genres. The queries run against the full dataset at Steam Hidden Gems on GitHub . If SELECT and WHERE are also new, start wi

By the end of this page you can say, out loud and in your own words, what every core piece of SQL does. What a table and a row really are. The six clauses, and the order they actually run in, which is not the order you type them. NULL , and why it breaks comparisons. Filtering, aggregation, GROUP BY and HAVING . Joins. CASE . Subqueries, CTEs and window functions. Keys and indexes. That list is most of what an analyst job, an interview, and a first real dataset will ask of you. Here is what to actually do with it. Go through once end to end without stopping, just for the shape. Then come back to the retrieval sheet near the bottom, cover the right-hand column, and try to say each answer before you read it. That second pass is where the learning happens, and there is measured evidence for it further down. The short version: SQL is one sentence with six parts, and every part answers a different question about your rows. Learn what each part does and where it runs, and the rest is vocabulary. One idea decides more of your SQL experience than any other, so it gets the picture. You write a query in one order. The database runs it in a different order. Almost every confusing SQL error is that gap. The original carries a diagram here. In words: Two columns of stacked boxes face each other. The left column, headed "you write", lists the clauses in typing order from top to bottom: SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT. The right column, headed "it runs", lists the same clauses in execution order: FROM, WHERE, GROUP BY, HAVING, SELECT, ORDER BY, LIMIT. Curved lines connect each clause on the left to the same clause on the right. Six of the seven lines run roughly straight across. One line, the one belonging to SELECT, is drawn in a strong accent color and sweeps steeply downward from the very top of the left column to the fifth position on the right, showing that SELECT is written first but runs almost last, after grouping has already happened. What this page

This guide gives you a test that takes ten seconds and tells you whether the thing you just built is a report or an analysis. Then it gives you four moves that turn one into the other. Every move has a worked SQL example and real numbers. The whole method is here. What you actually do: take the number you just produced, and ask what someone would do differently because of it. If the honest answer is nothing, you have a report. Then you run the four moves below, in order, until the answer is a specific action a specific person can take on Monday. The short version. Data analysis is looking at records of things that already happened and finding a pattern that changes what someone does next. If nothing changes, it was not analysis. It was a report. The same starting number, two endings. The test: what would someone do differently? Before you read the answer, look at the last thing you built and try it yourself. Who was going to act on it, and what were they going to do? Take any number you have produced and finish this sentence out loud: "Because of this, someone should do a specific thing ." Both blanks have to fill in with something real. A named person or team, and an action they control. Here is a real one. "Churn was 4.1% in Q3." Who acts, and how? Nobody can act on that. It is a true, correctly calculated, carefully formatted number, and it changes nothing. That is a report, and reports are useful. A dashboard that tells you the servers are up is doing its job. It is just not analysis. Now the same underlying data, worked further. "Monthly-plan accounts that never opened the import tool churn at 9.2%. Ones that did churn at 1.8%. The email introducing that tool goes out on day 14, and most cancellations happen on day 11." Who acts? The lifecycle marketing owner. What do they do? Move the email to day 3. That is analysis, and the only difference is that it ended somewhere a person can stand. The word "analysis" is doing a lot of quiet work in job descriptions, so

Search has got more powerful, but also more complicated It used to be that database queries were simple enough. Either you had an index, or you didn't, and either way you had some kind of optimizer (cost or rule) that turned your SQL statement into a viable plan for finding and returning your data. We now live in a world where in addition to the Boolean logic of traditional RDBMS queries, we also have: Geospatial queries Full text search queries, using BM25 . Vector Search And if that weren't enough, we have to consider that instead of a traditional application issuing the query, it might be an MCP server, and most importantly of all, the business need might be for two or more of these searches to happen at the same time, on the same data. For example: "An MCP server that uses a single, combined geospatial + full text query to identify towns in Bavaria with castles mentioned in text descriptions." How does CrateDB help with this? CrateDB is one of the limited number of products that not only supports all of these search types but is also capable of storing arbitrarily large quantities of data. This is important, as if you have to hit multiple different database servers to solve your business question, not only is your environment much more complicated, but you risk getting incorrect answers as your multiple databases may be out of sync. In this two-part example, based on our playable IOT Analytics scenario, we will show two things: Using Geo + Text to search a weather/tourism database Using CrateDB as a '360 view' for your MCP server If you want to follow along with this post, the setup is based on our IoT Analytics scenario. You can also just skip this blog post and jump straight to the scenario. Using Geo + Full text to search a weather/tourism database The table we're going to use is called ' German Regions ': CREATE TABLE IF NOT EXISTS demo . german_regions ( region_name TEXT PRIMARY KEY , geo_coords GEO_SHAPE , tourism_info TEXT INDEX USING FULLTEXT WITH ( anal

Roblox exposes game and group stats through public, unauthenticated endpoints — no login, no scraping tricks: GET https://games.roblox.com/v1/games?universeIds=<id>,<id>,... GET https://games.roblox.com/v2/groups/<groupId>/games?limit=50 I used them to pull the full public games list for a few independent creator groups that each ship multiple games in the same cheap-to-build "obby" template genre (think: dozens of studios building the same core traversal loop with a different skin). The question was simple: within one studio's own catalog, how concentrated is traffic in the single best title versus everything else they've shipped? The answer is the same shape every time: a small number of throwaway builds with near-zero traffic, and one outlier that accounts for nearly all of the studio's lifetime visits. Not "most games do okay and one does great" — more like one game is the studio, traffic-wise, and the rest are lottery tickets that didn't hit. As a sanity check against numbers that are already public knowledge (no anonymity concern), I ran the same script against Uplift Games' group (id 295182): $ python3 fetch_group_stats.py 295182 Group 295182: 371 published experiment(s) Total lifetime visits across all games: 44,412,921,224 Top title alone: 44,377,094,324 visits (99.9% of the group's total traffic) Visit-count distribution: 0-10K: 355 game(s) 10K-500K: 12 game(s) 500K-5M: 2 game(s) 5M-50M: 1 game(s) > 50M: 1 game ( s ) One title (Adopt Me) is 99.9% of that group's entire lifetime traffic across 371 shipped experiments. Same power-law concentration as the smaller, anonymized groups in the full writeup — just at a much larger scale. Why this is more than a curiosity : if you're building in a genre like this, the template itself is clearly not the moat — everyone in it ships near-identical mechanics. The variance between a 45-visit build and a 700M-visit build using the same template looks like it's mostly about timing and whatever the discovery algorithm rewar

"What if satellite data could help protect the livelihoods of millions who depend on Africa's largest lake?" Last weekend, our team JONAM had the privilege of participating in the Kijani Space Hackathon , where we proudly secured 3rd place while tackling Challenge 2: Sustainable Fisheries & Blue Economy . Rather than building another dashboard, we wanted to solve a real problem affecting millions of people around Lake Victoria : declining fish stocks caused by worsening water quality . The Problem Lake Victoria supports millions of people through fishing, transportation, agriculture, and tourism. However, over the years the lake has experienced: Increasing water pollution Poor water quality Frequent algal blooms Reduced fish breeding habitats Declining fish populations For fishing communities, these are not just environmental issues—they directly affect livelihoods, food security, and local economies. Our question became: Can Earth observation data help communities understand where water conditions are becoming unsuitable for fish before the problem becomes critical? Our Solution: JONAM JONAM is an AI-powered web application that combines satellite-derived environmental data with machine learning to monitor water quality and provide insights into conditions that may contribute to declining fish stocks. Instead of relying solely on manual sampling—which is expensive and only covers small areas—our platform continuously analyses satellite observations covering the entire lake. Why Copernicus? To build JONAM, we integrated the KijaniBox API , which provides access to environmental datasets from the Copernicus Programme . Copernicus is the European Union's Earth observation programme. It uses a constellation of Sentinel satellites together with in-situ observations to monitor Earth's atmosphere, land, and oceans. For our project, we focused specifically on live water telemetry variables available through the KijaniBox platform. 1. Water Temperature Satellites measure th

If you clip short-form video for money, you know Whop Content Rewards: hundreds of live campaigns paying $0.15–$20 per 1,000 views. The discover page lets you sort by budget. That sort is quietly costing you nights of work. Here's the number that changed how I pick campaigns: on the live board right now, 21% of active campaigns have never paid out a single cent. Big banner budget, $0 actually spent. A "$30,000 budget" campaign that has paid nobody in three weeks is not a $30,000 opportunity — it's a landing page. The problem: the board doesn't show you payout speed. You can see budget and budget left , but not how fast the money is actually moving — and that's the only number that separates a campaign that pays from a campaign that poses. The trick: the page already contains everything you need Every campaign card on Whop publishes three things: when it was funded, how much has been spent, and how many creators joined. From one snapshot — no monitoring, no state between runs — you can derive: dailyBurnUsd = budgetSpent / daysSinceFunded → is money moving? estimatedDaysLeft = budgetLeft / dailyBurnUsd → will it still be there? payoutPerCreatorUsd = budgetSpent / creators → what did the average clipper earn? budgetPace = "draining" | "healthy" | "slow" | "stalled" That last field is the shortcut. On today's board of 456 campaigns: pace meaning what to do draining <3 days of budget left skip — gone before your clip gains traction healthy 3–60 days this is where you clip slow 60–180 days fine, but budget may outlive the campaign stalled >180 days at current burn the "big budget" mirage — money posted, almost nobody paid null zero paid out so far unproven; could be brand new, could be dead Real example from today: two campaigns, both showing ~$30K budget. One burns $255/day and has paid the average creator $75 . The other burns $19/day — at that rate its budget lasts four years , which is a polite way of saying nobody is getting paid. On the default board they look ident

"Why not just use a cron job?" is the first question I get whenever someone sees an Airflow DAG. Fair question. Cron works. It's been around for decades. It's simple. The real answer isn't that cron is bad — it's that cron solves a different problem than Airflow does. Cron is a job scheduler . It runs a command at a fixed time. That's it. It doesn't know whether the command succeeded, whether its dependencies are satisfied, or whether it should even run at all today. It just fires the command and moves on. Airflow is a workflow orchestrator . It doesn't just schedule tasks — it models them as a graph of dependencies, tracks their state, retries failed ones, and gives you a UI to see what ran, what failed, and why. Here's where that difference actually matters. The problem cron can't solve Imagine a simple ETL pipeline: Extract raw data from an API Validate and clean it Load into a warehouse Run a transformation Send a Slack alert if anything fails With cron, you'd write five separate cron entries, one per step, and hope the timing works out. If step 2 fails but step 3 runs anyway, you now have bad data in your warehouse. If step 4 takes twice as long one day, you've silently broken your SLA. Nobody gets notified unless you manually add alerting logic to every script. With Airflow, you model this as a DAG: from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime with DAG ( dag_id = " daily_etl " , schedule = " 0 6 * * * " , start_date = datetime ( 2026 , 1 , 1 ), catchup = False , ) as dag : extract = PythonOperator ( task_id = " extract " , python_callable = extract_data ) validate = PythonOperator ( task_id = " validate " , python_callable = validate_data ) load = PythonOperator ( task_id = " load " , python_callable = load_to_warehouse ) transform = PythonOperator ( task_id = " transform " , python_callable = run_transformation ) extract >> validate >> load >> transform Airflow guarantees the order. If validate fail

The Quest Begins (The "Why") I still remember the first time I faced the “Next Greater Element” interview question. The array looked innocent enough, but every brute‑force attempt felt like I was hammering a nail with a sponge— O(n²) time, nested loops, and a sinking feeling that I was missing something elegant. I spent an hour sketching out the problem on a whiteboard, muttering, “There has to be a way to look ahead without looking back every single time.” That frustration is a rite of passage for many developers. We’re taught to think in terms of scanning left‑to‑right, but some array puzzles scream for a different perspective: we need to remember what we’ve seen in a way that lets us answer questions about the future elements instantly. Enter the monotonic stack—a deceptively simple data structure that turns those scary “look‑ahead” problems into straight‑line walks. The Revelation (The Insight) So what’s the secret sauce? A monotonic stack is just a stack that maintains its elements in strictly increasing or strictly decreasing order. Why does that help? Consider the Next Greater Element problem: for each index i , we want the first element to its right that’s larger than arr[i] . If we walk from left to right and keep a stack of indices whose next greater element we haven’t found yet, the stack will naturally be decreasing in value. Why decreasing? Imagine the stack holds indices [i₁, i₂, …, i_k] where arr[i₁] > arr[i₂] > … > arr[i_k] . When we encounter a new value arr[j] , any element on the stack that is smaller than arr[j] has just found its next greater element—namely arr[j] . We pop those indices, record the answer, and stop when we hit a value that’s not smaller (or the stack empties). Then we push j onto the stack. Because each index is pushed once and popped at most once , the total work is linear: O(n) . No nested loops, no repeated scans—just a single pass with a stack that does the heavy lifting. The same invariant works for other “first bigger/smal

Running LLM jobs over hundreds of items, the obvious shortcut is to collapse execution and verification into one model pass: one call, one output, ship it. It fails at scale, and a scoring system for 146 countries across 11 categories shows exactly why. Each score runs 0 to 100 on a single canonical dataset, the overall rating is the arithmetic mean of those 11, and there are no per-country exceptions. One yardstick, applied identically everywhere. Ask a cheap model to generate all 146 in one pass and you get speed with a hidden cost: drift. One country's "friendliness" score reads high because the model read it as social warmth rather than visa bureaucracy. Another's culture score inflates after the prompt happened to emphasize food over history. None of these are bugs, they're quiet inconsistencies, and at 146 items a 5% drift rate means seven countries silently failing the canonicity requirement while every individual score still looks reasonable. The pattern Step one: a cheap executor runs the mechanical pass. Fixed ruleset, all 146 countries in parallel batches, structured JSON out. No judgment calls, just apply rule X to field Y. Step two: a stronger gate verifies before anything ships. Same scale everywhere? Any statistical outlier? Did a category get reweighted mid-run? This is judgment work, holding many items in view at once, and it's what a single combined pass can't do reliably. A model doing both jobs at once optimizes for the wrong thing: it second-guesses the ruleset mid-run, adds nuance where the spec demanded consistency, and marks cases "exceptional" that shouldn't be. Splitting the two roles is faster and cheaper than one model trying to hold both contexts simultaneously. Where the consistency requirement bites The Country Comparison Tool's best-travel-months field works the same way: a month qualifies if it scores 70 or higher on a fixed weather index built from Open-Meteo data, no editorial override, no "tourists usually go in December anyway."

New research finds that by making the fossil fuel industry more productive, AI could help increase carbon emissions by up to nearly 5 percent—vastly outpacing the impact of data centers.

IBM and Red Hat have announced an expansion of Lightwell, introducing new commercial offerings designed to help organizations establish trusted, verifiable software supply chains for the age of AI-assisted software development. By Craig Risi