标签：#Data

I Gave Five Graph Databases 256MB of RAM Each. Here's What Broke. CognoDB Cloud's free tier gives you a graph database instance with half a CPU core and 256MB of RAM. That's not a lot. It's also, honestly, a pretty realistic starting point a lot of real side projects and early-stage products live exactly there, on whatever the free tier happens to give them, and find out the hard way what their database does under pressure. So I decided to actually find out. I took CognoDB and lined it up against four other graph databases Neo4j AuraDB, FalkorDB, and ArangoDB gave every single one of them the same tiny resource budget, threw the same 198,050-edge dataset at all of them, and ran the same queries. No cherry-picking, no "best case" numbers. Just: here's a small VM's worth of resources, go. One of the databases I originally planned to include never even made it into the results. It crashed on startup. Not "slow to start" a full segfault, reproducibly, across two different versions, with nothing I threw at it fixing it. More on that below, because it's honestly one of the more interesting parts of this whole thing. The setup, quickly Five candidates going in: CognoDB (mandatory, since that's the actual point of this), Neo4j AuraDB Free, Memgraph, FalkorDB, and ArangoDB. Same dataset for all of them a real social-graph-shaped dataset from Stanford's SNAP collection, ~18.7k nodes and ~198k edges, sized specifically to fit inside every platform's free tier without anyone getting an unfair advantage. Same queries too: I wrote every single query 1-hop, 2-hop, 3-hop traversals, point lookups, filtered lookups, aggregations exactly once, then translated each one into whatever query language a given platform actually speaks. No platform ever got a "friendlier" version of a query than another. And everyone ran under the same 0.5 vCPU / 256MB RAM ceiling, whether that was their real cloud free tier or a Docker container I capped by hand to match. The one that didn't survive Memgra

A small release, not a launch. The LAB tab on gex.live has a new rightmost rail called IDEA FEED . It is a stream of short, rule-shaped trading ideas about SPX dealer positioning — "fade the first touch of the call wall after a gap up", that kind of thing — collected daily by a scanner from what people actually discuss, rewritten into something the Lab compiler can parse, and published untested . That last word is the point. Why untested is the feature Every feed of trading ideas on the internet comes with a verdict attached: "this works", "78% win rate", a screenshot of a good month. The feed here refuses to do that. Each card says exactly two things about its idea: compiles clean (our compiler turned the text into a runnable rule without complaint) and untested (nobody has run it against the archive yet). The honest test is yours to run. One click drops the idea into the Lab conveyor. The compiler has already done the translation, so the first message in your session is the rule itself, stamped ↳ from IDEA FEED · compiles, untested . Running the backtest costs one Lab credit; a failed job refunds itself. If your balance is zero the button does not go dead — it turns into 0 CREDITS · BUY → , remembers the idea you picked, and comes back to it after. What you will not find No source attribution on the cards. The idea is the unit, not the poster. No win rates, no "rated", no thumbs. The archive is 1,000+ finished SPX sessions; the Lab tests against all of it with an out-of-sample split and tells you what survived, which so far is: very little. That verdict is worth more than a badge on a card. No approval gate. The scanner's finds ship directly every day, so the feed stays fresh by itself. "NEW" is personal — it means new since you last opened the rail, not new for everyone. Why build a feed that mostly produces "no" Because the alternative is pretending. The whole site is built on measuring dealer positioning from the tape instead of assuming it from yesterday's ope

gex.live has two halves. The terminal measures where SPX options dealers are positioned, every second, from the tape. The Lab is the half that asks the uncomfortable question: does any of that predict anything? What it is A browser-side conveyor with three stages and a credit meter. Compile. You describe a rule in plain text — "short the first touch of the put wall when net gamma is below the 20th percentile" — and the compiler turns it into a deterministic rule over the archive's fields: flip, walls, hold band, gamma percentile, DEX/VEX/vanna/charm per strike, time of day. Compiling is free. If the text is ambiguous the compiler says which part, instead of guessing. Backtest. The rule runs against the full session archive — 1,000+ finished SPX days, every one of them public at gex.live/sessions — with a fixed out-of-sample split. One credit per job; a job that fails refunds itself. Quant optimize. Optional. A LightGBM pass over the same feature store to see whether there is structure the hand-written rule missed, reported as out-of-sample AUC plus feature importance, not as a new "signal". The heavy part (DuckDB + LightGBM) runs in a scale-to-zero container that reads snapshots over HTTPS from the public archive. It depends on no machine and on no private data, which is the point: you are testing against the same files anyone can download. The honest-stats rule Every verdict comes with its baseline. "Your rule made 3% in-sample" means nothing next to "the unconditional drift over the same days was 2.8%". The report shows both, shows the out-of-sample half separately, and refuses to produce a headline number from the in-sample half. Most rules do not survive this. That includes our own: the site's own directional levels were tested three separate ways across the whole archive and none held out of sample — which is why the terminal sells measurement and not signals, and why the Lab exists at all. The free Idea Feed Next to the conveyor sits a rail of rule-shaped idea

The yeetcycling math resembles asteroid mining in reverse.

There's a moment every developer hits the first time they connect an AI assistant to a real database: it works beautifully, the model writes a clean SELECT , you get your answer in seconds — and then a small, cold thought arrives. What if it had written DELETE instead? That worry is healthy. An AI agent that can query your production database is also, by default, an AI agent that can UPDATE , DROP , and TRUNCATE it. Large language models are probabilistic. They hallucinate. They misread a vague prompt like "clean up the test users" as an instruction to actually delete rows. You don't want the only thing standing between a confused model and your orders table to be good intentions. The fix isn't to keep AI away from your data. It's to make write operations structurally impossible — read-only by design, enforced at layers the model can't talk its way past. This post walks through how to do that properly, from the database grant all the way up to query-level guardrails. Why "just prompt it to be careful" fails The tempting shortcut is to add "only run SELECT queries, never modify data" to your system prompt and call it a day. Don't rely on this. Prompt instructions are suggestions, not enforcement. A cleverly worded user request, an injected instruction hidden in some data the model reads, or a plain misunderstanding can all lead the model to generate a destructive statement anyway. Real read-only access is enforced below the model — in places where no amount of clever text can override it. Think of it as defense in depth, with at least three independent layers: Layer What it stops Enforced by Database permissions Any write reaching the engine SQL GRANT / REVOKE Connection / replica Writes even being routed to a writable node Read replica, read-only transaction Query parser / broker Non-SELECT statements before they run SQL parsing, allowlists Any one of these is decent. All three together mean a write has to defeat your database engine, your routing, and your parser s

The database platform Harper advocates for a single-runtime architecture that keeps application code and data together, with its benchmark against a Vercel-based stack reporting significantly better performance on live, personalized-data workloads. Harper recently released version 5.2, with a new record cache and more throughput per node. By Renato Losio

Last week I turned my data audit into a build step : a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree. It ended the era of the site contradicting itself. It did nothing about the site agreeing with itself on something false. That's not a hypothetical. The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison. Internal consistency was the camouflage . No diff between my own sources could ever have caught it, because every internal source was equally behind the world. A build gate proves agreement. Agreement is not truth. Something has to look outside. You can't diff against the world, but you can sample it The naive version of "look outside" is another audit — a human session checking primary sources jurisdiction by jurisdiction. I've done three of those now, and I know exactly what they're worth: they're correct the day they ship and they decay from that morning on. Laws don't change on my audit schedule. So the outside check became what the inside check became: a scheduled job. Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row. Fourteen, not all 271, because the selection is doing the real work: A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most. Everything else sits on a rotating cursor : eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents. The whole thing runs on about seven cents a week. Two rules were non-negotiable, both inherited from

@Service public class SimpleUserService { @Autowired private JooqTemplate jt ; public List < user > selectUserInDept ( UserParam param ) { //If deptIDs==null or deptIDs. isEmpty automatically ignores this query condition // SELECT * FROM user_table WHERE name LIKE '%?%' AND dept_id IN (?,?...); return jt . queryv ( "user_table" , User . class , "name%" , param . getName (), "dept_id:in" , param . getDeptIds ()); } public List < user > selectUserNotInDept ( UserParam param ) { // SELECT * FROM user_table WHERE name LIKE '%?%' AND dept_id NOT IN (?,?...); return jt . queryv ( "user_table" , User . class , "name%" , param . getName (), "dept_id:notin" , param . getDeptIds ()); } }

I have a set of React components that render live telemetry: an attitude indicator, a moving map, tapes and gauges, a scrolling event log. They take a data source, subscribe to it, and paint whatever numbers arrive. That works for a live feed. The obvious next thing you want is replay: load a recorded session, scrub a timeline, watch the same instruments play it back. The naive version of this is a trap, and it took me a wrong turn to see why. My first instinct was that replay is a data problem, load the samples, push them into the components in order, done. It compiled, it ran, and the charts were empty. Not broken, not erroring. Empty. The instruments that show a single current value worked fine. The time-series charts sat blank while correct data flowed into them. That empty chart is the whole story of this post, because the reason it's empty is the reason replay is more interesting than it looks. The components are watching a clock you forgot about Here's the data source interface these components consume. It's small on purpose: interface TelemetryValue { timestamp : number ; // wall-clock, unix ms value : number ; channel ?: string ; } interface AltaraDataSource { subscribe ( callback : ( value : TelemetryValue ) => void ): () => void ; getHistory (): TelemetryValue []; readonly status : ConnectionStatus ; destroy (): void ; } A live source stamps each sample with Date.now() as it arrives. A time-series chart, reasonably, assumes that's what timestamps mean: it anchors its x-axis to Date.now() and draws a moving window of the last few seconds, discarding anything older than windowMs because that's off the left edge of the view. Now replay a session recorded an hour ago. Every sample carries its original timestamp, an hour in the past. The chart buffers them correctly, then asks "is this within the last few seconds of now?", the answer is no for every single sample, and it draws nothing. The data is all there. It's just an hour to the left of the visible window,

The AI buildout shows no signs of slowing. And with hundreds of billions of dollars a year going into data centers and GPUs, compute has become the single biggest cost for anyone building AI products. But for all that spending, there still isn’t a straightforward way to put a price on compute — or for firms to hedge their exposure when the price changes. Silicon Data […]

En el post anterior exploramos qué es un Data Lake y por qué son tan importantes en el ecosistema de datos actual. Ahora es momento de ensuciarnos las manos y ver exactamente qué servicios de AWS necesitamos para construir un Data Lake completamente serverless y cómo orquestarlos. Los Servicios Fundamentales Un Data Lake serverless en AWS se construye sobre cinco pilares fundamentales que trabajan en conjunto para crear una solución escalable y costo-eficiente: Storage Procesamiento Catalogo Seguridad Explotación Amazon S3 - El Corazón del Storage S3 no es solo nuestro sistema de archivos, es la piedra angular del Data Lake. Aquí almacenamos tanto los datos crudos como los procesados, y su organización es crucial para el rendimiento y los costos. Estructura de carpetas de un data lake estandar: data-lake-bucket/ ├── raw/ # Datos sin procesar │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ ├── processed/ # Datos transformados │ ├── bronze/ # Limpieza básica │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ │ ├── silver/ # Transformaciones de negocio │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ │ └── gold/ # Datos listos para consumo │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ └── athena-results/ # Resultados de queries Notarás que todo el data lake se encuentra en un mismo bucket, esto es lo más recomendable ya que S3 tiene un límite de 100 bucket que podemos crear por cuenta (no importa la región, ya que S3 es un servicio global) Configuraciones clave en S3: Versionado habilitado para auditoría y rollback Lifecycle policies para optimizar costos (Standard → IA → Glacier) Server-side encryption con KMS para seguridad si es necesario. Cross-region replication para disaster recovery AWS Glue - El Motor de Transformación Glue es suite de servicios de data serverless que maneja tanto el descubrimiento de esquemas como las transformaciones de datos. Componentes principales: Glue Jobs : Herramienta predilecta para ejecutar ETLs, nos permite procesar y transformar los dato

TerraPower's nuclear power plant possesses a strategic advantage over competitors, especially when chasing after data center deals.

WhatsApp is testing Scam Alert in limited beta, using on device machine learning to detect potential scam messages from non contacts. Meta's architecture keeps message content on the device while using confidential computing, Oblivious HTTP, differential privacy, and model transparency to measure performance and protect model delivery. By Leela Kumili

The cyberattack at CareCloud resulted in one of the largest reported data breaches in the U.S. healthcare industry this year.

ICE collected nearly a million DNA samples last year.

The people-search tool ClarityCheck says its reverse image search service is “private and secure”—but it left a database containing more than 9 million image files exposed.

Relativity Networks deals in hollow-core fiber, a rarely deployed technology that allows data to be transmitted 30% faster than conventional fiber.

In my last post I wrote about an error that cost me a full evening: my pom.xml had the MongoDB starter, but my code was still full of JPA annotations. The compiler kept saying cannot find symbol: class Entity . That post was about the error. This post is about the fix — every single line I had to change to move my Task Manager project from MySQL to MongoDB. If you are planning the same switch, this is the checklist I wish I had. 1. The dependency Before (MySQL + JPA): <dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-data-jpa </artifactId> </dependency> <dependency> <groupId> com.mysql </groupId> <artifactId> mysql-connector-j </artifactId> <scope> runtime </scope> </dependency> After (MongoDB): <dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-data-mongodb </artifactId> </dependency> One starter replaces two dependencies. And this is exactly where my problem started — I added the new one but never removed the old one, so half my code still compiled and half did not. Remove the JPA starter completely. If you leave it in, the jakarta.persistence annotations still resolve, and you will not notice you are mixing two worlds until something breaks at runtime. 2. application.properties Before: spring.datasource.url = jdbc:mysql://localhost:3306/taskmanager spring.datasource.username = root spring.datasource.password = yourpassword spring.jpa.hibernate.ddl-auto = update spring.jpa.show-sql = true After: spring.data.mongodb.uri = mongodb://localhost:27017/taskmanager Five lines became one. No ddl-auto because MongoDB has no schema to create. No dialect because there is no SQL being generated. The database and the collection are created automatically the first time you insert a document. 3. The model class This is where most of the work was. Here is my actual Task class after the migration: package com.taskmanager.task_manager ; import com.fasterxml.jackson.annotation.JsonIgnore ; import org.

Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it. The Problem With k-Fold in Time Series Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap . A label at bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future. The model leaks. V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain TimeSeriesSplit is only marginally better; it still lets adjacent windows bleed into each other. Purged + Embargoed CV For each test window [t0, t1] : Purge any train row whose label window overlaps the test window. Embargo max_training_horizon bars after the test window — drop those too. Overlapping labels are not i.i.d. Purging + embargoing makes the split honest. def purged_embargo_split ( n , n_splits = 5 , embargo_frac = 0.02 ): idx = np . arange ( n ) fold = np . array_split ( idx , n_splits ) splits = [] for i in range ( n_splits ): test = fold [ i ] emb = int ( len ( test ) * embargo_frac ) lo , hi = max ( 0 , test [ 0 ] - emb ), min ( n , test [ - 1 ] + emb + 1 ) train_mask = np . ones ( n , bool ); train_mask [ lo : hi ] = False splits . append (( idx [ train_mask ], test )) return splits Tune Only When You Have Enough Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless. Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the date range or symbol basket first; don't trust the trial. Three-Way Split, Always train (fit) → validation (early stop + HP select) → disjoint calibration set (sigmoid/ isotonic) → test (untouched, final score only). V1 sometimes conflated validation and calibration. Keep them separate. The Promotion Gate Log every trial's train/val/test gap, not just the winner's test score. Promote only if replay AND shadow (≥1 live session) both beat baseline on buyer metrics : 1.5x

If you have ever copy-pasted a table out of a PDF, you already know what happens. Rows collapse into one long line of text. Columns interleave. Numbers land in the wrong cell, or no cell at all. The table on the page looks perfectly structured, but a PDF has no real concept of "table." It only knows where individual characters sit on a page. Every extraction tool, ours included, has to reconstruct the table from scratch, using nothing but the position of each word. That gap between "looks like a table" and "is structured data" is where almost every free PDF tool falls apart. Here is how we handle it, what actually works, and where it still doesn't. Two different jobs, two different tools PDFHaul splits this into two separate tools because they solve different problems. PDF to Excel rebuilds the whole document as a single spreadsheet, in the order it appears on the page: form labels, key-value pairs, section titles, and tables all together. It is for documents where you want the full content, not just the numbers, things like invoices, time sheets, and reports. Extract Tables does the opposite. It ignores everything that isn't a table and hands back one clean sheet per table, nothing else. It is for people who want structured data out, ready to sum, sort, and filter, not a copy of the document. Both tools share the same underlying geometry engine. The difference is what each one keeps and what it throws away. How Extract Tables actually decides what's a table The core problem with table extraction is that "looks tabular" and "is tabular" are not the same thing. A vector chart's axis box, a form's outlined signature field, and a two-column list of allergen names all produce something that a naive extractor will happily read as a grid. None of them are tables. Our pipeline handles this in four phases, all before anything is written to a spreadsheet: Phase 1: classify the page. Every page is scored as bordered (has ruled lines or filled-rectangle grid lines), stream (no