标签：#data

Anthropic and OpenAI are racing to scale up while reducing dependence on Nvidia.

You've done it. You're finally done building the website or application you've been working on for quite a while. Proud and elated, you go to share this on your socials or to your buddies — uh oh, what's this now? The preview in WhatsApp shows no headline, your avatar is cropped and dimensions seem wrong. I've been there too. The site looked fine in the browser. The problem was everything outside the browser: link previews, search snippets, and tab icons all use a separate metadata layer most of us skip until something breaks. So, how do you fix this? Use this as a pre-launch checklist — or run it on a site that's already live but sharing badly. What I ran into on my own portfolio When I ran this audit on shwethaadiraj.com , the site rendered fine — but sharing it told a different story. I had pointed both the favicon and Open Graph image at my profile avatar. At tab size the illustration was unreadable; in link previews it got cropped awkwardly. An OG validator then flagged two things I hadn't considered: the image was 512×512 (most platforms expect 1200×630 ), and there was no headline or CTA on the image itself — so Slack and LinkedIn showed a plain square with none of the context from my meta title. I replaced the favicon with a simplified monogram, regenerated the OG image at the correct aspect ratio with my name, tagline, and site URL on it, and re-ran the debuggers. Even then, previews didn't update until I hit Scrape Again — platforms cache OG data aggressively, so fixes on your end won't show up until you bust that cache. None of this required rethinking the app. It was a metadata pass — the kind of work that's easy to defer and annoying to discover at the share button. Before we get into the specifics, here's a primer on what metadata can actually impact: What is metadata for? Metadata, simply put, is data about data. Search engines, crawlers and social sites all parse different metadata from your app. Search & Discovery: The title and description in your

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible." This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic. 1. The Actionability Gap (The Golden Question) The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is: "Does the company have the operational capacity to make a decision in milliseconds?" If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch. 2. The Hidden Complexity and the Cloud Bill Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug. Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule. 3. "Micro-Batch" Solves 99% of Your Problems There is a perfect middle ground that the hype industry tries to ignore: the micro-ba

GitHub malware advisories no longer stop at npm. Here's how we wired OpenSSF's malicious-packages data into the Advisory Database, and why we built the pipeline paranoid. The post How we took malware advisories beyond npm appeared first on The GitHub Blog .

Connor Moucka pled guilty to hacking and stealing data from more than 165 Snowflake customers, which net him and his accomplices more than $2.5 million in ransom payments.

If your files already live in Amazon S3, the pitch for storage that branches with your database is appealing but the word "migration" makes it sound like a project. It mostly is not. Neon's object storage speaks the S3 API, so the code you already wrote, the AWS SDK calls and presigned URLs, keeps working. What changes is how you point the client and where the bucket comes from, and that is a small, mechanical diff. The actual data move is a copy loop you can run once. The one thing to do up front is confirm the object operations your app actually relies on: the demo here exercises PutObject , GetObject , listing, and presigned URLs, and I flag the S3 features you should check for yourself further down. This post is the practical version: what stays identical, the exact config that changes, a script to copy the objects across, and an honest list of the S3 features that do not have an equivalent so you know what to check before you commit. The repo with the working client is at the end. TL;DR Neon object storage is S3-compatible. Your @aws-sdk/client-s3 code for the common operations, PutObject , GetObject , getSignedUrl , listing, works unchanged (these are what the demo verifies). Confirm anything beyond that, like multipart for large objects, against the current preview. The diff is the client config: point endpoint at the Neon storage endpoint, pin region: 'us-east-2' , set forcePathStyle: true . The bucket is declared in neon.ts instead of created in the console, and credentials are injected per branch. Move the data with a list-and-copy loop between two S3 clients (source AWS, destination Neon). What does not carry over: S3 bucket policies, event notifications and Lambda triggers, storage classes and Glacier transitions, and cross-region replication. Object CRUD and presigning do. The payoff is everything else in this series: once the files are on Neon, they branch with your database. Prerequisites An existing S3 bucket and credentials that can read it A Neon p

If your app stores files and you want real preview environments, you eventually hit the same wall: each preview needs its own storage, so you start provisioning a bucket per environment. That sounds cheap until you write it down. For every ephemeral environment you create a bucket, attach a policy, mint an IAM role or access keys, set CORS, add a lifecycle rule so it eventually cleans up, wire the credentials into the preview's config, and register a teardown step for when the PR closes. Then you find the orphaned buckets the teardown missed, months later, still billing. The reason this is painful is that the bucket is a separate resource from the database, so it needs its own lifecycle. Neon collapses that: the bucket is declared as part of the branch, so it is created and destroyed with the branch and needs no per-environment provisioning at all. This post compares the two approaches and shows the branch version working with no bucket-management code in sight. The repo is at the end. TL;DR Isolated storage per preview usually means provisioning a bucket per environment: policy, IAM, CORS, lifecycle, credential wiring, teardown. It is slow, it drifts, and it leaves orphaned buckets that keep costing money. On Neon the bucket is declared once in neon.ts . Creating a branch brings the bucket (with a copy-on-write copy of the files) and injects scoped credentials; deleting the branch removes it. There is no per-environment bucket to create, no IAM role to mint, and nothing to orphan. Copy-on-write means fifty preview buckets do not cost fifty times the storage, only what each one changes. Prerequisites A Neon project on the platform preview (object storage, us-east-2 ) The Neon CLI, and a CI system that opens/closes preview environments Familiarity with S3 buckets and IAM if you have done the manual version The per-environment bucket, written out Here is what "just give the preview its own bucket" actually expands to, per environment: Create a bucket with a unique nam

The naive way to accept file uploads is to POST them to your API, let the server read the bytes, and write them to object storage. It works until the files get large or the traffic gets real. Now every upload crosses your infrastructure twice, once from the client to your server and once from your server to storage, and your server holds the whole file in memory or on disk while it does. On a serverless function it is worse, because functions have request-size and duration limits that a big upload runs straight into. Presigned URLs are the standard fix, and they predate serverless by a decade. Your server does not move the bytes; it hands the client a short-lived, pre-authorized URL and the client uploads directly to object storage. The server only issues permission and records metadata. On a Neon Function this is the same AWS S3 SDK you already use, pointed at the branch's storage endpoint. This post builds it and tests the whole round trip. The repo is at the end. TL;DR Proxying uploads through a function sends the bytes across it, burning bandwidth and memory and hitting request-size limits. A presigned URL is a time-limited, pre-authorized link to one object key. The client PUTs the bytes straight to storage; the function never touches them. On Neon Functions you generate it with getSignedUrl from @aws-sdk/s3-request-presigner , the same code as any S3-compatible store. I tested the full flow: presign, the client PUT straight to storage returned 200 , a metadata record was saved, and downloading the object returned the exact bytes. One gotcha to pin: the injected AWS_REGION is the storage-cell host, not a region, so set region: 'us-east-2' on the client. Prerequisites A Neon project on the platform preview with a declared bucket (object storage, us-east-2 ) The AWS SDK: @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner Familiarity with S3-style object storage and HTTP PUT Why not just proxy the upload Sending the file through the function has three costs that

Chevron and Williams are big winners in the race to power artificial intelligence as they build out gas-fired power plants and pipelines.

Wiz Research disclosed CosmosEscape, a chain that escaped Azure Cosmos DB's Gremlin sandbox and reached a platform-wide key granting read and write access to every database on the service. Microsoft blocked the entry point within two days but took until July 2026 to remove the key. Practitioners debated shared responsibility and what that rearchitecture actually cost. By Steef-Jan Wiggers

AI workflows have two needs that trade off directly. Running reliably in production requires persisting and distributing every step so it survives crashes, deploys, and restarts. But that same machinery is what makes runs too heavy for the fast, throwaway loop you need to check an LLM's output quality. The properties that buy durability are the ones that kill iteration speed. By Mateus Moury

Vercel Labs has introduced Zero, an experimental systems programming language aimed at AI rather than human users. It employs unique features like a specific toolchain contract and structured error messages. Reaching version 0.3.4, it compiles to native binaries for major operating systems. The language prioritizes size, speed, and agent usability, though it is still in development. By Daniel Curtis

Last month I wrote about building a 1,800-page calculator site solo. Since then the recipe hub on that site grew to 501 dishes from 127 countries — and today I'm releasing all of it as an open dataset. Download JSON (full dataset, ~2.6 MB): https://theunitools.com/data/unitools-recipes-v1.json CSV (one dish per row): https://theunitools.com/data/unitools-recipes-v1.csv Docs + sample record : https://github.com/farcrak/unitools-recipes Dataset page: https://theunitools.com/en/data What's inside 501 home-cooking recipes, 127 countries, bilingual (English + Russian, both written by hand — no machine translation) Per-serving nutrition (calories, protein, fat, carbs) on every single dish 3,200+ steps, each annotated with minutes Ingredients with stable ids and scaling rules : meat scales linearly with servings, salt and spices are damped — the way an actual kitchen scales a recipe, not naive multiplication Human-reviewed Wikimedia Commons photos with author + licence per photo Why the scaling rules matter Most recipe datasets store "2 tbsp salt for 4 servings" and leave scaling to you. Multiply salt linearly to 16 servings and the dish is inedible. Each ingredient in this dataset carries a scaling field ( linear | damped | fixed ), so a portion calculator can be built directly on top of the data. That's exactly how the recipe pages on the site work. Licence CC BY-SA 4.0 — free for commercial use. Credit "UniTools — theunitools.com" and share derivatives under the same licence. Photos carry their own Commons licences (in the data). Honest caveats Nutrition is computed from ingredients, not lab-measured — a planning reference, not medical data. The dataset is maintained by one person; if you spot an error, open an issue on the repo and the fix lands in the next version. If you build something with it — a meal planner, a viz, a model fine-tune — I'd genuinely love to hear about it in the comments.

Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery Ten times in a row I predicted what my next submission would score before I uploaded it. The worst miss was 0.0025 on a number around nineteen. I took that as confirmation that the physics underneath was correct. It was confirmation that I can do arithmetic. Two days before this competition closed I pointed a review at my own endgame, expecting notes about the code. It came back with three errors and none of them were in the code. All three were in my reasoning, and all three had the same shape: I had run something that felt like a measurement and was not one. This is the fourth entry in this series and the one I would keep if I had to burn the other three. The models are competition-specific. This part is not. The competition in one breath Perseverance carries an environmental station called MEDA. Some of its surface pressure readings are missing, and the competition is to reconstruct them. Scored on mean squared error. The wrinkle is the split. Training covers sols 1 through 100, when pressure is climbing toward its seasonal peak. Test covers sols 201 through 300, when it is falling hard toward the aphelion minimum. Sols 101 through 200 do not exist in either file. Every prediction is outside the range the model was fit on. The first entry covers the first submission, which contained no machine learning at all and took the top of the board at 61.04. Six weeks and seven versions later the public score was 18.99. Almost everything in between was selected by one signal. Not cross-validation. Cross-validation here can only hold out sols from the rising limb, so it is structurally blind to the regime I am scored on. The leaderboard was the only thing that could see the falling limb, so the leaderboard picked every scalar that mattered: the residual shrink, the blend weight, a constant seasonal offset, a diurnal scaling. Hold onto that. It becomes the joke about four hundred words from

The query every backend developer has needed and nobody enjoys writing In March 2016, npm removed an 11-line package called left-pad. Within minutes, builds began failing across the JavaScript ecosystem. It broke thousands of projects, including tools like Babel. Many developers didn't choose left-pad directly; it was hidden in their dependencies and went unnoticed until it vanished. That incident points at a question you have probably asked about your own stack: what is actually in my dependency tree? Not just the 30 packages in your package.json . Everything they pull in, and everything those pull in, all the way down. In a relational database, dependencies live in a self-referencing join table. "Everything, all the way down" means a recursive CTE. Here is that query on a snapshot of the npm registry. It finds the full runtime dependency tree of express : WITH RECURSIVE closure ( name , depth ) AS ( SELECT depends_on_name , 1 FROM dependencies WHERE package_name = 'express' AND dep_type = 'runtime' UNION SELECT d . depends_on_name , c . depth + 1 FROM closure c JOIN dependencies d ON d . package_name = c . name AND d . dep_type = 'runtime' ) SELECT count ( DISTINCT name ) AS transitive_deps , max ( depth ) AS max_depth FROM closure ; Here is the result: It works. Here it shows 63 packages, 11 levels deep. But it is twelve lines, and every line matters. There is an anchor part, a recursive part, and a UNION doing quiet work to remove duplicates. A graph asks the same question in two lines: MATCH ( :Package { name: 'express' }) - [ :DEPENDS_ON * ] -> ( dep ) RETURN count ( DISTINCT dep ) AS transitive_deps That is not a shortened excerpt. That is the whole query. It returns the same 63 packages. Cypher is Neo4j's query language. For a SQL developer, it is less a new language than a new notation for questions you already know how to ask. This article proves that claim with ten translations. They run from "this is just SQL with arrows" up to the query above, plus one

JioHotstar explains the distributed architecture behind its real-time ad request workflow, covering ad decisioning, waterfall tiering, pacing algorithms, latency optimization, and service coordination required to select and deliver personalized advertisements during streaming playback at scale. By Leela Kumili

Originally published at Programming Tech Lab . Welcome to the Garage: What is Linear Regression? Step away from the kitchen counter and step into a bustling auto garage. Imagine you are an experienced mechanic evaluating used cars brought in for trade-ins. A customer drives in a sedan with 50,000 miles on the odometer and asks: "How much is my car worth?" Without needing a complex computer program, your brain instantly draws a connection: as the mileage on a car goes up, its resale price goes down. If a car has 0 miles (brand new), it commands peak market price. If it has 200,000 miles, it drops significantly toward scrap value. This straight-line relationship between two factors—where changes in one variable cause a predictable increase or decrease in another—is the core concept behind Linear Regression . Deconstructing the Formula (Without the Headache) In high school math, you probably saw the classic line equation: y = mx + b In machine learning, Linear Regression uses this exact same formula to make predictions: Predicted Value (y) = ( Slope m × Input Feature x ) + Starting Point b Let's map this directly to our mechanic's garage evaluation: Target (y): The estimated resale price of the car ($). Input Feature (x): The total miles on the odometer. Starting Point / Intercept (b): The price of the car when mileage is 0 (Brand New MSRP). Slope / Weight (m): The rate of depreciation (e.g., losing $0.10 in value for every 1 mile driven). If a car starts at a baseline price of $30,000 and depreciates by $0.10 per mile, a car with 50,000 miles is predicted to be worth: Predicted Price = $30,000 - ($0.10 × 50,000) = $25,000 How the Algorithm Draws the Perfect Line: Least Squares If you plot 100 used cars on a graph where the horizontal axis (X) is Mileage and the vertical axis (Y) is Price, the dots won't form a perfectly straight laser line. Some owners took great care of their vehicles; others had minor scratches. So how does a Linear Regression algorithm draw the sin

Exactly a year ago we released SurrealMCP, a server you had to install and run in order to connect...

Most Snowflake-to-Databricks migrations get sold on cost and delivered on something else. The credit line item is what gets the project funded, but the teams that finish happy are usually the ones that moved for a different reason: they wanted ML, streaming and GenAI workloads living next to the analytics data instead of shuttling between two platforms. If your only justification is the bill, read the breakeven section below before you commit — the honest number is longer than the deck says. We're a Databricks shop , and we've written elsewhere about how to choose between the two platforms if you haven't committed yet. This post assumes you have. What actually changes underneath The two platforms look similar from a SQL console and are structurally different behind it. The mapping worth internalising before planning anything: Layer Snowflake Databricks Storage Proprietary micro-partitions inside Snowflake Delta Lake files in your own S3/ADLS/GCS bucket Compute Virtual warehouses, T-shirt sized Job clusters, all-purpose clusters, SQL Warehouses, Photon Governance Role hierarchy, row access policies, masking policies Unity Catalog across tables, models, notebooks, dashboards Sharing Secure Data Sharing Delta Sharing (open protocol) Billing unit Credits DBUs, priced differently per compute type The storage row is the one with the most downstream consequences. On Snowflake, storage and compute are separate line items on the same bill; on Databricks, storage is your cloud provider's problem and your cloud provider's invoice. That's a genuine benefit — the data stays readable by other engines — but it also means your "Databricks cost" and your "data platform cost" stop being the same number, and finance needs to know that before the first invoice arrives. Pick a strategy before you pick a tool Three patterns, and the choice determines everything after it: Lift-and-shift. Replicate schemas one-to-one, translate the SQL, cut over. Fastest, and it faithfully preserves every

Governor who touted Texas as AI “epicenter” pauses data center grid connections.