标签：#base

Search has got more powerful, but also more complicated It used to be that database queries were simple enough. Either you had an index, or you didn't, and either way you had some kind of optimizer (cost or rule) that turned your SQL statement into a viable plan for finding and returning your data. We now live in a world where in addition to the Boolean logic of traditional RDBMS queries, we also have: Geospatial queries Full text search queries, using BM25 . Vector Search And if that weren't enough, we have to consider that instead of a traditional application issuing the query, it might be an MCP server, and most importantly of all, the business need might be for two or more of these searches to happen at the same time, on the same data. For example: "An MCP server that uses a single, combined geospatial + full text query to identify towns in Bavaria with castles mentioned in text descriptions." How does CrateDB help with this? CrateDB is one of the limited number of products that not only supports all of these search types but is also capable of storing arbitrarily large quantities of data. This is important, as if you have to hit multiple different database servers to solve your business question, not only is your environment much more complicated, but you risk getting incorrect answers as your multiple databases may be out of sync. In this two-part example, based on our playable IOT Analytics scenario, we will show two things: Using Geo + Text to search a weather/tourism database Using CrateDB as a '360 view' for your MCP server If you want to follow along with this post, the setup is based on our IoT Analytics scenario. You can also just skip this blog post and jump straight to the scenario. Using Geo + Full text to search a weather/tourism database The table we're going to use is called ' German Regions ': CREATE TABLE IF NOT EXISTS demo . german_regions ( region_name TEXT PRIMARY KEY , geo_coords GEO_SHAPE , tourism_info TEXT INDEX USING FULLTEXT WITH ( anal

If you built a Supabase app quickly - with an AI coding tool or by hand - the row-level-security policies were often written last, or generated for you. That is fine. What is not fine is shipping without knowing whether those policies actually isolate one user's rows from another. Supabase ships a database linter, and you should run it first - it is free and it catches the obvious cases: RLS switched off, and RLS switched on with no policy behind it. But a linter checks whether a policy exists , not whether the policy is correct . Those are different questions, and the second one is where cross-user leaks live. The 2-second test I put a minimal, synthetic reproduction on GitHub: supabase-rls-leak-demo . Same test suite on two branches, differing only by db/policies.sql : broken -> 4 failed, 1 passed (an authenticated user reads another user's row) fixed -> 5 passed npm ci npm run test :ci No Docker, no Supabase project, no credentials. The tests run PostgreSQL in PGlite locally and exercise database-level row security. They do not model Supabase Auth, PostgREST, the Data API, or the network path - the result proves only the row-level gate in the fixture, which is exactly the gate people get wrong. On broken , the failing assertion is readable on purpose: x does not let user B read any row owned by user A -> user B received 1 row(s) belonging to another user: ["A: card ending 4471, expiry 09/29"] (That is synthetic seed data, not a real card.) Run the free checks against your own database The repo also ships audit/rls-audit.sql - nine read-only queries against the system catalogs, MIT-licensed, nothing to install and nothing to send anywhere. Every one is SELECT -only, so it is safe to paste into the Supabase SQL editor. They tell you: RLS coverage per table Every policy and the roles it actually applies to (an empty roles array means no TO clause, so the policy is evaluated for anon too) The effective write check, and which columns its predicate never mentions What

You can load-test search latency with a script and a graph. Relevance has no such gauge by default, so most teams ship a new ranking rule, eyeball a handful of queries, and hope nothing important regressed. The fix is a small, boring relevance test suite: a fixed set of queries, human-judged expected results, and a metric you compute the same way every time — so "did this ranking change help?" becomes a number you can diff, not an argument you have in Slack. This post is a build guide. By the end you'll have a judgments file, a scorer that outputs precision@k, MRR, and nDCG, and a before/after comparison you can wire into CI. The examples use Postgres full-text search, but the harness is engine-agnostic — Elasticsearch, Meilisearch, or a vector store all slot into the same shape. Why can't I just load-test relevance the way I load-test latency? Latency is a property of the system. Relevance is a property of the match between a query and what a human expected to see — and that judgment lives outside the database. A commenter on an earlier post about running Postgres search in production put it well: latency can be load-tested, but quality needs query sets, expected result buckets, bad-query examples, and a way to compare changes before shipping a new ranking rule. That's the whole job, and none of it comes for free with your index. The trap is thinking a passing query proves relevance. SELECT ... WHERE tsv @@ query returning rows tells you the index matched. It says nothing about whether the right rows landed in the top 5, which is all a user ever sees. The takeaway: relevance is measured against human judgments, not row counts — so the first artifact you build is the judgments, not the query. Building the golden query set Start with 20–50 real queries. Pull them from your search logs if you have them (the head terms plus a long tail of specific ones), or write them from real user intents if you don't. For each query, mark which documents should come back and how rel

I was building a runtime N+1 query detector for Node. The detection part worked on the first afternoon. Getting it to tell you which line of your code caused the problem took considerably longer, and taught me something about how ORMs execute queries that I had not thought about before. This is that story, and the fix. The symptom The detector instruments your database driver. When the same query shape runs many times inside one request, it reports it — along with the file and line that issued it, which is the part that actually saves you time: nplusone 1 finding in GET /orders — 51 queries, 840ms N+1 query 50× SELECT * FROM items WHERE order_id = ? at src/routes/orders.ts:47:38 (loadOrdersPage) 612ms spent here That worked. Then I pointed it at an app using Drizzle and got this instead: N + 1 query 12 × select "id" , "order_id" from "items" where "items" . "order_id" = $ 1 < unknown call site > Detected, counted, and attributed to nothing. Do not theorise. Dump the stack My first instinct was that my frame filter was too aggressive — it skips node_modules , node:internal , and the library's own frames, so maybe it was eating something it should not have. Rather than guess, I printed the whole stack at the exact moment the driver was called: const originalQuery = pg . Client . prototype . query ; pg . Client . prototype . query = function (... args ) { const previous = Error . stackTraceLimit ; Error . stackTraceLimit = 100 ; const stack = new Error (). stack . split ( " \n " ). slice ( 1 ); Error . stackTraceLimit = previous ; console . log ( " FRAMES: " , stack . length ); stack . forEach (( line , i ) => { const mine = ! /node_modules|node:internal/ . test ( line ); console . log ( ` ${ String ( i ). padStart ( 3 )} ${ mine ? " >>> " : " " } ${ line . trim ()} ` ); }); return originalQuery . apply ( this , args ); }; Here is what came back for a single await db.select().from(items).where(...) : FRAMES: 12 0 at Proxy.<anonymous> (.../nplusone/dist/adapters/postgre

I remember the old days of manual scaling. You'd jump into a CLI, check your metrics, realize you needed another node or a capacity adjustment, log into a web console, navigate three layers deep into some proprietary dashboard, and hope you didn't click the wrong thing while trying to find a specific cluster ID. Now we have AI agents. But most people are using them wrong. They treat Claude or Cursor as just better search engines for code, rather than giving them hands. If you're running high-availability workloads on something like TiDB Cloud, the friction isn't in writing the SQL—you already know how to do that. The friction is in the operational visibility: knowing exactly what’s happening across your serverless instances versus your dedicated clusters without leaving your IDE. The Gap Between Code and Infrastructure The reason I spend so much time building things like MCPFusion is precisely because of this disconnect. An LLM might help you write a complex join perfectly, but if it doesn't know whether the target TiDB X instance is actually healthy or which project ID handles your staging environment, it's basically flying blind. You end up copy-pasting JSON blobs from your terminal into the chat window just to give the model context. That's slow, prone to error, and frankly, beneath what modern tooling should look like. This is why we released the TiDB Cloud (Serverless Distributed SQL) MCP server on Vinkius. It closes that loop. What This Actually Does (And Doesn't) Let's be very clear about what this tool allows you to do through an agent like Claude or Cursor. We aren't looking for "magic" here; we want predictable utility. The current implementation focuses on discovery and inspection. In DevOps terms, it provides a controlled read-only view of your topology. Here is what's available: Organization Discovery: You can call list_projects to see everything sitting under your umbrella and pull metadata via get_project . This solves the "what was that project ID ag

Three semantic layers, three architectures, three very different bills. All three will define what a metric means. None of them proves an AI agent is allowed to run it. Quick orientation dbt Semantic Layer Cube AtScale Core idea Metrics as version-controlled code Headless API in front of metrics OLAP-style aggregate acceleration Strongest when You want engineering discipline Many apps consume the same numbers Heavy, stable aggregate workloads Modelling Hand-authored YAML Hand-authored data model Hand-authored cubes Cost driver Plan tier + query volume Pre-aggregation builds + compute Quote-based licence + compute Governance Upstream, in the warehouse In front of the API On the cube Each is competent at what it was built for. If your consumers are dashboards and analysts, any of the three will serve you. The question none of them answers An agent doesn't arrive with a metric name. It arrives with an intent in English and has to work out which entities, which grain, which joins, and whether it's entitled to any of it. That exposes two gaps every one of these shares: Undefined intent has no answer. Coverage is whatever someone remembered to model. Business questions don't respect that boundary. Authorisation is checked around the query, not inside it. A filter applied after execution means the data already moved. What to actually evaluate on Ignore feature matrices and score these five: Answer a question nobody modelled, on your schema Show why one join path was chosen over two others Same question, two users with different entitlements — show both SQL statements Ask something ambiguous. Refusal or guess? Reproduce a number from six months ago with the definitions then in force Most evaluations stop at 1. Numbers 3 and 5 are the ones that decide whether the thing ships in a regulated business. The full breakdown — architecture-by-architecture comparison, cost profiles, and the migration implications of each — is here: 👉 dbt Semantic Layer vs Cube vs AtScale: Choosing a

COUNT looks like the simplest function in SQL, and it is the one that quietly trips up the most people in interviews and on the job. The confusion is almost always the same: COUNT(*) , COUNT(column) , and COUNT(DISTINCT column) look nearly identical but count three different things. Once you can say out loud what each one counts, a lot opens up. You can verify a data migration, find duplicates, and measure how complete a column is, all with the same little function. This guide is that explanation, with lots of small examples you can copy. The one-sentence version. COUNT(*) counts rows . COUNT(column) counts rows where that column is not NULL . COUNT(DISTINCT column) counts how many different non-NULL values that column has. Everything below is just that sentence, slowed down. The three forms of COUNT and what each one counts Picture one small table, customers , with a region column where two rows were never filled in: id name region 1 Maya North 2 Jordan South 3 Alex North 4 Sam NULL 5 Taylor NULL Now run the three forms on it: SELECT COUNT(*) AS all_rows, COUNT(region) AS rows_with_region, COUNT(DISTINCT region) AS different_regions FROM customers; all_rows rows_with_region different_regions 5 3 2 COUNT(*) = 5. Every row, no exceptions. The * means "the row itself," so NULLs never matter. COUNT(region) = 3. Only the rows where region has a value. Sam and Taylor are skipped because their region is NULL. COUNT(DISTINCT region) = 2. The different values are just North and South . The two Norths collapse to one, and NULL is not counted. The NULL rule that makes them disagree Predict it first. A table has 100 rows. Twenty of them have no email address. What does counting the email column give you? Say the number before you read on. Here is the whole trick in one line: COUNT(*) counts rows. COUNT(something) counts non-NULL values of that something. So the moment a column has any NULLs, COUNT(column) comes back smaller than COUNT(*) . That gap is not a bug, it is informat

I have been building a thing that lets a language model propose an UPDATE , then executes it for real inside a transaction, measures the actual before and after values, and always rolls back. A human reads the measurement and decides. Only then does anything commit. The pitch is one sentence: what you approve is not the model's description of its SQL, it is what the database did when the SQL ran. Last week I found that the thing showing you that measurement was showing you a subset of it, and had been since the first release. The failure Real output, from @hyuga/llm-safe-sql@0.4.0 installed from npm. One row: name = 'Tanaka' , postcode = '00100' . UPDATE customers SET name='Sato', postcode='00100' WHERE id=1 What this touches customers — Customer records. The postcode is used for billing address and delivery. 1 row would change, across 1 column: name Measured by running the statement and rolling it back id = 1 name: 'Tanaka' -> 'Sato' One row, one column. postcode is not mentioned, and that is correct — it is being assigned the value it already holds, so nothing about it changes. The card is describing the diff accurately. Approve it. Then, before it is applied, somebody else notices the postcode is wrong and fixes it: UPDATE customers SET postcode = '90210' WHERE id = 1 ; Now apply the approved plan: Applied: UPDATE on customers, 1 row(s), at 2026-08-10T09:49:12.049Z. DB now: [{"name":"Sato","postcode":"00100"}] The fix is gone. Zero warnings. The word postcode never appeared on the approval card, never appeared in the audit record, and never appeared in the comparison the tool makes before it commits. One variable doing two jobs The diff was built like this: const changed : string [] = []; for ( const c of Object . keys ( before )) { if ( same ( before [ c ], after [ c ])) continue ; // drop what did not move if ( auto . has ( lower ( c ))) continue ; // drop what the DB maintains itself changed . push ( c ); } That is a correct answer to "what should the card sho

In Blog 5, we built a SQL parser. It can take this: INSERT INTO payments VALUES ( 500 , payment_1 , pending , 1 ) and turn it into a struct: InsertIntoTable { TableName : "payments" , ColumnValues : [] string { "500" , "payment_1" , "pending" , "1" }, } But this is still not enough for the storage engine. Our storage engine only knows how to store key-value pairs. It does not know what a table is. It does not know what a column is. It does not know that 500 is an integer, pending is a string, and 1 is a boolean. So, in this post we solve the missing bridge of persisting these in our key-value store. CREATE and INSERT are PUT operations This is the first major realisation. A key-value store is extensible to store literally anything. This is what we have been saying from the first post itself. But now we will be taking actual examples to prove that. CREATE TABLE Example Let's start with the create table example and see what should be the key and the value. Serialisation The key should be something that uniquely identifies the table, which is straightforward enough in this case as the table name . The value becomes everything else except the key, which is the schema of the table. So, in order to store the table name, we can append a reserved keyword as prefix like schema as a unique identifier. The structure of the key becomes _schema:<table_name> . The next question to answer is: How do we store a struct like below into our key value store where the value is always string? CreateTable { TableName : "payments" , ColumnDetails : [] Column { { ColumnName : "amount" , DataType : Int }, { ColumnName : "id" , DataType : String }, { ColumnName : "status" , DataType : String }, { ColumnName : "captured" , DataType : Bool }, }, PrimaryKeyColumnPosition : 1 , } One way is to serialise the entire struct into a string and store that directly. But in that case, deserialisation is a complex logic. JSON or struct serialisation and deserialisation is both space-heavy and compute inte

Every API that sits behind an unreliable network eventually faces the same problem: a client sends a request, the connection drops before the response arrives, and the client has no idea whether the operation happened. Did the payment go through? Did the order get created twice? The client's only safe move is to retry — which means your server needs a story for what happens when the same "create this thing" request arrives more than once. That story is idempotency keys, and getting the details right is more subtle than it first looks. The core idea The client generates a unique token — typically a UUID — once per logical operation, and attaches it to every retry of that operation: POST /orders Idempotency-Key: 7c3fd9a2-df01-4b3e-9a55-1e5f9b6b6d55 {"sku": "WIDGET-1", "qty": 2} The server's job is to guarantee that no matter how many times a request with that key arrives, the side effect (charging a card, creating an order, sending an email) happens at most once, and every retry gets back the same response the original request would have produced. Note what this is not: it is not deduplicating by request body. Two requests with identical bodies but no key are legitimately two different orders for two widgets. The key is what marks them as "the same attempt," not the payload. The naive approach, and why it breaks A common first pass is a table like: CREATE TABLE idempotency_keys ( key TEXT PRIMARY KEY , response_body JSONB , status_code INT ); On each request: check if the key exists, and if so return the cached response; otherwise do the work and insert the result. This looks right and is wrong in a specific way: it has a race condition. Two retries can arrive concurrently (a client that timed out and fired a second attempt while the first was still in flight), both miss the cache check, and both execute the underlying operation. You've now charged the card twice. Making the check-and-do atomic The fix is to claim the key before doing the work, using the database's ow

Migrating from MongoDB Atlas to a self-hosted replica set bought us control and cut our bill. It also quietly removed something we had stopped thinking about: Atlas had been taking continuous backups for us the entire time. After the migration, production data for Prochesta lived in /var/db/mongo on a single VPS. No snapshots. No off-box copy. A rm -rf , a bad migration script, or a dead disk would have been the end of it. We had written "backups" as a follow-up task in the migration spec, which is the engineering equivalent of a sticky note on a bank vault. The requirement we actually cared about was narrower than "back up the database". Most real-world data loss at our scale isn't hardware failure — it's a deploy that writes garbage, or someone running an update without a filter. Recovering to last night doesn't help when the damage happened at 14:20 and you noticed at 14:50. We needed to recover to an arbitrary moment , not to a nightly snapshot. The constraint nobody mentions: Community has no $backupCursor We chose Percona Backup for MongoDB (PBM), and immediately hit the limitation that shapes every decision downstream. PBM offers physical backups — fast file-level copies that restore in minutes and barely touch the running server. They work by opening a backup cursor via the $backupCursor aggregation stage. That stage exists in Percona Server for MongoDB and in MongoDB Enterprise. It does not exist in MongoDB Community, which is what the official mongo:8.0 image ships. So on Community, PBM gives you logical backups only: every document read out through mongod , compressed, and shipped off-box. Two consequences, both accepted deliberately rather than discovered later: Backups cost CPU on the primary — and with a single-member replica set there's no secondary to offload the read to. Restores insert documents and rebuild indexes, so restore time grows with data size much faster than backup time does. At our current size that's minutes, not hours. It's also the t

The engineering team at Stripe recently described how they automated database incident recovery by modeling their global infrastructure as a graph. Using graph search algorithms together with state machines, the team computes and executes remediation plans automatically. By Renato Losio

When building a nested reply system, most developers worry about deep tree complexity or messy data structures. For Vlox , I took a different approach: keeping things flat, fast, and secure by reusing a single Mongoose schema with smart atomic limits. Here is a deep dive into how I engineered a production-ready, race-condition-safe reply mechanism using MongoDB transactions, strict type sanitization, and automated limits. How It Works 🛠️ User Action: A user clicks the reply icon and submits their reply. The Payload: Vlox's system sends 3 fields via the endpoint /api/v1/reply/comment/post/:id : id : The post ID (passed as a URL parameter). rootCommentId : The ID of the root comment being replied to. reply : The raw text entered by the user. Sanitization & Validation: The incoming reply is instantly converted to a trimmed string. It then passes through two critical validation checks: Existence Check: The reply must exist. (If a malicious actor sends a payload without a body, the string literally evaluates to "undefined" and gets blocked). Length Limit: The reply must be under 201 characters, enforcing the standard comment limit. Atomic Transactions: If the validation checks pass, the system initiates a Mongoose transaction to execute the following steps safely: Permission Check: It verifies if the user has permission to reply by checking the post's status via await schemas.Posts.findOne(hotQueries.find_user_post(id, req.session.userId)); . Creation: If permissions are valid, it creates a new reply. (Fun fact: It reuses the exact same schema as standard comments!) The Reply Schema Structure: The reply object functions just like a normal comment, with two distinct exceptions: It does not contain a repliesCount field. It includes an extra rootId field, which explicitly points to the ID of the root comment being replied to. Concurrency & Caps: To guarantee that a single comment never receives more than 10 replies while simultaneously incrementing the counter, the system r

After months of development, I'm excited to share DBNavigator, a cross-platform database IDE that I've been building from scratch using Java and JavaFX. ✨ Current Features ✅ PostgreSQL support ✅ MySQL support ✅ Modern Datagrip-inspired UI ✅ Multi-tab SQL editor ✅ Syntax highlighting ✅ Schema explorer ✅ Query execution ✅ Professional dark theme ✅ Cross-platform (Windows, Linux & macOS) This project has been an incredible learning journey in desktop application development, JavaFX UI design, database connectivity, and IDE architecture. I'm sharing it with the developer community because I'd genuinely appreciate your honest feedback. I'd love to hear your thoughts on: UI/UX design Performance Missing features Overall developer experience Architecture and code quality Any bugs or improvements you notice Whether you're a Java developer, DBA, or someone who works with databases every day, your feedback would mean a lot and help shape the next version of the project. ⭐ If you find the project interesting, please consider giving it a star on GitHub. GitHub: DBNavigator Thank you for taking the time to review it. Every suggestion, issue report, and critique is greatly appreciated! 🙌

I have been building machin for a while — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot recently, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything? It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had. The benchmark was measuring the order I ran things in machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20-25% . That claim also shipped inside machin guide , which is what every coding agent reads to learn the language. When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output: for kernel in kernels : for lang in [ machin , rust , zig ]: for _ in range ( 5 ): # all 5 machin, THEN all 5 rust, THEN all 5 zig time ( binary ) It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last . Zig always went last. Zig always looked slowest. The fix is four lines — interleave the rounds, rotate who starts each one. Here is what my headline number did: intsum 10^9 before (blocked) after (interleaved) machin 2832 ms 3079.7 ms rust 3764 ms 3223.8 ms zig 3556 ms 3189.7 ms "machin +20-25%" machin +3% = a TIE A 20-25% win became a tie. I deleted the claim from the README and from machin guide . The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured was 41% of the min sample. Calling winners inside that is how benchmarks start

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible." This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic. 1. The Actionability Gap (The Golden Question) The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is: "Does the company have the operational capacity to make a decision in milliseconds?" If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch. 2. The Hidden Complexity and the Cloud Bill Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug. Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule. 3. "Micro-Batch" Solves 99% of Your Problems There is a perfect middle ground that the hype industry tries to ignore: the micro-ba

GitHub malware advisories no longer stop at npm. Here's how we wired OpenSSF's malicious-packages data into the Advisory Database, and why we built the pipeline paranoid. The post How we took malware advisories beyond npm appeared first on The GitHub Blog .

If your files already live in Amazon S3, the pitch for storage that branches with your database is appealing but the word "migration" makes it sound like a project. It mostly is not. Neon's object storage speaks the S3 API, so the code you already wrote, the AWS SDK calls and presigned URLs, keeps working. What changes is how you point the client and where the bucket comes from, and that is a small, mechanical diff. The actual data move is a copy loop you can run once. The one thing to do up front is confirm the object operations your app actually relies on: the demo here exercises PutObject , GetObject , listing, and presigned URLs, and I flag the S3 features you should check for yourself further down. This post is the practical version: what stays identical, the exact config that changes, a script to copy the objects across, and an honest list of the S3 features that do not have an equivalent so you know what to check before you commit. The repo with the working client is at the end. TL;DR Neon object storage is S3-compatible. Your @aws-sdk/client-s3 code for the common operations, PutObject , GetObject , getSignedUrl , listing, works unchanged (these are what the demo verifies). Confirm anything beyond that, like multipart for large objects, against the current preview. The diff is the client config: point endpoint at the Neon storage endpoint, pin region: 'us-east-2' , set forcePathStyle: true . The bucket is declared in neon.ts instead of created in the console, and credentials are injected per branch. Move the data with a list-and-copy loop between two S3 clients (source AWS, destination Neon). What does not carry over: S3 bucket policies, event notifications and Lambda triggers, storage classes and Glacier transitions, and cross-region replication. Object CRUD and presigning do. The payoff is everything else in this series: once the files are on Neon, they branch with your database. Prerequisites An existing S3 bucket and credentials that can read it A Neon p

If your app stores files and you want real preview environments, you eventually hit the same wall: each preview needs its own storage, so you start provisioning a bucket per environment. That sounds cheap until you write it down. For every ephemeral environment you create a bucket, attach a policy, mint an IAM role or access keys, set CORS, add a lifecycle rule so it eventually cleans up, wire the credentials into the preview's config, and register a teardown step for when the PR closes. Then you find the orphaned buckets the teardown missed, months later, still billing. The reason this is painful is that the bucket is a separate resource from the database, so it needs its own lifecycle. Neon collapses that: the bucket is declared as part of the branch, so it is created and destroyed with the branch and needs no per-environment provisioning at all. This post compares the two approaches and shows the branch version working with no bucket-management code in sight. The repo is at the end. TL;DR Isolated storage per preview usually means provisioning a bucket per environment: policy, IAM, CORS, lifecycle, credential wiring, teardown. It is slow, it drifts, and it leaves orphaned buckets that keep costing money. On Neon the bucket is declared once in neon.ts . Creating a branch brings the bucket (with a copy-on-write copy of the files) and injects scoped credentials; deleting the branch removes it. There is no per-environment bucket to create, no IAM role to mint, and nothing to orphan. Copy-on-write means fifty preview buckets do not cost fifty times the storage, only what each one changes. Prerequisites A Neon project on the platform preview (object storage, us-east-2 ) The Neon CLI, and a CI system that opens/closes preview environments Familiarity with S3 buckets and IAM if you have done the manual version The per-environment bucket, written out Here is what "just give the preview its own bucket" actually expands to, per environment: Create a bucket with a unique nam

The naive way to accept file uploads is to POST them to your API, let the server read the bytes, and write them to object storage. It works until the files get large or the traffic gets real. Now every upload crosses your infrastructure twice, once from the client to your server and once from your server to storage, and your server holds the whole file in memory or on disk while it does. On a serverless function it is worse, because functions have request-size and duration limits that a big upload runs straight into. Presigned URLs are the standard fix, and they predate serverless by a decade. Your server does not move the bytes; it hands the client a short-lived, pre-authorized URL and the client uploads directly to object storage. The server only issues permission and records metadata. On a Neon Function this is the same AWS S3 SDK you already use, pointed at the branch's storage endpoint. This post builds it and tests the whole round trip. The repo is at the end. TL;DR Proxying uploads through a function sends the bytes across it, burning bandwidth and memory and hitting request-size limits. A presigned URL is a time-limited, pre-authorized link to one object key. The client PUTs the bytes straight to storage; the function never touches them. On Neon Functions you generate it with getSignedUrl from @aws-sdk/s3-request-presigner , the same code as any S3-compatible store. I tested the full flow: presign, the client PUT straight to storage returned 200 , a metadata record was saved, and downloading the object returned the exact bytes. One gotcha to pin: the injected AWS_REGION is the storage-cell host, not a region, so set region: 'us-east-2' on the client. Prerequisites A Neon project on the platform preview with a declared bucket (object storage, us-east-2 ) The AWS SDK: @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner Familiarity with S3-style object storage and HTTP PUT Why not just proxy the upload Sending the file through the function has three costs that