标签：#production

I watched an AI agent send 200 emails to the wrong recipients because I forgot one validation check. The emails were well written. The offers were real. The recipients were just... not our leads. That was early. I learned fast. Every agent I build now has three layers of guardrails before it touches a database or an API. Here's exactly what those layers look like and why they're non-negotiable for production. Input Validation: Your Prompt Is Not a Schema The first mistake people make is trusting the LLM to produce valid output. It won't. Not reliably. I've seen GPT-4 return a JSON key called "emial" instead of "email" in a critical pipeline. One typo, and the whole record is garbage. The fix is a strict validation layer that runs before any data reaches your system. In my AI resume tailor, I use a JSON schema with conditional presence flags. Every field that must be real has a has_* boolean guard. If the LLM tries to fabricate a phone number, the schema rejects it. const resumeSchema = z . object ({ contact : z . object ({ email : z . string (). email (), phone : z . string (). optional (), has_phone : z . boolean () }). refine ( data => { // If phone is present, the guard must be true return data . phone ? data . has_phone : ! data . has_phone }, " Phone number present but has_phone flag is false " ) }) This pattern catches hallucinations before they corrupt your database. The schema is the contract. The LLM is just a suggestion engine. Permission Scoping: Give Agents the Minimum They Need An agent should never have write access to tables it doesn't need. That sounds obvious, but I've seen production systems where a job description rewriting agent had full CRUD access to the user table. When I built the LLM scoring pipeline for a job board platform, I created separate database roles. The scoring agent only had SELECT on the job listings table and INSERT on a scoring results table. It never touched users, applications, or configuration. Even if the prompt was hijack

Every AI-generated site we have inherited was missing the same eighteen things. None of them are visible in a screenshot. All of them are visible to Google. July 10, 2026 An AI-generated site looks done. It has a hero, sections, a color palette, and copy that reads well in a screenshot. Then we open the page source, and the production work is missing. Not some of it. The same eighteen things, every time. None of them change what a human sees in a browser. All of them change what a crawler, a link preview, or a cache does with the page. Here is the list we run before we call anything live. Crawlability and indexing This is where the gap is widest, because a client-rendered single-page app hands crawlers an empty div and expects them to run JavaScript to fill it. Many will not. We fix that with static work. Prerendered static HTML per route , so the first paint is real content and not a loading spinner. A sitemap.xml generated from a single route manifest , so it lists every page and no page twice. A robots.txt that points at that sitemap and does not accidentally disallow the whole site. A canonical URL on every page , because a screenshot cannot show you a missing canonical tag. A meta title and description written per page , not one template repeated across the whole site. Structured data as JSON-LD for the page types that support it. IndexNow submission on deploy , so search engines learn about changes without waiting for a crawl. An llms.txt file describing the site for the AI crawlers that now read it. Sharing and presentation A link is content too. When someone pastes the URL into Slack or iMessage, the site is representing itself, and the defaults are usually blank. Open Graph tags for the title, description, and image. Twitter card tags , which are close to Open Graph but not identical. A per-page share image at 1200x630 in PNG. WebP renders unreliably in LinkedIn and iMessage previews, so we ship PNG here even though we prefer WebP elsewhere. Descriptive alt

Request logging: vì sao mỗi log line phải có request id, và structured log cứu debug thế nào khi có sự cố production Request logging không phải là in console.log('got a request') cho vui. Nó là dấu vết duy nhất còn lại khi một request đã đi qua service, đi qua vài downstream, gặp lỗi, và bị đóng socket. Nếu log không có gì để nối lại các dòng thuộc cùng một request — không có requestId , không có traceId — thì log của một service RPS trung bình biến thành một mớ text xen kẽ giữa hàng chục request đồng thời, và câu hỏi "request X đã đi tới đâu, fail ở service nào" trở thành không trả lời được. Structured log (mỗi dòng là một JSON object với field cố định) + một correlation id truyền qua toàn bộ pipeline chính là cái làm log có thể query được, thay vì grep mù. Hai framework Express và Fastify tiếp cận khác nhau: Fastify tích hợp pino sẵn và tự sinh req.id , Express phải tự dán qua pino-http hoặc middleware tay. Nhưng cả hai đều vỡ theo cùng một kiểu khi correlation id bị đứt giữa các service. Cơ chế hoạt động Ba miếng ghép: (1) một structured logger — thường là pino trong ecosystem Node vì nó ghi NDJSON và có async destination, (2) một cơ chế sinh/nhận requestId , (3) một cách propagate id đó qua async work và qua HTTP call sang service khác. pino là JSON logger tối giản: mỗi lời gọi logger.info({...}, 'message') xuất một dòng NDJSON ra một WritableStream (mặc định là stdout). Log level là số ( trace=10, debug=20, info=30, warn=40, error=50, fatal=60 , theo pino docs), và có child logger — logger.child({ reqId }) tạo một logger mới bind sẵn các field, mọi log line từ child đều có reqId mà không phải truyền tay. Với Fastify , chỉ cần logger: true là có pino và request logging tự động — Fastify sinh request.id (mặc định là monotonic counter, đổi được qua option genReqId ) và log một cặp dòng "incoming request" / "request completed" cho mỗi request. Trong handler, request.log là child logger đã bind reqId : import Fastify from ' fastify ' import crypto from ' node:crypto

Production lock analysis: vì sao pg_stat_activity một mình không đủ, và join với pg_locks mới ra root cause Lock contention trong Postgres hiếm khi báo bằng error — nó báo bằng wait_event_type = 'Lock' ở pg_stat_activity và bằng latency tăng từ phía application. Khi một incident xảy ra ("API treo, không ai biết tại sao"), thứ team cần trong 60 giây đầu là một bức tranh: PID nào đang đợi, đợi lock loại gì trên object nào, bị block bởi PID nào, PID block đó đang chạy query gì và đã giữ transaction bao lâu . pg_stat_activity một mình chỉ trả lời được nửa câu hỏi ("ai đang đợi"); pg_locks một mình chỉ trả lời nửa còn lại ("ai giữ gì"). Phải join hai view này — và bám theo pg_blocking_pids() — để dựng được blocking tree. Không có dashboard cho luồng dữ liệu này là lý do điển hình một production freeze kéo dài 30 phút thay vì 3 phút: incident commander phải mò ad-hoc bằng psql , gõ sai query, miss idle in transaction đang giữ AccessExclusiveLock của một migration nửa đời trước. Cơ chế hoạt động pg_locks là một view phơi nội dung trực tiếp của shared lock manager trong shared memory. Mỗi dòng là một lock request (đã granted hoặc đang chờ) thuộc một backend. Theo Postgres docs phần "System Views → pg_locks", các column then chốt: locktype ( relation , transactionid , tuple , virtualxid , advisory ...), relation (OID — join pg_class ), transactionid , virtualtransaction , pid (backend PID), mode ( AccessShareLock , RowExclusiveLock , ShareUpdateExclusiveLock , AccessExclusiveLock ...), granted (bool), fastpath (lock đi qua fast-path tránh shared lock manager), và waitstart (timestamp bắt đầu chờ — bổ sung sau v14, hữu ích để đo lock wait time mà không cần snapshot diff). pg_stat_activity là view phơi trạng thái runtime của mỗi backend: pid , usename , datname , application_name , client_addr , backend_start , xact_start , query_start , state ( active , idle , idle in transaction , idle in transaction (aborted) ), wait_event_type , wait_event , backend_xid , backend_xmin , qu

Structured output broke on us three times. The third time taught us what "operator-ready" means. Last quarter we shipped a contract-extraction agent to an enterprise legal team. Schema validation passing at 97%. Human reviewers satisfied with the output quality in testing. Rollout went smoothly. Then it broke. Three times. In three completely different ways. The first two failures we fixed with better prompts and stricter schemas. The third one taught us something the first two hadn't: that "operator-ready" is not a technical checklist. It's a claim about your agent's behavior under conditions you didn't design it for. Failure one: the validation paradox Week two. A lease agreement came through with a renewal clause formatted as a table instead of prose. Our extractor looked for renewal terms in a specific JSON path. The table format populated the schema differently. Validation passed. The extracted renewal date was off by two years. The fix was obvious in retrospect: add a canonical-format normalization step before extraction. But the lesson was sharper than that. Schema validation tells you the shape of the output, not whether the content is correct. A JSON object with the right keys and the right types can still contain wrong values. Our 97% validation success rate was measuring the wrong thing. It was measuring structure conformance, not content accuracy. After this failure, we separated validation into two signals: schema validity (does the object have the required fields) and field confidence (do we have evidence the content is correct). We started logging both. An output is trusted only when both signals are above threshold. Failure two: the retry loop that lies Month one. A particular clause type appeared in a contract format we hadn't trained our test set on. The extractor failed schema validation on the first attempt. Our retry logic kicked in, filled missing fields with model-inferred defaults, and passed validation on the third try. The output looked rig

How to Create an AI Agent: A Production Walkthrough The first agent I shipped to production failed at 3am on a Sunday. It looped on a tool call, burned through $40 in tokens before my budget alarm fired, and left a half-written draft in the database with no way to resume. That night taught me more about agent design than any framework tutorial. Since then I have built a pattern I trust enough to leave running unattended for weeks at BizFlowAI, where agents research, write, optimize and publish content without me touching them. This is that pattern, stripped down to what actually matters. Start with the job spec, not the framework Before you pick LangGraph, CrewAI, or roll your own, write the agent's job spec like you would for a junior engineer. One paragraph. What it owns, what it must never do, what "done" looks like, and which signals tell you it failed. Here is the spec for one of my production agents: The Topic Researcher owns generating a ranked list of 20 content topics per site per week. It reads from keyword_pool and search_console_perf , writes to topic_queue . It must never publish, never call paid APIs more than 8 times per run, and must finish in under 6 minutes. Done = 20 topics with score >= 0.6 and zero duplicates against the last 90 days. Failure signal = empty queue after a run, or any topic flagged by the dedupe check. If you cannot write this paragraph, do not build the agent. You will end up with a "do everything" prompt that hallucinates its way through ambiguous tasks. The job spec becomes your evaluation rubric later, so write it carefully. Rule of thumb I use : if the spec needs more than 5 tools or more than 3 decision branches, it is two agents, not one. Design the tools before you write the prompt Most agent failures I have debugged were not prompt failures. They were tool failures. The model called a tool with wrong arguments, the tool returned a 4MB JSON blob, or two tools had overlapping responsibilities and the model picked the wrong

The AI Implementation Process I Use With Every Client Most AI projects do not fail at the model. They fail in the six weeks before anyone writes a prompt, and in the six weeks after the demo lands in a Slack channel and nobody knows who owns it. I have run enough of these now (from one-off automations to multi-agent content systems running unattended) that the process has converged into something stable. This is the version I actually use. It has five phases: scoping, POC, integration, evaluation, operations. Each phase has an exit criterion. If we cannot meet the exit criterion, we do not move forward. That single rule has saved more projects than any clever architecture choice. Phase 1: Scoping (1 to 2 weeks, fixed price) Scoping ends with a written document that names the workflow being automated, the system of record it touches, the success metric in hours or dollars, the data we have access to, and the smallest possible first slice. No model is chosen yet. No code is written. If we cannot produce that document, the engagement stops here and the client keeps the document. The hardest part of scoping is resisting the urge to solve the interesting problem. Clients almost always describe the AI-shaped fantasy ("an agent that handles all support tickets") when the real opportunity is narrower and uglier ("triage tier-1 tickets that mention billing, route to the right queue, draft a reply for human approval"). The narrower version ships. The fantasy does not. I run scoping as three sessions: Workflow walkthrough. Someone who actually does the work shows me their screen for an hour. I record it. I take timestamps. The point is to find the moments where a human is doing pattern matching that an LLM can do, and to find the moments where they are doing judgment that an LLM should not do. Data audit. Where does the input live? Where does the output need to go? What is the auth story? If the data is locked inside a SaaS product with no API and no export, that is the projec

7 Things I Wish I Knew Before Scaling Next.js + Supabase to 100K Users Six months ago, we launched our SaaS with Next.js and Supabase. The stack was perfect for our MVP: fast development, great DX, and it just worked. Then we hit 10K users. Then 50K. Then 100K. Everything that worked beautifully at small scale started breaking. Database queries that took 50ms now took 5 seconds. Our Supabase bill went from $25/month to $800/month. Users complained about slow page loads. Here's what I wish someone had told me before we started. 1. RLS Policies Are Not Optional (Even in Development) We skipped RLS in development. "We'll add it before launch," we said. Launch day came. We enabled RLS on all tables. The app broke in 47 different places. Queries that worked suddenly returned empty arrays. Inserts failed with permission errors. We spent 12 hours fixing RLS policies while users waited. What I'd do differently: Enable RLS from day one. Write policies as you create tables: CREATE TABLE posts ( id UUID PRIMARY KEY DEFAULT uuid_generate_v4 (), title TEXT NOT NULL , user_id UUID REFERENCES auth . users ( id ) ); -- Enable RLS immediately ALTER TABLE posts ENABLE ROW LEVEL SECURITY ; -- Write policies now, not later CREATE POLICY "Users can view own posts" ON posts FOR SELECT USING ( auth . uid () = user_id ); Test with RLS enabled. If it works in development, it'll work in production. 2. Database Indexes Are Not Premature Optimization "We'll add indexes when we need them." We needed them on day 3. Our posts feed query went from 50ms to 8 seconds as we hit 10K posts. Users complained. We scrambled to add indexes during peak traffic. The query: const { data } = await supabase . from ( ' posts ' ) . select ( ' *, profiles(*) ' ) . eq ( ' published ' , true ) . order ( ' created_at ' , { ascending : false }) . limit ( 20 ) The fix: CREATE INDEX posts_published_created_at_idx ON posts ( published , created_at DESC ) WHERE published = true ; Query time dropped to 12ms. What I'd do di