标签：#Engineering

TL;DR Traditional decoupled ETL pipelines (like the "Modern Data Stack") are too brittle and complex to handle the unpredictable, heavily nested data generated by AI and LLM features. Agentic data serving solves this by focusing on dynamic query routing and semantic discovery, letting AI agents discover and query data autonomously using schema-resilient tools and codified business logic. You can build an agentic data stack by pairing S3 storage with DuckDB's native JSON handling and schema-agnostic Parquet reading ( union_by_name=true ), eliminating failure-prone parsing steps. The open Model Context Protocol (MCP) replaces custom, hacky LangChain tools by providing a standard interface for agents to discover schemas and execute queries securely. The open Model Context Protocol (MCP) and DuckDB's embeddable architecture make it practical to connect agents directly to your data with minimal infrastructure overhead and elastic, consumption-based compute. For years, broken ETL jobs powered my pager and my morning coffee. I am a staff engineer, and like many of you, I have spent a ridiculous amount of my career babysitting data pipelines. It is a thankless job that often feels like patching holes in a sinking ship. You are not alone in this. A Forbes survey shows data teams notoriously spend up to 80% of their time just moving and cleaning data instead of doing the interesting work of analysis. And the financial magnitude of this bottleneck is staggering: the ETL market is projected to reach $20.1 billion by 2032 at a 13% CAGR. This proves that massive industry capital is flowing into solving these pipeline bottlenecks, but throwing more money at the same old architecture was not going to save my mornings. This constant firefighting was frustrating, but manageable. Then came the new mandate: build the data backbone for our next-gen AI and LLM-based product features. The unpredictability of the queries and the sheer complexity of the data, nested JSON everywhere, were th

Google recently announced that WebMCP is entering origin trials in Chrome 149. The new WebMCP standard proposal lets sites expose tools (e.g., JavaScript functions and HTML forms) to in-browser AI agents, which can thus reliably simulate user actions instead of resorting to possibly expensive (e.g., on-screen reading) and often unreliable guesswork (e.g., DOM scraping). By Bruno Couriol

AI Fluency for Software Engineers: A Practical Playbook Beyond Prompting A few years ago, being productive with AI mostly meant knowing which tool to open and what question to ask. Today, that is not enough. For software engineers, AI is no longer just a chatbot sitting outside the workflow. It is becoming a thinking partner for architecture decisions, code reviews, production incidents, documentation, test planning, onboarding, and product discovery. But there is a problem: many teams are using powerful AI tools with weak operating habits. They ask vague questions. They paste too much context. They trust the first answer. They forget privacy boundaries. They use AI for speed, but not always for better engineering judgment. That is where AI fluency matters. AI fluency is not just prompt engineering. It is the ability to work with AI clearly, safely, and practically while staying in control of quality, reasoning, and responsibility. Here is a practical playbook I would recommend for software engineers and engineering teams. 1. Start with clarity, not clever prompts A weak prompt sounds like this: “Review this design and tell me if it is good.” The AI can answer, but the answer will likely be generic. A stronger prompt gives the AI a clear role, context, constraints, and output format: You are a senior backend architect. Review this proposed API design for a high-traffic order processing system. Evaluate: - correctness - scalability - failure handling - observability - backward compatibility - operational complexity Do not rewrite the whole design unless required. Separate critical risks from optional improvements. Output format: - Executive summary - Key risks - Recommended changes - Open questions - Final decision recommendation The difference is not word count. The difference is control. A fluent AI user does not hope the AI understands the task. They make the task hard to misunderstand. 2. Give enough context, but not everything AI output quality depends heavily o

I inherited a SQL Server database with 2.3 million rows. Queries took 45 seconds. Users were frustrated. Dashboards timed out. Here is exactly what I did. Step 1: Find the slowest queries I used SQL Server's query store to identify the top 10 worst performing queries. Step 2: Check the execution plan Missing index warnings everywhere. Also saw table scans on a 2 million row table. Step 3: Add targeted indexes Created two non-clustered indexes on the most filtered columns. No over-indexing. Just what the queries actually needed. Step 4: Rewrite the worst join One query was joining 6 tables with a cross apply that made no sense. Restructured to inner joins with proper filter ordering. The result 45 seconds down to 8 seconds. An 82% improvement. Real-time dashboards started working again. Key lesson Check what is actually slow before changing anything. Most people skip this and waste time optimizing the wrong thing. What is your fastest query optimization win?

I inherited a SQL Server database with 2.3 million rows. Queries took 45 seconds. Users were frustrated. Dashboards timed out. Here is exactly what I did. Step 1: Find the slowest queries I used SQL Server's query store to identify the top 10 worst performing queries. Step 2: Check the execution plan Missing index warnings everywhere. Also saw table scans on a 2 million row table. Step 3: Add targeted indexes Created two non-clustered indexes on the most filtered columns. No over-indexing. Just what the queries actually needed. Step 4: Rewrite the worst join One query was joining 6 tables with a cross apply that made no sense. Restructured to inner joins with proper filter ordering. The result 45 seconds down to 8 seconds. An 82% improvement. Real-time dashboards started working again. Key lesson Check what is actually slow before changing anything. Most people skip this and waste time optimizing the wrong thing. What is your fastest query optimization win?

Google's Angular team has released a repository called angular/skills, focusing on Agent Skills that enhance AI coding agents' ability to write modern Angular code. The repository includes skills for generating code and scaffolding applications, reinforcing current Angular conventions. It serves as a snapshot, aiming to improve AI suggestions by providing updated context. By Daniel Curtis

Pinecone has announced a new integration between its Nexus knowledge engine and Microsoft OneLake, aiming to fundamentally change how enterprise AI agents access and reason over corporate data. By Craig Risi

When code becomes cheaper, what still makes an engineer valuable? Recently, while writing my cover letter for remote roles and Upwork projects, I asked myself a very direct question: Why should a remote team or client choose me, especially in the AI era? I do not think the answer should be: “Because I am the strongest engineer technically.” That is not how I want to position myself. What I want to become is this: A backend engineer who can turn unclear business problems into reliable, maintainable systems. AI is making implementation faster. It can generate code, explain technologies, and provide alternatives. At the same time, remote work and platforms like Upwork make competition more global. We are not only competing with engineers nearby, but also with engineers from everywhere. If the only question is “Who knows more frameworks, patterns, or tools?”, many ordinary engineers may feel hopeless. But I believe there is another path. In real systems, code is only part of the work. Someone still needs to understand the business workflow. Someone still needs to define what “correct” means. Someone still needs to identify risks, edge cases, performance concerns, and reliability boundaries. My usual way of working starts from these questions: What is the real requirement? What does correctness mean in this workflow? What data must stay consistent? What edge cases could break the process? What performance or reliability signals should be protected? Where should the module boundary be? Who should orchestrate the main flow, and who should act as collaborators? This “orchestrator + collaborators” thinking helps me keep the main business process clear. The orchestrator owns the workflow. The collaborators handle specific responsibilities such as validation, translation, persistence, messaging, or external integration. I also use AI in this process, but not only to generate code. I use it to challenge my assumptions, explore alternatives, find missing cases, improve naming, r

There's a quiet lie rotting at the heart of modern software development, and almost nobody wants to say it out loud: the industry is drowning in people who can ace a LeetCode interview, argue endlessly about architectural patterns, and hold strong opinions about tabs versus spaces — but who have never actually shipped a product that real people use. We've built an entire culture around the performance of engineering rather than the practice of it. Walk into any dev team today and you'll find brilliant people who can recite the SOLID principles from memory, who have strong feelings about hexagonal architecture, and who will spend three weeks bikeshedding a folder structure — but who have never felt the particular anxiety of watching your own code process a real user's real money. Never felt that accountability. Never shipped something and then lived with the consequences. That's a problem! The industry has confused credentials and vocabulary with competence. A computer science degree, a GitHub profile full of tutorial clones, and the ability to reverse a linked list in a whiteboard setting are now routinely mistaken for the ability to build software that matters. They're not the same thing. The people who actually make software work — who ship under pressure, who fix production bugs at 11pm, who make pragmatic calls with incomplete information — often look terrible on paper. They'll choose a boring, proven technology over the shiny new thing. They'll skip the elegant abstraction in favour of something they can debug in a hurry. They understand that a running system that's 80% right beats a perfect system that's still in design. AI has made this worse. Now you can generate plausible-looking code, confidently-written documentation, and technically-accurate architecture diagrams without ever having built a system that stayed up under real load, without ever having migrated a production database at 2am, without ever having owned something from idea to invoice. Knowing ho

The Budget You Approved Isn't the Budget You'll Pay You approved $180K for a senior AI engineer. Eighteen months later, you've spent $282K and you're still not sure the hire is working out. This isn't unusual. It's the rule. Companies hiring AI engineers for the first time routinely underestimate total cost by 40–60%. Here's a breakdown of where that gap comes from — and why most founders don't see it until it's too late. The 56% Gap: Where It Comes From 1. Recruiting Costs Are Higher Than You Think (~12–18% of first-year salary) AI engineer recruiting isn't like standard software recruiting. Specialized headhunters charge 20–25% of first-year salary. Even if you find someone through your network, you'll spend founder or VP time on 15–30 hours of interviewing, plus take-home evals that the best candidates increasingly decline. If you use a staffing firm, add the markup. If you DIY it, add the opportunity cost. Typical recruiting overhead: $22,000–$40,000 per hire 2. Onboarding Takes Longer for AI Roles (~2–3 months of ramp) An AI engineer hired to build production agent systems isn't productive on day 1. They need to understand your domain, your data, your existing architecture, and your risk tolerance for AI-generated outputs. The ramp is real — most teams see 60–90 days before meaningful output. At $180K salary, two months of ramp is $30,000 in salary with limited ROI. Add engineering time for mentoring (typically 20% of a senior engineer's time during ramp), and you're adding another $15,000–$20,000. Ramp cost: $30,000–$50,000 3. Infrastructure Spend Scales With Experiments AI engineers experiment. That's the job. Every experiment has a GPU bill, an API bill, and a storage bill. Early-stage teams routinely see $3,000–$8,000/month in AI infrastructure spend once they've hired their first AI engineer — much of it from exploratory work that doesn't ship. Over a year: $36,000–$96,000 in infra costs that weren't in the original headcount budget 4. Tooling and Data Cos

A complete solution: expressive range types in your domain layer, full PostgreSQL translation in your data layer - no compromises at either end The Two-Column Trap Almost every developer has written it at least once. An object with two date properties: public class MemberSubscription { public int Id { get ; set ; } public int MemberId { get ; set ; } public DateTime StartDate { get ; set ; } public DateTime EndDate { get ; set ; } } Imagine you need to answer a seemingly simple question in a booking system: "Is this subscription still active, and does it conflict with the proposed new one?" With two bare fields, that code ends up looking something like this: // With two bare DateTime fields — the check you always end up writing public static bool IsActive ( MemberSubscription sub , DateTime at ) => sub . StartDate <= at && ( sub . EndDate == default || sub . EndDate > at ); public static bool ConflictsWith ( MemberSubscription a , MemberSubscription b ) { // Partial overlap: a starts inside b if ( a . StartDate >= b . StartDate && a . StartDate < b . EndDate ) return true ; // Partial overlap: b starts inside a if ( b . StartDate >= a . StartDate && b . StartDate < a . StartDate ) return true ; // b is fully contained by a if ( a . StartDate <= b . StartDate && a . EndDate >= b . EndDate ) return true ; // What about open-ended subscriptions? What about same-day boundaries? // What about inclusive vs exclusive end dates? ... return false ; } It looks perfectly reasonable. But start asking questions — as Steve Smith (Ardalis) does in his essay on making the implicit explicit — and you notice how much invisible knowledge this design requires. Should EndDate ever precede StartDate ? The type system doesn't say. Can a subscription have a null end date meaning it never expires? Nothing in the model communicates that. Is a subscription that ends today still active at 11:59 PM? Ask three developers and get three answers. The EndDate == default sentinel for open-ended subsc

One of the reasons I often find myself disagreeing with modern software trends is that many conversations revolve around features. How many features does it have? How quickly can we add more? What can we put on the marketing page? What can we announce next? Features matter. But I care far more about systems. Because at the end of the day, people don't buy features. They buy outcomes. And outcomes come from systems. The Car Analogy One of the easiest ways to explain my thinking is with cars. A car is made up of thousands of individual components. An engine. A transmission. Suspension. Brakes. Fuel systems. Electrical systems. Cooling systems. Sensors. Wiring. Each component is important. But nobody walks into a dealership and says: "I'd like to purchase six pistons, a transmission housing, and a fuel injector." They buy a car. They buy transportation. They buy a complete system. The individual parts only matter because they contribute to the overall experience. The customer doesn't want to think about every moving piece. They want to get in, turn the key, and drive. Drivers and Mechanics This is where I think technology often loses its way. Users are drivers. Engineers are mechanics. A driver should be able to: Start the vehicle Fill it with fuel Check the oil Wash it Perform light maintenance That's about it. They shouldn't need to understand combustion timing, transmission gearing, or electrical diagnostics to get to work. The mechanic, however, lives in the details. They tune the system. They replace parts. They troubleshoot failures. They recommend upgrades. They understand how the pieces fit together. Technology is exactly the same in my mind. Users should be able to focus on their goals. Engineers should focus on the machinery. Features Are Parts This is where I think software conversations sometimes become backwards. A feature is a component. A login screen is a component. A dashboard is a component. A database is a component. An API is a component. AI integra

Recently open-sourced by Microsoft, pg_durable is a PostgreSQL extension that enables durable workflows to run natively inside the database, eliminating the need for external orchestration systems. By Sergio De Simone

Hello everyone! Following up on my previous post , Day 1 of my Modern Data Stack migration was an absolute rollercoaster of refactoring and deep data auditing. I’m moving our legacy system (spreadsheets and Qlik) into a robust pipeline using Python, ClickHouse, and dbt . Here is what went down over the last 24 hours. 1. From Messy Scripts to a Single, Parameterized Extraction Engine 🛠️ In the legacy setup, each company had its own folder, its own .env file, and its own duplicated Python extraction script. It was a maintenance nightmare. Yesterday, I completely refactored this structure: Centralized Configuration: Merged all separate environments into a single, global .env file at the root level, mapping all 8+ companies and their branches. Eliminated Code Duplication (DRY): Instead of having identical extraction logic copied across folders, I built a single, unified codebase. Now, we have one universal script for Sales, one for Stock, one for Orders, etc. The behavior changes dynamically based on the company argument we pass to the CLI (e.g., python -m extract.run extract --source company1 ). To speed up this refactoring, I used Claude to generate the initial application skeleton. Since the AI already had the context of our legacy extraction logic, translating it into this new clean architecture was incredibly smooth. 2. Highs and Lows: The Data Parity Challenge With the pipeline modernized, I ran the pilot ingestion for Company #1 . To minimize friction for our downstream BI consumers, I kept the ClickHouse Bronze tables structured 1:1 with the legacy CSV schemas. The Good News: The data ingestion into the Bronze layer worked flawlessly. Moving up to the Silver layer (where we do data cleaning and domain-specific transformations), everything validated beautifully. Row counts matched perfectly. The "Fun" Part (The $2 Million Gap): When I materialized the Gold layer (our consolidated group business models), I hit a massive wall. The new pipeline reported $2 million U

Adi Polak discusses the architecture required to transition from stateless prompts to state-aware, context-rich AI agents. Drawing on 15 years in distributed systems, she shares how engineering leaders can leverage Apache Kafka and Flink for real-time stream processing, dynamic memory tiering, and tool orchestration via MCP to solve token limits, cost spikes, and latency bottlenecks. By Adi Polak

Every low-code vendor says "we support customization." But supports is a weasel word — recoloring a button is customization, and rewriting a scheduling engine is also customization. What actually decides whether a platform locks you in is how far up its extensibility goes. Here's a ruler. The three levels of customization Level What you can do Most no-code A real dev framework L1 — Config Fields, forms, workflows, permissions, themes ✅ ✅ L2 — Extension Custom components, custom actions, external API calls, business rules ⚠️ limited ✅ L3 — Framework Modify/extend the core, custom engines, deep rewrites, source under control ❌ wall ✅ (when open/controllable) Where it stops is where your ceiling is. Plenty of no-code platforms are delightful at L1, then hit "can't do that" at L2/L3 — and you retreat to writing your own thing next to it. Now low-code is the burden. Why you get locked in Black-box SaaS — no source, so any extension point the vendor didn't expose is simply out of reach. Two sources of truth — your extension code and the platform's config live in different systems, so a platform upgrade breaks/voids your work. Crippled self-hosting — the on-prem edition quietly drops extension capabilities. Closed ecosystem — only their component marketplace; your stack can't get in. How model-driven + open source raises the ceiling One unified extension system — your extensions (custom fields/components/actions) and the platform itself are built on the same metadata. Extension isn't a bolt-on, it's a first-class citizen — upgrades don't wipe your customizations. Source under your control — open + self-hostable is what makes L3 framework-level extension actually possible: an extension point you can't reach, you can add. AI at the metadata layer — AI-generated extensions land in the same model, so they stay maintainable and evolvable. That's the road Oinone takes: 100% metadata-driven, front + back end open source, self-hostable — customization reaches L3. How to stress-tes

Most developers know Haskell as a language for functional programming, type safety, compilers, parsers, and beautiful mathematical abstractions. But can Haskell also be used to build an interactive engineering simulator? That was the motivation behind my project: Four-Bar Mechanism Haskell Simulator Repository: https://github.com/mohammadijoo/Four-Bar-Mechanism-Haskell This project is a browser-backed desktop-style GUI application written in Haskell. It visualizes, classifies, and animates a planar four-bar linkage mechanism, which is one of the most classical mechanisms in mechanical engineering, kinematics, and machine design. The GUI is built with Threepenny-GUI , so the interface runs in a local browser window, while the mathematical model and mechanism logic remain written in Haskell. For me, the interesting part was not only drawing a moving linkage. It was about connecting mechanism design theory , computational geometry , and functional programming in one small educational simulator. What is a four-bar linkage? A four-bar linkage is a closed-loop mechanical system made from four rigid links connected by four revolute joints. In this project, the four links are: Symbol Name Description g Ground link Fixed distance between pivots A and B a Input link Rotating link from A to moving pivot C b Output link Link from fixed pivot B to moving pivot D f Floating link / coupler Link connecting moving pivots C and D The fixed pivots are placed at: A = ( 0 , 0 ) , B = ( g , 0 ) The input link rotates by angle α . Therefore, point C can be computed directly as: C = ( a cos α ,; a sin α ) Point D is more interesting. It must satisfy two geometric distance constraints: ∣ D − C ∣ = f ∣ D − B ∣ = b So the simulator solves the position of point D using a circle-intersection method. One circle is centered at C with radius f . The other circle is centered at B with radius b . Where those two circles intersect, the mechanism can close. That is the basic geometric heart of the sim

Shareable blog post edition: https://andymaleh.blogspot.com/2026/06/andys-laws-of-ai-in-software-engineering.html Law #1: "The more Software Developers use AI, the more valuable Software Engineers who do not use AI become." Software Engineers who are masters at delivering Software without using AI will actually have increased job security the more Software Developers in the worldwide Software Development community rely on AI to deliver Software without having true mastery over Software Engineering. As more Software Developers become fully dependent on AI to build Software without truly understanding how AI gets work done, Software Engineers who do understand what is going on under the hood will dwindle and become more valuable than ever. In other words, they will have a competitive advantage over Software Developers who can only deliver Software features with AI as well as Software Developers who have not mastered Software Engineering. Also, there will always be a need for Software Engineers who can maintain the Software of AI itself. Law #2: "Software Developers benefit from AI in direct proportion to how weak they are in Software Engineering" The weaker Software Developers are at Software Engineering the more they benefit from AI. After all, AI learns from Master Software Engineers and then applies its learnings in code generation done for lower-level Software Developers who lack mastery in Software Engineering. So, users of AI simply place themselves lower in the expertise hierarchy to be on the receiving end of what Master Software Engineers feed AI with their code. This explains why many experts like Linus Torvalds do not find AI very useful while devs who have zero degrees and qualifications feel like they get a lot from AI. A beneficial thing to learn from this law is that it is more valuable for a Software Developer to hone in their Software Engineering skills (including the completion of university degrees) than to hone in their AI usage skills because if t

(The following example is fictionalized.) A notification template feature shipped six months ago. It let each tenant customize the messages sent to their own customers without requiring a back-end change every time the wording changed. The code reviewer could tell the design was hard to follow, especially the path from template to rendered value. But "this is hard to follow" is difficult to turn into a concrete objection when the feature works, the tests pass, and nothing is obviously unsafe or wrong. The design risk was real, but there wasn't an obvious bug to point to. QA signed off, and the feature went into production. Then a bug report came in: one customer had received a notification containing another customer's information. Somewhere in the notification pipeline, the system was leaking PII. At first, the fix sounded small: make sure notifications only render data belonging to the intended recipient. Then the assigned developer, who wasn't the original author, started looking for the place to make the fix. The templates were stored in the database. There were six template types, and each one populated its real values in a different part of the codebase. Some values came from customer-facing records, some came from internal workflow state, and some came from template-specific logic. The placeholder-to-value mapping lived somewhere else. Email and SMS channels shared part of the rendering path, but not all of it. Before the developer could decide where to fix the leak, they had to answer a more specific set of questions: Which placeholder rendered the wrong value? Where did that value come from? Which template types could use that placeholder? Did email and SMS resolve it the same way? What evidence would show that the leak was fully contained? The system was hard to change because it made the behavior hard to understand. What the developer needed was not just "clean code." They needed trustworthy signals they could use as evidence to answer harder questions: w

Netflix engineer Casey Bleifer shares how to achieve rapid, automated code changes across a massive, diverse software fleet. She discusses building an event-driven orchestration platform using composable, Lego-like steps, and explains how Netflix utilizes automated canary validation, compliance checks, and a custom "confidence metric" to eliminate the long tail of manual engineering migrations. By Casey Bleifer