标签：#Engineering

Building a personalization engine for a multi-format content feed, without machine learning, and the testing process that forced us to rebuild it. TL;DR We run a collaborative cultural curation platform (think: user-submitted recommendations for movies, books, games, music, and long-form posts, all mixed into one feed) on a fairly ordinary PHP + MySQL stack. Over about a year we went through two full generations of the feed ranking algorithm. The first version solved the obvious problem (stop being purely chronological) but quietly failed at real personalization. The second version fixed that by rethinking what "user taste" even means, moving scoring out of SQL and into application code, and adding a layer of post-ranking business rules. This post walks through both generations, why the second one had to happen, and how we actually tested and calibrated a feed ranking system without a data science team or an ML pipeline. No exact weights, table names, or formulas below — just the engineering story. The starting problem: one feed, five content shapes Before personalization is even on the table, a multi-format feed has a normalization problem. Movies, books, games, music, and editorial posts live in different tables, with different columns, different publishing cadences, and engagement numbers on completely different scales. "1,000 likes" on a music post and "1,000 likes" on a book review are not the same signal. So the very first architectural decision — before any ranking logic existed — was building a unification layer that maps every content type into a shared shape (type, author, title, cover, category, engagement counters, timestamp) before any scoring happens. Everything downstream depends on that layer being consistent. Generation 1: a weighted ranker living inside a single SQL query The first real version of the algorithm — internally we called it the hybrid model — had a modest goal: get away from a purely chronological feed without building anything resembl

Why Sports Data Is Harder Than Most People Think Building believable cross-era simulations turned out to be less about the engine and more about the data underneath it. Here is what we learned. MicroLeague Dev Blog, Vol. 3 By Eddie Solar When we started building MicroLeague Sports, I assumed the simulation engine would be the hard part. The vision was ambitious enough to justify that assumption. Let fans ask whether the 1996 Bulls beat the 2017 Warriors. Whether the 1985 Bears could slow down Patrick Mahomes. Which Cowboys team was actually the greatest. Teaching software to play those games across eras felt like the mountain. I was wrong about which mountain it was. The engine is hard, but it is a solvable, bounded kind of hard. The data underneath it is a different animal. Like most developers approaching this for the first time, we figured sports data was largely a collection exercise: gather historical teams, player stats, schedules, and box scores, feed it to the model, done. That assumption fell apart almost immediately, and the reason it fell apart is the subject of this article. Sports data is not a collection problem. It is an identity problem. Franchises do not stay the same thing. Players are not one entity. And the historical record does not agree with itself. The Real Problem Is Modeling Identity Over Time Volume 2 covered the era problem: statistics are confounded by the conditions that produced them, so a raw number pulled across decades lies to you. That is a normalization challenge, and it is real. But normalization assumes you already know what you are normalizing. Before you can compare the 1992 Cowboys to the 2023 Chiefs, your system has to have a confident answer to a more basic question: what exactly is a "team," and what exactly is a "player," when your dataset spans a hundred years? Those sound like trivial questions. They are not. They are the questions that ate most of our early engineering time, and getting them wrong quietly corrupts ever

Instacart introduced Blueberry, an AI-assisted incident response system that helps on-call engineers investigate production issues faster. It combines AI agents, operational data, and historical incident knowledge to generate grounded root cause hypotheses in Slack. It uses parallel subagents, MCP integrations, and incident history to reduce investigation time while keeping engineers in control. By Leela Kumili

Artificial intelligence is rapidly changing how engineering teams respond to production incidents, offering the ability to summarize incident channels, analyze unfamiliar code, suggest remediation steps, generate pull requests, and increasingly assist with diagnosis. By Craig Risi

In today's digital world, businesses face the challenge of developing, deploying, and scaling applications faster and more efficiently. One of the key technologies supporting this transformation is container technology. What are Containers and Why Are They Important? Containers allow applications to be packaged into lightweight, self-contained, and portable units that can run consistently in any environment—from a local development machine to a cloud platform. This reduces dependencies and significantly simplifies application deployment and scalability. Unlike virtual machines (VMs), containers share the operating system kernel, making them more resource-efficient. This leads to higher efficiency and allows businesses to run more applications on the same infrastructure. Introduction to Kubernetes: Orchestration of Containers While containers represent a revolutionary approach to developing and running applications, it’s not enough to simply have containers. Once applications consist of dozens or hundreds of containers, managing, orchestrating, and scaling them becomes critical. This is where Kubernetes comes in. Kubernetes is the world’s most widely used container orchestration platform. It enables the automatic deployment, scaling, and management of containerized applications in clusters. With Kubernetes, companies can ensure their applications are always available, automatically recover from failures, and roll out new versions without downtime. Azure Kubernetes Services (AKS): Kubernetes in the Cloud Azure Kubernetes Services (AKS) is Microsoft’s fully managed Kubernetes solution. With AKS, businesses benefit from simplified Kubernetes deployment by offloading infrastructure management to Microsoft. This means you can focus on developing and scaling your applications while AKS simplifies the management and maintenance of Kubernetes clusters. Benefits of AKS: Fully managed: AKS takes care of the management and patching of Kubernetes, allowing businesses to focus on

Introduction As software applications grow in size and complexity, managing object creation becomes challenging. Creating objects directly using constructors can result in tightly coupled code that is difficult to maintain and extend. The Factory Method Design Pattern solves this problem by separating object creation from object usage. It provides a flexible and reusable approach for creating objects, making applications easier to modify and scale. What is the Factory Method Design Pattern? The Factory Method Design Pattern is a Creational Design Pattern that provides an interface for creating objects without specifying their exact classes. Instead of directly instantiating objects using the new keyword, a factory class creates and returns the required object. Definition Factory Method Design Pattern: A creational design pattern that defines an interface for creating objects while allowing subclasses or factory classes to decide which object to instantiate. Why Do We Need It? In traditional programming: The client creates objects directly. Code becomes tightly coupled. Adding new object types requires modifying existing code. Maintenance becomes difficult. The Factory Method pattern solves these problems by centralizing object creation inside a factory class. How It Works The client requests an object from the factory. The factory checks the requested type. The appropriate concrete object is created. The factory returns the object to the client. The client uses the object without knowing how it was created. Java Example interface Shape { void draw(); } class Circle implements Shape { public void draw() { System.out.println("Drawing Circle"); } } class Rectangle implements Shape { public void draw() { System.out.println("Drawing Rectangle"); } } class ShapeFactory { public Shape getShape(String type) { if(type.equalsIgnoreCase("Circle")) return new Circle(); if(type.equalsIgnoreCase("Rectangle")) return new Rectangle(); return null; } } public class FactoryPatternDem

Understanding the Prototype Design Pattern in Java Introduction When developing software, there are situations where creating a new object from scratch is expensive or time-consuming. For example, an object may require complex initialization, database access, or extensive configuration. In such cases, instead of creating a new object every time, we can duplicate an existing object. This is where the Prototype Design Pattern becomes useful. The Prototype Design Pattern is one of the Creational Design Patterns in Java. It allows developers to create new objects by cloning existing ones rather than instantiating them using constructors. What is the Prototype Design Pattern? The Prototype Design Pattern creates new objects by copying an existing object, known as the prototype. This approach improves performance by avoiding repeated initialization and allows developers to create multiple similar objects efficiently. In Java, cloning is commonly implemented using the Cloneable interface and overriding the clone() method. Why Use the Prototype Pattern? The Prototype Pattern offers several benefits: Reduces the cost of object creation. Improves application performance. Simplifies the creation of complex objects. Avoids repeated initialization code. Makes object creation more flexible. Real-World Example Imagine an online shopping application where thousands of product objects share similar properties. Instead of creating every product from scratch, the application can clone a prototype product and modify only the required attributes such as name or price. Other real-world examples include: Document templates Game characters Employee records Vehicle configurations Graphic design objects UML Structure The Prototype Design Pattern generally includes: Prototype Interface – Declares the clone operation. Concrete Prototype – Implements the cloning functionality. Client – Creates new objects by cloning existing prototypes. Java Implementation Step 1: Create the Prototype Class cla

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible." This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic. 1. The Actionability Gap (The Golden Question) The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is: "Does the company have the operational capacity to make a decision in milliseconds?" If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch. 2. The Hidden Complexity and the Cloud Bill Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug. Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule. 3. "Micro-Batch" Solves 99% of Your Problems There is a perfect middle ground that the hype industry tries to ignore: the micro-ba

Dumanshu Goyal discusses optimizing data layers for low-latency workloads like AI feature stores. Drawing lessons from NASA's Space Shuttle, he explains how proxy architectures introduce hidden CPU costs, elevated tail latencies, and blast-radius risks. He demonstrates how direct-access Valkey architectures achieve microsecond latency, improve resilience, and slash infrastructure costs. By Dumanshu Goyal

AI workflows have two needs that trade off directly. Running reliably in production requires persisting and distributing every step so it survives crashes, deploys, and restarts. But that same machinery is what makes runs too heavy for the fast, throwaway loop you need to check an LLM's output quality. The properties that buy durability are the ones that kill iteration speed. By Mateus Moury

Vercel Labs has introduced Zero, an experimental systems programming language aimed at AI rather than human users. It employs unique features like a specific toolchain contract and structured error messages. Reaching version 0.3.4, it compiles to native binaries for major operating systems. The language prioritizes size, speed, and agent usability, though it is still in development. By Daniel Curtis

🎯 What a “Stacked PR” Is (and Why You’ll Want One) A stacked pull request (sometimes called a stacked PR , stacked diff , or dependent PR ) is a series of PRs that build on top of each other, each one containing a small, logically‑isolated change. main ──► A ──► B ──► C │ │ │ │ │ └─ PR‑C (depends on B) │ └─ PR‑B (depends on A) └─ PR‑A (directly on main) A is based on main . B is based on A (its head). C is based on B , etc. When you eventually merge the stack in order (A → B → C), each change lands cleanly, and reviewers can focus on one cohesive piece at a time. Why Stack PRs? Problem Stacked PR Solution Huge, monolithic PRs that are hard to review & cause long CI times Break the work into bite‑size PRs (e.g., “feature flag”, “data model”, “UI”) Inter‑dependent changes (e.g., a new API + its consumer) Each dependent change lives in its own PR, but they still get tested together because they are built on top of each other Rebasing on main constantly drags in unrelated changes Only the bottom PR needs to be rebased onto main ; the rest stay on top of it Need to ship part of a larger change early Merge the first PR in the stack; the rest stay pending until they’re ready CI resources Only the bottom PR runs the full suite against main ; higher PRs can run a lighter subset because they already passed lower‑level tests 📦 The Landscape of Tools (as of 2026) Tool / Service Key Features Installation / Setup Typical Workflow ghstack (GitHub CLI plugin) - Creates stacked PRs automatically from a series of commits. - Handles base‑branch updates, resolves merge conflicts, and can re‑stack after rebases. - Works with GitHub's GraphQL API, so you get “dependent PR” links in the UI. pip install ghstack (or brew install ghstack ). Requires a personal access token with repo scope. bash git checkout -b feature/stacked\n# create many commits …\nghstack push\n# later, after rebasing on main\nghstack rebase . | | GitTown (aka git-town ) | - git town ship can ship a stack of dependent br

A Deep Dive into the Memory Model From Source Code to Machine Instructions A five-part journey through compilers, executables, virtual memory, and the CPU Introduction: What Really Happens When Code Runs Consider a simple C program: include <stdio.h> int value = 10; int add(int a, int b) { return a + b; } int main() { int x = 5; int result = add(x, value); printf("%d", result); return 0; } Most programmers look at this and see only the visible outcome: 5 + 10 = 15 But behind that single printed number lies a much deeper story. Where does the data actually live? Who moves it from one place to another? How does the CPU find the instructions it needs to run? And how does the result finally make its way to the screen? Answering these questions means understanding a concept that many programmers use daily but rarely examine closely: the memory model. What Is a Memory Model, Really? Ask most developers what a "memory model" means, and the answer usually comes back in two words: stack and heap. That answer isn't wrong - it's just incomplete. A memory model is really a description of five things at once: How data is stored How data is accessed How long data exists Who is responsible for managing that lifetime How different parts of a system communicate through memory A program never leaps directly from C source code into RAM. Several distinct layers sit between the two, each one translating the layer below it into something the layer above can reason about. This article walks through all of them, one at a time, and then reassembles the full picture. The Four Layers, at a Glance Layer What It Deals With Typical Concepts 1. Programming Language Human-readable code scope, lifetime, ownership 2. Compiler Translating code to instructions registers, optimization, assembly 3. Operating System Running the program as a process virtual address space, .text/.data/.bss 4. CPU Architecture Executing raw instructions registers, cache, pipeline, ALU The rest of this article follows a sing

"A Trie isn't designed to store words. It's designed to make finding everything that shares the same beginning incredibly efficient." In the previous article, we explored a different kind of software problem. Some systems don't search using complete values. Instead, users provide only part of the information they know. The system must immediately suggest possible matches. Once you recognize that requirement, another question naturally follows. How should the system organize data so prefix searches become fast and natural? This is exactly the problem a Trie solves. Think About a Dictionary Imagine opening a physical dictionary. Suppose you're looking for the word: Application Do you start reading from page one? Of course not. You first go to the words beginning with: A Then you narrow further. Ap Then: App Every additional letter reduces the search space. A Trie works in a very similar way. Instead of repeatedly searching through every word, it follows the characters one by one. What Is a Trie? A Trie is a tree-like data structure where each node represents a character. Words that begin with the same characters share the same path. Consider these words. car card care cart A Trie stores them like this. Root ↓ c ↓ a ↓ r ├── end ├── d → end ├── e → end └── t → end Notice something interesting. The prefix: car is stored only once. Every longer word simply continues from that shared path. Every Data Structure Answers a Different Question By now we've seen several data structures, each solving a different design problem. A HashMap asks: Where is this exact object? A Heap asks: Which item has the highest priority? A Queue asks: Which task should happen next? A Stack asks: What is the current working context? A Trie asks: What begins with these characters? Choosing the right data structure starts with identifying which question your software needs to answer. Inserting a Word Imagine inserting: cat The Trie creates a path. Root ↓ c ↓ a ↓ t Now insert: car The beginning alread

Uploading a PDF and calling a translation model looks like a two-step feature. In production, it is a job pipeline with untrusted input, two different extraction paths, several expensive stages, and an output that can be fluent while still being wrong. That distinction matters for a small SaaS team. The translation request may come from support, sales, or an internal operations task. Nobody wants to operate a document platform, but the workflow still needs to answer basic questions: Was the upload actually a PDF? Does the file contain selectable text or scanned page images? Can a retry create a second charge or a conflicting result? What happens when page 37 fails after the first 36 pages succeed? How do we know the translated PDF is not blank or visually broken? When are the source and result deleted? The translation model is one component. Reliability comes from the system around it. Define the Job Contract First I would not let a file reach an extractor until the API has established a narrow contract. For example, a translation request might include: type TranslationStyle = " general " | " technical " | " academic " ; interface CreateTranslationJob { uploadId : string ; sourceLanguage : string | " auto " ; targetLanguage : string ; style : TranslationStyle ; idempotencyKey : string ; containsRestrictedData : boolean ; } The request should be rejected when the source and target languages are identical, the upload is missing, the target language is unsupported, or policy says the document cannot leave an approved environment. File validation should also be explicit. Do not trust the filename or browser-supplied MIME type. Check at least: the actual byte size; the file signature; whether the parser can open the document; whether the PDF is encrypted; the page count; whether the job fits the account or product limit. A 20 MB limit is simple to explain in a user interface, but size alone is not a good predictor of work. A compressed 200-page text PDF can be smaller th

AWS introduced Kiro Crew on Tuesday as a new open-source orchestration platform. This tool aims to help businesses shift from interactive AI coding assistants toward autonomous engineering workflows. The system manages tasks across various repositories and developer tools over multiple work sessions to increase overall efficiency. Orchestrating autonomous development cycles Kiro Crew goes beyond simple code generation by coordinating multiple AI agents simultaneously. It schedules recurring work and maintains project context even when a session ends. This allows the system to integrate with standard developer tools for investigating incidents or monitoring pull requests. It triages tickets and automates software engineering tasks while developers are away from their workstations. The platform functions as an application layer that turns AI coding agents into self-learning teammates. It features persistent memory and multi-agent orchestration tools to ensure continuity. Security remains a priority with features like sandboxing and signed audit logs. Users can monitor activity through a dedicated web and desktop dashboard designed for transparency. Before its public release, the project existed inside Amazon as an internal tool named MeshClaw. More than 39,000 Amazon builders adopted it in less than six months. This internal success paved the way for the current open-source offering. Companies can deploy the platform entirely within their own environments, such as on local laptops or virtual machines. Reference applications and practical use cases AWS launched several reference applications to show how the platform functions in real-world scenarios. DevFleets manages worktrees, while Issue Radar handles the triage of pull requests and tickets. Task Runner focuses on executing engineering tasks that require a long duration to complete. These apps use specific interfaces combined with the core orchestration engine. These tools are not standalone products but rather exam

Most Snowflake-to-Databricks migrations get sold on cost and delivered on something else. The credit line item is what gets the project funded, but the teams that finish happy are usually the ones that moved for a different reason: they wanted ML, streaming and GenAI workloads living next to the analytics data instead of shuttling between two platforms. If your only justification is the bill, read the breakeven section below before you commit — the honest number is longer than the deck says. We're a Databricks shop , and we've written elsewhere about how to choose between the two platforms if you haven't committed yet. This post assumes you have. What actually changes underneath The two platforms look similar from a SQL console and are structurally different behind it. The mapping worth internalising before planning anything: Layer Snowflake Databricks Storage Proprietary micro-partitions inside Snowflake Delta Lake files in your own S3/ADLS/GCS bucket Compute Virtual warehouses, T-shirt sized Job clusters, all-purpose clusters, SQL Warehouses, Photon Governance Role hierarchy, row access policies, masking policies Unity Catalog across tables, models, notebooks, dashboards Sharing Secure Data Sharing Delta Sharing (open protocol) Billing unit Credits DBUs, priced differently per compute type The storage row is the one with the most downstream consequences. On Snowflake, storage and compute are separate line items on the same bill; on Databricks, storage is your cloud provider's problem and your cloud provider's invoice. That's a genuine benefit — the data stays readable by other engines — but it also means your "Databricks cost" and your "data platform cost" stop being the same number, and finance needs to know that before the first invoice arrives. Pick a strategy before you pick a tool Three patterns, and the choice determines everything after it: Lift-and-shift. Replicate schemas one-to-one, translate the SQL, cut over. Fastest, and it faithfully preserves every

Instead of one huge, un-reviewable pull request, teach coding agents to decompose work into a clean, ordered stack with GitHub stacked pull requests. The post Turn one giant AI-generated pull request to a reviewable stack appeared first on The GitHub Blog .

Quotient CEO Lizzie Matusov explains why soaring AI spend often fails to improve software delivery. She presents a research-backed AI maturity framework designed to help engineering leaders move beyond vanity metrics like token usage, align organizational AI adoption, and address critical bottlenecks across the software development life cycle to deliver measurable business outcomes. By Lizzie Matusov

When you start as a junior developer, you think software engineering is about writing code. A few years in, you think it's about choosing the right architecture and frameworks. After ten-plus years in the trenches - shipping features, surviving on-call disasters, and watching "perfect" codebases turn into unmaintainable monsters - you realize the truth: Software engineering is mostly about managing complexity, human communication, and trade-offs. Here are 25 mistakes I made, witnessed, or had to clean up over the past decade. Hopefully, reading them saves you a few years of painful trial and error. 1. Code & Architecture 1. Abstracting Too Early The DRY (Don't Repeat Yourself) principle is heavily drilled into beginners, but premature abstraction is far worse than duplicate code. Abstracting before you have 3–4 concrete use cases leads to rigid, over-engineered abstractions that are nightmare-inducing to change. Duplication is far cheaper than the wrong abstraction. 2. Falling in Love with "Clever" Code If your code requires a three-minute internal monologue or a complex diagram just to parse a single line, it's not smart - it's a liability. Write obvious, clear, and boring code. Your future self on a 2 AM incident response call will thank you. 3. Misunderstanding the Cost of Dependencies Adding a third-party library to solve a small problem feels like a quick win. In reality, every dependency is a contract you sign with an external team. You inherit their bugs, security vulnerabilities, breaking updates, and maintenance cycles. Ask yourself: Can we build the 5% of this library we actually need in 20 lines of code? 4. Over-Architecting for Scale You Don't Have Designing a system for 10 million daily active users when you currently have 500 is a classic trap. You end up with distributed microservices, message queues, and complex caching strategies that slow down development speed by 10x. Build for today's scale, but keep the boundary clean enough to refactor tomorrow