标签：#pipeline

If your pipeline isn't idempotent, it isn't production-ready; it’s just a fragile script waiting to ruin your weekend. Most engineers treat "idempotency" as an academic concept for distributed systems papers, but in the trenches of fintech and healthcare, it’s the difference between a minor blip and a regulatory filing. If you can’t run your job five times in a row with the exact same input and get the exact same state, you aren't doing data engineering—you're doing data gambling. I’ve spent six years cleaning up the messes left by "append-only" thinking. I’ve seen millions of dollars in duplicate ACH transactions and patient records corrupted by "just one more retry" logic. This guide covers the patterns I use to make sure that when the scheduler kicks off at 3 AM, I can sleep through the alarm because the system knows how to fix itself. 1. Stop relying on "Append" mode The biggest sin in data engineering is assuming that your destination table is a clean slate. When a job fails halfway through, you don't want a partial load sitting in your production warehouse. Never use INSERT INTO blindly. If you are using BigQuery, Snowflake, or Databricks, use MERGE or overwrite-on-partition. If you are using SQL-based ELT, write your transformations to stage data in a transient table before swapping it into production. Never push directly to the target. -- The wrong way: INSERT INTO target_table SELECT * FROM staging -- The right way: Use an atomic swap or a MERGE statement MERGE INTO production . transactions AS T USING staging . transactions AS S ON T . transaction_id = S . transaction_id WHEN MATCHED THEN UPDATE SET T . amount = S . amount , T . status = S . status WHEN NOT MATCHED THEN INSERT ( transaction_id , amount , status ) VALUES ( S . transaction_id , S . amount , S . status ); Photo by 🇻🇪 Jose G. Ortega Castro 🇲🇽 on Unsplash 2. Partitioning is your safety net If your pipeline runs daily, your data must be partitioned by that day. If you are loading data without a

What Rebuilding Kafka From Scratch Actually Teaches You There's a gap between using a system and understanding it. Most engineers never close that gap, and honestly, most of the time that's fine. Kafka works. Topics, producers, consumers, pull the levers, ship the data. Done. But then you hit a weird latency spike, or a consumer group stalls in a way that doesn't match the docs, or replication starts behaving like it has feelings. And suddenly "I know the terminology" doesn't cut it anymore. That's exactly why this rebuild post is worth your time. The Abstraction Tax Every framework you use charges you an abstraction tax. The tax isn't the dependency. It's the mental model debt you carry when something goes wrong and you don't know what layer to blame. Kafka's tax is particularly sneaky because its concepts sound simple: topics are channels, partitions are buckets, offsets are counters. You can get productive fast. And then that simplicity starts lying to you. Why does lag spike when throughput looks fine? Why does adding consumers past the partition count do nothing? Why does a rebalance tank your throughput for 30 seconds? These aren't Kafka quirks. They're direct consequences of how the log is actually structured, consequences that become obvious the second you implement it yourself. What the Rebuild Exposes When you write the log append yourself, the offset model stops being abstract. An offset isn't just a cursor, it's a byte position in a segment file. Consumers aren't "reading from a partition," they're replaying a structured log from a known position. Replication isn't a background checkbox, it's a follower explicitly fetching and acknowledging write positions. A few things that tend to click when you go through this kind of exercise: Segment files and retention , Kafka doesn't delete old messages by scanning. It deletes whole segment files once they're past the retention boundary. If you've ever been surprised by how Kafka handles disk, this is why. Why par

In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi. By Srikanth Mamidala

Most teams don't consciously choose their messaging infrastructure. They inherit it. Someone used Service Bus on the last project, it worked fine, and now it's the default answer for every async communication problem that comes up. Two years later, you're bending it into shapes it was never designed for, and the operational pain gets blamed on "distributed systems being hard" rather than on the actual culprit: a tool being asked to do a job it doesn't fit. The problem isn't that Service Bus, Event Grid, or Kafka are bad. It's that they solve genuinely different problems, and conflating them doesn't just create technical debt — it creates architectural liability that compounds over time. The Real Difference Is the Communication Contract, Not the Feature List When you put these three tools side by side in a comparison table, you'll find overlapping columns. All three move messages between systems. All three have some delivery guarantee story. That's where the surface-level comparison breaks down and people make bad decisions. The more useful question is: what contract does your system need to uphold with the data it moves? Service Bus is fundamentally about reliable, ordered processing with strong delivery guarantees. It's designed for the case where every message matters individually, where you need competing consumers pulling from a queue, where poison message handling and dead-lettering are first-class concerns. If you're coordinating business process steps or handling financial transactions where exactly-once semantics matter, this is the right shape of tool. Event Grid is about reactive routing. Something happened in your infrastructure or your application, and you want other things to respond to it. It's push-based, fan-out-friendly, and optimized for low-latency notification rather than high-volume throughput. It's not trying to be a buffer. If you're triggering downstream workflows in response to blob uploads, resource state changes, or custom application even

1. Subsystem Decomposition 1.1 The Decomposition Problem A subsystem decomposes a codebase into smaller, cohesive units. Two primary axes of decomposition exist: Technical axis : grouping by component type (controller, service, model, view) Functional axis : grouping by business capability (cataloguing, circulation, etc.) 1.2 Tension Between Framework Prescriptions and Decomposition Strategy Organizing top-level subsystems functionally may create friction with frameworks that prescribe a technical-first structure. Concrete examples: Rails enforces model, view, and controller directories at the root level, making functional decomposition awkward without additional mechanisms like Rails Engines Sinatra (a microframework) imposes minimal structure, leaving architectural decisions entirely to the team Frameworks with rigid prescriptions constrain architectural choices. Frameworks with no structure shift the entire burden onto the team with no guidance. This second approach might be fine for teams that know what they are doing and how to shape the architecture properly. Not everyone needs guidance from the framework. 1.3 Contexts as a Middle Ground Phoenix provides contexts as a compromise: Explicit, guideline-oriented subsystems that enable functional decomposition without rigid enforcement Contexts define functional boundaries while allowing technical organization to remain nested within them Functional blocks may later evolve into microservices, but this is optional The same decomposition serves equally well in a modular monolith or a distributed architecture The choice depends on team needs, scaling requirements, and operational maturity, not on the decomposition strategy itself 2. Pipeline Topology and Data Flow 2.1 The Unix Pipeline Model Unix pipelines model data flow through a single stream connecting stdout to stdin. This forms a linear chain where each stage's output becomes the next stage's input. Key characteristics: Each stage has exactly one input and one o

The AI Implementation Process I Use With Every Client Most AI projects do not fail at the model. They fail in the six weeks before anyone writes a prompt, and in the six weeks after the demo lands in a Slack channel and nobody knows who owns it. I have run enough of these now (from one-off automations to multi-agent content systems running unattended) that the process has converged into something stable. This is the version I actually use. It has five phases: scoping, POC, integration, evaluation, operations. Each phase has an exit criterion. If we cannot meet the exit criterion, we do not move forward. That single rule has saved more projects than any clever architecture choice. Phase 1: Scoping (1 to 2 weeks, fixed price) Scoping ends with a written document that names the workflow being automated, the system of record it touches, the success metric in hours or dollars, the data we have access to, and the smallest possible first slice. No model is chosen yet. No code is written. If we cannot produce that document, the engagement stops here and the client keeps the document. The hardest part of scoping is resisting the urge to solve the interesting problem. Clients almost always describe the AI-shaped fantasy ("an agent that handles all support tickets") when the real opportunity is narrower and uglier ("triage tier-1 tickets that mention billing, route to the right queue, draft a reply for human approval"). The narrower version ships. The fantasy does not. I run scoping as three sessions: Workflow walkthrough. Someone who actually does the work shows me their screen for an hour. I record it. I take timestamps. The point is to find the moments where a human is doing pattern matching that an LLM can do, and to find the moments where they are doing judgment that an LLM should not do. Data audit. Where does the input live? Where does the output need to go? What is the auth story? If the data is locked inside a SaaS product with no API and no export, that is the projec

Overview Today's focus was on automating article publishing to CSDN and WeChat MP (微信公众号) using Playwright, after CSDN deprecated its public Open API. Key achievements include: injecting Markdown content into CSDN's dynamic editor, handling title input quirks, implementing QR code login for WeChat MP, updating the Dev.to API publisher, and consolidating platform configs into a single YAML file. We also fixed session log capture after a Claude Code update changed the log file path. Problems and Solutions 1. CSDN Open API Deprecation → Browser Automation Background : In early 2026, CSDN silently shut down its public Open API. All endpoints returned 404/403. We needed a fallback to keep publishing to China's largest developer platform. Solution : Use Playwright to simulate a real user login and article creation. The approach: Launch a headless Chromium browser. Navigate to CSDN's login page. Perform one-time manual login via QR code. Serialize cookies to csdn_cookies.json . On subsequent runs, load the cookies and skip login. Go to the editor, inject Markdown content via DOM manipulation, fill the title, and click publish. Code snippet : import asyncio from playwright.async_api import async_playwright async def publish_to_csdn ( title : str , content_md : str ): async with async_playwright () as p : browser = await p . chromium . launch ( headless = True ) context = await browser . new_context ( storage_state = " csdn_cookies.json " if exists else None ) page = await context . new_page () await page . goto ( " https://mp.csdn.net/mp_blog/creation/editor " ) # Inject content await page . evaluate ( f ''' () => {{ const editor = document.querySelector( ' .editor-content ' ); if (editor) {{ editor.innerHTML = ` { escaped_content } `; editor.dispatchEvent(new Event( ' input ' , {{ bubbles: true }})); }} }} ''' ) # Fill title await page . fill ( ' #title-input ' , title ) await page . click ( ' button:has-text( " 发布 " ) ' ) await page . wait_for_url ( " **/mp_blog/manage/ar