标签：#Ring

How do you ship a software product with 0 lines of manually-written code? A friend asked me this today, and I realized I didn't have a simple answer. So I dug deeper. It turns out the answer is in how you engineer your harness. Wait now, what? What is harness engineering? There is a reason this is the most important trend right now around coding agents. The biggest question these days is how to validate AI-generated code without reading every single line. How do you make sure an agent doesn't break production or delete your data? A blog by OpenAI shared an interesting experiment where a team of 3 engineers have built and shipped an internal beta of a software product with 0 lines of manually-written code. Every line of code: application logic, tests, CI configuration, documentation, observability, and internal tooling, has been written by Codex. How did they do it? They didn't write the app. They designed the harness. What exactly is a harness? Think of an AI agent like a powerful racehorse. The harness is the track, the blinders, and the jockey's reins that keep it running in the right direction instead of jumping into the stands. As my colleague Arthur Thompson explained today: for agents — the harness is composed of all the deterministic components that wrap the LLM. Balaji Subramaniam details those deterministic components in his blog — the orchestration layer, execution sandboxing, state persistence, and verification tools. If you want to build reliable agentic systems, your job shifts from writing the logic to designing the environment. Here is what you need to focus on: Set strict boundaries: Don't let the agent guess what it can touch. Enforce strict access rules (like confining it to a specific sandbox) so it can't accidentally wipe out production data. Build "Repair Loops": Agents will inevitably make mistakes. A great harness automatically traps errors, like a failed build or a test failure, and feeds those clean logs right back to the agent so it can fix

Preventing Cache Penetration in Spring Boot Using Redis and Bloom Filters Cache penetration occurs when high-frequency requests query non-existent keys, bypassing the Redis cache completely and hitting the relational database directly. Here is how we set up a Bloom Filter guard layer in front of Redis and PostgreSQL. 1. The Bloom Filter Guard Concept A Bloom Filter is a space-efficient probabilistic data structure that tests whether an element is definitely NOT in a set or MIGHT be in a set. @Component public class CachePenetrationGuard { private final BloomFilter < String > accountFilter ; public CachePenetrationGuard () { // Expected insertions: 500,000, False positive probability: 0.01 (1%) this . accountFilter = BloomFilter . create ( Funnels . stringFunnel ( StandardCharsets . UTF_8 ), 500000 , 0.01 ); } public void registerKey ( String accountId ) { accountFilter . put ( accountId ); } public boolean mightContain ( String accountId ) { return accountFilter . mightContain ( accountId ); } } 2. Service Layer Verification Before querying Redis or PostgreSQL, verify with the Bloom Filter: @Service public class AccountService { private final CachePenetrationGuard guard ; private final RedisTemplate < String , AccountDto > redisTemplate ; private final AccountRepository repository ; public AccountDto getAccount ( String accountId ) { // Step 1: Bloom filter pre-check if (! guard . mightContain ( accountId )) { return null ; // Instant rejection, saves DB from unnecessary lookups } // Step 2: Redis lookup AccountDto cached = redisTemplate . opsForValue (). get ( "acc:" + accountId ); if ( cached != null ) return cached ; // Step 3: DB fetch and cache populate AccountDto dbResult = repository . findByAccountId ( accountId ); if ( dbResult != null ) { redisTemplate . opsForValue (). set ( "acc:" + accountId , dbResult , Duration . ofMinutes ( 30 )); } return dbResult ; } } 3. Summary Combining Bloom Filters with TTL jitter in Redis shields backend databases from cache

What the court actually held Amazon sued Perplexity in November 2025 over its Comet browser, pleading the federal Computer Fraud and Abuse Act and California's Comprehensive Computer Data Access and Fraud Act. A district court granted a preliminary injunction in March 2026. The Ninth Circuit stayed it pending appeal, and in August 2026 vacated it. The reasoning is the part worth carrying away. On the record before the panel, the systems were being accessed by Amazon's own customers, signed into their own accounts, using software they had chosen. Perplexity was not the one accessing Amazon. On that basis Amazon was unlikely to prevail on a statute written about unauthorised access. It is the first federal appellate ruling on whether AI agents acting for a user may access an online platform, and the panel was careful to say it was deciding that record rather than announcing a doctrine. What it did not hold It did not say agents are welcome, and it did not say a site has lost control of its own front door. Contract claims were not what the panel found weak. Terms of service, trademark questions and state-law theories are all untouched. A different record with different facts, particularly one where the agent operates at scale rather than for one signed-in customer, could come out differently. The useful summary is narrow and worth stating without decoration: computer-misuse law is a weak instrument against software a customer chose to run on their own account. The distinction the ruling turns on Crawler User's agent Acting for Its operator One signed-in customer Scale Many sites, high volume One session at a time Authenticated Usually not As the customer Data ends up In the operator's product In front of the person who asked The ruling's reasoning Does not apply Applies Most blocking rules in the wild do not make this distinction. A blanket refusal of automated access catches a customer's own agent alongside the scraper it was aimed at, and those two are commercially o

Warum KI-Agenten Leitplanken brauchen: Operatives Gedächtnis statt Over-Engineering Ki-Agenten sind nicht böse. Sie sind nicht einmal unzuverlässig im klassischen Sinne. Das eigentliche Problem ist vielmehr ihre beständige Bereitschaft zu helfen, gepaart mit einem fehlenden Verständnis für die Grenzen ihrer Befugnisse. Sie wollen das Problem lösen, das ihnen gestellt wird, oft mit einer Aggressivität, die menschliche Manager selten aufbringen. Wenn ein Agent eine Produktionsdatenbank bereinigen soll, tut er es. Wenn er eine Datei löschen soll, die er für überflüssig hält, weil sie im aktuellen Kontext nicht erwähnt wurde, wird er es tun. Wir haben in unserem Engineering-Team 182 sogenannte Guards implementiert. Diese Zahl klingt auf den ersten Blick nach extremem Over-Engineering. Nach 182 Prüfungsschritten, die vor jeder Aktion eines autonomen Agents laufen, könnte man meinen, wir hätten ein unverhältnismäßig komplexes System gebaut. Doch jeder einzelne dieser Guards entstand nicht aus theoretischer Vorsicht. Jeder einzelne steckt in einem echten Vorfall, bei dem ein Agent ohne diese Barriere etwas getan hätte, das wir nicht rückgängig machen konnten oder das immense Kosten verursacht hätte. Dies ist kein Over-Engineering. Das ist operatives Gedächtnis. Was ist ein Guard? Ein Guard ist eine schlanke, deterministische Prüflogik, die zwischen der Entscheidungsfindung der KI und der tatsächlichen Ausführung einer Aktion liegt. Die KI plant eine Aktion. Zum Beispiel: "Führe einen SQL-Update-Befehl auf der Tabelle 'users' aus." Bevor dieser Befehl an die Datenbank geschickt wird, läuft er durch eine Pipeline aus Guards. Ein Guard fragt nicht nach dem "Warum" der KI. Das ist die Domäne des Large Language Models. Der Guard fragt nach den "Was" und "Wie" der realen Welt. Er prüft Fakten, nicht Absichten. Ein typischer Guard könnte so aussehen: def check_write_scope ( agent_action : dict ) -> bool : """ Stellt sicher, dass Schreiboperationen nur auf spezifisch erlaubten Tab

Most data engineers write pipelines the way most people write shell scripts: run it, eyeball the output, ship it. That works right up until a schema changes upstream, a null slips through a join, or someone "fixes" a transformation and silently breaks three downstream tables. By then the bug isn't your problem anymore — it's a bad number in someone's dashboard. Software engineers solved this problem decades ago with automated testing. Data engineering has been slower to adopt the habit, partly because our code touches messy external reality (files, databases, clusters) in a way a typical web app doesn't. But that's exactly why testing matters more here, not less. This article is a practical, DE-flavored crash course in pytest — the dominant Python testing framework — plus the patterns you actually need for pandas, Polars, and PySpark pipelines. Why bother testing a data pipeline? A few concrete failure modes that tests catch before production does: A column gets renamed upstream and your join silently produces all-null matches instead of erroring. A "cleaning" function that's supposed to drop duplicates accidentally drops valid rows too. A date-parsing function works on your local machine's locale and breaks in the CI environment. A refactor changes an aggregation from sum to mean and nobody notices until finance asks why revenue looks 90% smaller. None of these require exotic testing techniques. They require the habit of writing small, deterministic checks against small, deterministic inputs — which is exactly what pytest is built for. Where pytest fits — and where it doesn't Before diving in, it's worth being precise about scope, because "testing a data pipeline" actually covers two different questions, and conflating them is a common source of confusion: Is my code correct? Given a known input, does the transformation logic produce the right output? This is a property of your code , and it doesn't change based on what day it is or what a source system decided to

developer context switching security DevSecOps flow state developer velocity security alerts batch security patching ROI cost of context switching developer productivity security security alert fatigue developer cognitive load ad-hoc security patching interrupting developer flow engineering vp productivity metrics DevSecOps velocity context switching recovery time 23 minute recovery context switch batching security alerts SLA-backed fix campaigns security SLA for developers minimizing context switching feature delivery vs security developer experience DevSecOps The Real Cost of Context Switching What Security Alerts Actually Do to Developer Flow Back to blog What interruptions actually cost Is it worse for developers specifically? The research says probably yes The alert volume isn't imaginary — but be careful which numbers you cite The fix: batch the routine work, protect the calendar The important exception: not everything can wait for the batch A more honest way to estimate the ROI The takeaway Sources The Real Cost of Context Switching: What Security Alerts Actually Do to Developer Flow Companies keep investing in better frameworks, tighter deployment gates, and broader platform suites — and feature delivery keeps getting slower anyway. For engineering leaders trying to explain that paradox to the board, the usual suspects (headcount, tooling, talent) rarely hold up. The more useful place to look is something less visible: how often developers get pulled out of what they're doing, and what it costs them to get back in. As "shift-left" security practices spread, developers absorb a steady stream of vulnerability alerts, automated pull-request comments, and one-off Jira tickets throughout the day. The goal — a more secure codebase — is the right one. The delivery mechanism is often the problem. Scattering fixes across random moments in the workday erodes productivity without necessarily making the codebase safer any faster. The alternative a growing number of engi

Episode 1: Was in einer ZST-Datei passiertEpisode 1: Was in einer ZST-Datei passiertZST-Dateien begegnen uns immer häufiger bei großen Downloads, Softwarepaketen, Backups und Serverdaten. Sie sind oft deutlich kleiner als die ursprünglichen Dateien und lassen sich trotzdem sehr schnell wieder entpacken. Doch wie funktioniert das? Warum werden Dateien komprimiert? Eine Datei besteht aus Daten. Je mehr Daten sie enthält, desto mehr Speicherplatz wird benötigt und desto länger dauert ihre Übertragung. Kompression versucht, dieselben Informationen mit weniger Daten darzustellen. Beim späteren Entpacken muss daraus wieder exakt die ursprüngliche Datei entstehen. Nach dem Entpacken ist die Datei Bit für Bit identisch mit dem Original. Es wird nichts weggelassen und nichts vereinfacht. Wiederholungen benötigen unnötig viel Platz Betrachten wir diesen Satz: Kleine Katzen kuscheln auf kleinen Kissen, junge Katzen kuscheln auf bunten Kissen und alte Katzen kuscheln auf weichen Kissen.Die folgenden Teile kommen mehrfach vor: A = Katzen kuscheln auf B = KissenWenn wir die wiederkehrenden Textteile durch die Variablen A und B ersetzen, können wir den Satz kürzer darstellen: Kleine A kleinen B, junge A bunten B und alte A weichen B.Damit ist der Text noch nicht vollständig. Zusätzlich müssen wir speichern, wofür A und B stehen: A = Katzen kuscheln auf B = KissenAus diesen Informationen lässt sich der ursprüngliche Satz wiederherstellen. Jedes A wird durch Katzen kuscheln auf und jedes B durch Kissen ersetzt. Das ist bereits die grundlegende Idee der verlustfreien Kompression: Wiederkehrende Daten werden nicht jedes Mal vollständig gespeichert. Stattdessen werden sie einmal gespeichert und anschließend durch kürzere Verweise ersetzt. ### Zstandard verwendet keine Variablen Unsere Variablen A und B dienen nur dazu, das Prinzip verständlich zu machen. Zstandard versteht weder Wörter noch Sätze. Es weiß nicht, was Katzen oder Kissen sind. Für das Programm besteht eine Datei lediglich

A preview of the September cohorts of the InfoQ Online Certification Programs, and the facilitators leading them: Luca Mezzalira, Michelle Brush, Zichuan Xiong, and Premanand Chandrasekaran. By Artenisa Chatziou

The symptom is simple: you open /dashboard?search=invoice&page=2 , copy an old snippet, and get undefined , stale values, or the wrong API entirely. The root cause is that Next.js now has two routing models, and the correct query-string API depends on where you read the params: App Router Server Component page: use the searchParams prop App Router Client Component: use useSearchParams() Shared client component across both routers: useSearchParams() still works Here is the exact fix for each case. The App Router server-side fix If you are inside app/.../page.tsx , use the page prop. In the current Next.js docs, searchParams is a promise in modern App Router pages. // app/dashboard/page.tsx export default async function Page ({ searchParams , }: { searchParams : Promise < { [ key : string ]: string | string [] | undefined } > }) { const { search = '' , page = ' 1 ' } = await searchParams return ( < main > < h1 > Dashboard </ h1 > < p > Search: { search } </ p > < p > Page: { page } </ p > </ main > ) } Use this when the query string affects data fetching, pagination, filtering, or metadata for the page itself. The App Router client-side fix If the component is interactive and already marked 'use client' , use useSearchParams() from next/navigation . ' use client ' import { useSearchParams } from ' next/navigation ' export default function SearchSummary () { const searchParams = useSearchParams () const search = searchParams . get ( ' search ' ) ?? '' const page = searchParams . get ( ' page ' ) ?? ' 1 ' return ( < p > Searching for < strong > { search || ' everything ' } </ strong > on page { page } </ p > ) } Two details matter: useSearchParams() is read-only. In the App Router docs, Next.js explicitly recommends the page searchParams prop if you are already in a Server Component page. The shared-component pattern that survives both routers This is the cleanest answer if you are migrating gradually or sharing a search bar between pages/ and app/ . ' use client ' impo

Nathan E. Sanders and I are writing a series of essays on real-world examples of democratic technologies for The Renovator . I haven’t been posting the full text on the blog because they’re a bit long, but here are links. Part 1 is about the Japanese digital democracy party, Team Mirai. Part 2 is about the Swiss Public AI model, Apertus. Part 3 is about the civic technologists of Open Knowledge Brazil. And the new one, Part 4 , is about civic AI in Scotland.

What building a warehouse management system taught me about the data operational software leaves behind — and the engineering it takes to make that data trustworthy. The second that outlives itself A picker holds a handheld scanner, points it at a carton, and pulls the trigger. There's a beep. They type 10, confirm, and move to the next location. The whole thing takes about a second. For a long time I thought of my job as making that second work. I built the screen, the endpoint behind it, the repository behind that. My definition of done was that the user completed the workflow, the API returned success, and the right rows landed in the database. What changed my thinking was noticing what was still there afterwards. The screen closes, the session ends, the app ships a new version, the picker changes jobs, the device is replaced. The row stays — and the row isn't a record of a UI interaction. It's a durable claim about the physical world: at this time, this person, on this device, ten units of this product moved. The application is the instrument. The data is the measurement. A measurement is only ever worth what the instrument's precision allows. This article is about the gap between those two definitions of done, and the specific decisions — retry semantics, timestamps, identity, status codes, conflict resolution — that determine which side of it you land on. Almost all of them get made by application developers, inside feature work, long before anyone tries to analyze anything. What warehouse owners actually do with this data now Worth being concrete about the stakes first, because "data quality matters" is the kind of statement everyone agrees with and nobody acts on. What's changed isn't that owners suddenly became analytical. It's that operational systems started producing enough granular, attributed, time-stamped movement data that previously unanswerable questions became answerable. Inventory accuracy is a working-capital decision. Stock you can't trust is s

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback....

서론 AI Native 커리어 캠프에 참여한 지 한달이 지났다. 처음 5일 간은 최신 AI 기술 트렌드나 현업에서 AX 전환이 어떻게 되고 있는지, 포트폴리오에 어떻게 연결하는게 좋을 지에 대한 강의를 들었고, 현재까지는 AI 리터러시를 높이기 위해 프롬프트 작성 방법에 대한 이론과 실습을 병행하고 있다. 나름 프롬프트 작성에 대한 노하우가 쌓였다고 생각했는데, 실습을 하다보니 개선할만한 패턴이 발견됐다. 따라서 이번에 프롬프트 작성 방식에 대한 회고를 해보려고 한다. 본론 문제 인식 실습은 개인과 조별로 진행을 한다. 개인 실습의 예로는 모호한 지시를 4요소(역할, 맥락, 지시, 형식)을 포함해 개선해나가는 식이고, 조별 실습은 사내 회의 시나리오가 주어지고 안건으로 올릴 요약 대시보드 표를 만드는 식이다. 내가 실습을 할 때는 바로 요청사항을 지시하기보다, 아래 내용을 포함해 프롬프트 생성 자체를 지시한다. (메타 프롬프팅이라고 한다.) 프롬프트 엔지니어링 전문가라는 역할을 부여 간단한 요청사항 추가 더 필요한 정보는 질문해달라고 언급 이 방법이 내가 직접 작성하는 것보다 빠르고 생각치 못한 부분도 챙겨줘서 애용한다. 그런데 비슷한 실습을 반복하면서 이런 방식이 AI 활용 역량 향상에 도움이 될까 하는 의문이 들었다. 또한 조별 실습과 발표를 할 때에도 어떤 흐름으로 할 지 AI에게 물어보고, 응답을 조합해서 발표하다보니 어딘가 알맹이가 빠진 듯한 느낌을 받았다. 그 느낌은 다른 조의 발표를 들으면서 뚜렷해졌다. 어떤 문제를 해결하는 프롬프트를 작성하고 개선하는 실습이 있을 때, 문제를 해결하기 위한 방법을 조원들과 논의하고 직접 작성해서 응답을 받아본 다음, 아쉬운 점과 개선 방향을 논의해 다시 지시하는 것을 반복하는 흐름이었다. AI가 개입한 지점은 요청 사항대로 응답한 부분 뿐이었다. 문제 의식을 가지고 지시를 하고, 결과물에 대한 판단은 사람의 몫이었다. 알맹이의 정체는 ??이었다. 목표 재정의 및 프롬프트 작성 방식 비교 AI 활용 역량을 키우기 위해선 기존 방식을 벗어나야했다. 또한 확실한 인사이트를 얻기 위해 실습마다 개인적인 목표를 정의했다. 실습은 내 업무에 반복적으로 사용할 직무 프롬프트를 만드는 것이었다. ( 링크 ) 상황을 정해서 프롬프트를 만드는 실습이었는데, 여기에 개인적으로 달성할 목표를 추가했다. 실습 목표 AI가 프롬프트도 잘 만들어주는 시대에, 나 혼자서도 역할/맥락/지시/형식을 채울 수 있는 감을 키우기 AI가 만들어준 것과 내가 목적에 맞게 작성한 것의 결과 비교해보고 핵심 인사이트 얻기 뭐든 초반에 직접 생각해보지 않고 AI에게 통으로 맡기는 습관 회고해보기 따라서 직접 작성 / 메타 프롬프팅 방식 두 가지를 모두 사용했다. 4요소(역할, 맥락, 지시, 형식)을 개별적으로 작성 직접 작성: 4요소를 참고해 하나의 프롬프트로 작성 (완벽주의가 있으니, 최소 목표를 지정하라는 개인적인 맥락 추가) 메타 프롬프팅: 4요소를 붙여넣고 이런 상황에 사용할 직무 프롬프트를 만들어 달라고 요청 응답을 비교했을 때 아래 기준을 만족하며, 두 방식 모두 작업을 이어나가는데에는 충분했다. 프롬프트 평가 기준 정확성: 원본(문서·이미지·검색 결과)과 맞는지 대조했는지 형식: 원하는 형식(표·길이 등)으로 잘 나왔는지 활용도: 즉시 쓸 수 있는지, 손이 얼마나 더 가는지 안전: 개인정보나 회사 기밀이 담긴 파일은 올리지 않았는지 그러나 직접 작성한 프롬프트에 ‘완벽주의가 있는 특성’을 추가한 차이로, 내 단점을 보완하고 시간 내 작업하는 데에 더 유리할 것이라 판단했다. 그 맥락 또한 메타 프롬프팅에 추가했으면 응답 결과물 차이가 거의 없었을 수 있다는 것도 포인트였다. 사람 손을 많이 탈 수록 결과물이 좋을 거라 생각한 부분도 빗겨나갔다. 두 방식의 결과에 대한 차이는 근소했지만, 목표를 정의하고 실험해보고 고민하는 과정을 통해 생각을 이어나간 과정은 유의미했다. 결론 작업에 대한 맥락만 명확하다면 직접 작성과 메타 프롬프팅 모두 기준에 충족되는 응답을 했다. 그렇다면 결과물의 질을 높이기 위

Falsifier first: if you can find a fourth production call site that builds a Transformation and reconstructs its target field differently from the three I'm about to describe, this post is wrong about "all of them." I counted by grepping for the one function that computes a transformation's identity and checking every call site by hand. Three. If there's a fourth, the bug I'm describing isn't fully fixed. Here's the shape of it. Engine::plan_shape has a doc comment that says, more or less, "this isn't a second place where transformation identity gets defined, because it's the same code as the one true place." That claim was false, and it had been false since the field it's talking about was added. The actual second place was Engine::rehydrate_committed . Its job is to rebuild a Transformation from the journal when a fresh CLI process needs to undo something a previous process committed. Every gx undo call from a cold process goes through it. And for one field, target , it wasn't rebuilding anything. It wrote a hardcoded placeholder. Nobody noticed, because nothing disagreed with the placeholder. Every adapter shipping at the time also produced the placeholder for that field, by omission rather than by design, so the two sides matched by coincidence. A missing value that's always missing on both sides of a comparison is invisible. cargo check doesn't catch it because the type is Option<T> and None is a completely legal value of that type. Nothing was wrong, until something else became right. What made it right was landing the two adapters that finally do predict target , fs and git, so their production plan() calls started filling in the real value instead of leaving it empty. The moment that shipped, cold-process undo broke for every fs or git transformation: gx_code=INTERNAL detail="TransformationId(...) is Committed, and 43 §3 has no `rehydrate: the rebuilt transformation names another id, so the intent supplied is not the one this transformation was planned from`

Scaling Kafka Consumers in Spring Boot: How We Cut Lag and Saved Latency When scaling high-throughput event-driven microservices in fintech, default Spring Kafka consumer configurations often run into throughput limits under peak loads. Here is the exact production setup we engineered to resolve consumer lag and reduce API processing latency by 35%. 1. Concurrency Tuning Over Single-Threaded Listeners By default, @KafkaListener operates with concurrency = 1. When a partition receives high message volume, processing gets backlogged. @Configuration @EnableKafka public class KafkaConsumerConfig { @Bean public ConcurrentKafkaListenerContainerFactory < String , PaymentEvent > kafkaListenerContainerFactory ( ConsumerFactory < String , PaymentEvent > consumerFactory ) { ConcurrentKafkaListenerContainerFactory < String , PaymentEvent > factory = new ConcurrentKafkaListenerContainerFactory <>(); factory . setConsumerFactory ( consumerFactory ); factory . setConcurrency ( 6 ); // Matches number of partition splits factory . getContainerProperties (). setAckMode ( ContainerProperties . AckMode . MANUAL_IMMEDIATE ); return factory ; } } 2. Explicit Batch Processing and Idempotency Instead of committing offset per message, processing batches with manual acknowledgments ensures atomic handling: @Service public class PaymentEventConsumer { @KafkaListener ( topics = "payment.settlement.v1" , containerFactory = "kafkaListenerContainerFactory" ) public void consume ( ConsumerRecord < String , PaymentEvent > record , Acknowledgment ack ) { try { processPayment ( record . value ()); ack . acknowledge (); } catch ( Exception ex ) { log . error ( "Failed processing record key: {}" , record . key (), ex ); // Route to Dead Letter Queue (DLQ) handleDeadLetter ( record ); ack . acknowledge (); } } } 3. Key Takeaway Scaling Kafka consumer pipelines requires matching topic partition count with container concurrency, tuning database connection pools and implementing dead letter queues for fail

DoorDash has moved engineering agent workloads from developer laptops to its Flux cloud platform. The platform automated 130,000 engineering tasks in one month and supports more than 25,000 automated code reviews weekly. Flux uses isolated Firecracker microVMs, an MCP gateway, reusable playbooks, and multiple invocation surfaces to run agent workflows with scoped access and centralized auditing. By Leela Kumili

I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On paper, it looked like it should scale fine across the cluster. In practice, the job would race through most of its tasks and then stall. The Spark UI told the real story: almost every task finished in a few minutes, but one or two tasks ran for over 40 minutes while their executors sat at 80–90% CPU, and the rest of the cluster sat mostly idle waiting for them to finish. This post walks through what data skew actually is, how I confirmed it was the cause, and the fix that brought the job back under control. What data skew actually is Data skew happens when one or a few keys hold a disproportionate share of the data. When Spark distributes work across partitions — usually via hash partitioning on a join or group-by key — all the rows for a given key land in the same partition. If one key has millions of rows and most others have a few thousand, that one partition (and the single task processing it) ends up doing far more work than every other partition combined. The result is a job where 95% of tasks look completely healthy, and the remaining 5% become the actual bottleneck. Total job time is dictated by the slowest task, not the average one — so a single skewed key can dominate your entire runtime even if it represents a tiny fraction of your total row count. How I confirmed it The Spark UI's stage view was the first clue: a chart of task durations with almost every bar clustered together, and one or two bars stretching far beyond the rest. That pattern — uniform short tasks plus one long outlier — is close to a signature for skew. To confirm which key was responsible, I ran a simple aggregation on the join key before doing anything else: from pyspark.sql import functions as F df . groupBy ( " customer_id " ) \ . count () \ . orderBy ( F . desc ( " count " )) \ . show ( 20 ) The output made it obvious: a small number of cu

Text-to-Speech Is Not a speak() Call The challenge 🧪 If you have ever assumed Text-to-Speech on Android is straightforward, this article is for you. But first, let us test your skills. Think you can make this speak on Android? 🙏 अव्यक्तोऽयमचिन्त्योऽयमविकार्योऽयमुच्यते । "Invisible, beyond thought, unchanging." — Krishna describing the nature of the self. Today is World Sanskrit Day, so the timing is fitting. 🕉️ Try playing it on the plain TextToSpeech API that Google provides — but specifically with a Sanskrit voice. Build a minimal Android app, initialize the TTS engine, set the language to Sanskrit, and call speak() on this string. Chances are it will not speak anything. Not even a single letter would be uttered. 🔇 That is the moment when a developer realizes that TTS is not a simple API call. The twist 🔄 Use a Marathi or Hindi voice instead. Same engine. Same text. Same API call. It plays perfectly. 🗣️ Same engine. Same verse. Different voice. Completely different result. The boundary between "speakable" and "not speakable" is not at the engine level. It is at the voice level within the engine. The Sanskrit voice within Google's TTS engine cannot handle this verse. But the Marathi voice — which shares much of the same Devanagari character set — handles it without issue. This changes how you think about TTS integration. What happened in production 🏭 This is not a theoretical exercise. This is what we actually hit. In Bhagavad Gita, the player screen uses TTS to read verses aloud. The experience is designed to feel like playing a media file: continuous, flowing, uninterrupted. But certain words — especially compound words and special conjunct characters — were being silently skipped. Not errored. Not logged. Just... silent. The engine would skip the entire word if it couldn't speak something in it. So a verse that should take 15 seconds to read would finish in 8. The user would hear a flowing recitation with missing pieces and never know what was lost. 😶 The worst

First of all, I warmly welcome everyone out there in the DEV Community. [Completely open for discussion — drop your thoughts below.] From my perspective, it feels like everyone is racing towards AI/ML. The moment someone says they want to become an AI/ML Engineer, the conversation immediately shifts towards: Python → ML → Deep Learning → LLMs → Latest AI Tools And thinking about it, well, it’s quite understandable too. AI is one of the most exciting areas in technology right now. BUT, I have a question… Why are we starting to treat AI/ML Engineering as something completely different from Software Engineering? I often see people following an extremely narrow path towards AI/ML while completely skipping the fundamentals of Software Engineering. Backend development gets ignored. Databases, networking, operating systems, system design — all of them get ignored. And afterwards: APIs, deployment, testing, distributed systems… All of these seem quite trivial, right? Because the end goal is simply to create or automate something with AI. But it’s quite clear to me that AI can’t possibly live by itself. For any AI model to thrive, we need data. That data needs storage and pipelines. A model needs an application around it. That application needs APIs. Those APIs need backend infrastructure. And now we have an actual system. That system needs to be monitored for bugs, optimized for CPU and memory efficiency, refactored when necessary, maintained over time, and tested against new use cases. So thinking about all of this: How does one even fathom becoming an AI/ML “Engineer” without understanding what they are actually engineering into and working on? Maybe AI/ML Engineering and Software Engineering aren’t two completely different entities. Maybe they are different components of the same system. Now, I’m not saying: “You should become an expert in everything.” Specialization is indeed important. But specialization doesn’t necessarily mean abandoning the fundamentals that the spe

How to combine exponential backoff, circuit breakers, and graceful fallbacks for production-grade agentic workflows. The Bottleneck in Production AI agents are only as reliable as the tools they invoke. When an LLM decides to search the web, scrape a URL, or fetch database records, it depends entirely on network stability. In production, external APIs fail constantly. A sudden surge causes 429 rate limits, a third-party microservice throws a 504 timeout, or a target endpoint goes down entirely. The naive approach—executing raw tool calls directly inside the agent loop—is a ticking time bomb: # The Naive Anti-Pattern: Fragile Tool Execution def execute_agent_tool ( tool_name : str , payload : dict ): # One 500 error here kills the entire multi-step reasoning chain response = requests . post ( f " https://api.service.internal/ { tool_name } " , json = payload ) return response . json () When this call breaks, the unhandled exception crashes the runtime. You lose the entire reasoning graph, waste LLM tokens, and degrade the user experience. The System Architecture: Layered Tool Defense To keep multi-step agents alive, you need a defensive execution pipeline wrapped around every tool. Instead of allowing errors to bubble up and kill the agent, we handle failures across three distinct layers: Exponential Backoff : Mitigate transient network glitches and minor rate spikes by retrying with increasing delays. Circuit Breaker : Detect persistent downtime. If an API fails three times consecutively, trip the breaker to stop sending doomed requests. Graceful Fallbacks & Partial Degradation : When a primary service is down, route the query to a replica, cached store, or lightweight fallback (e.g., cached search index instead of a live browser scrape). [ Agent Core ] │ ▼ ┌───────────────────────────────┐ │ Circuit Breaker Check │ │ (Is Primary Service Up?) │ └──────────────┬────────────────┘ OPEN │ CLOSED (Healthy) ┌───────┴────────┐ ▼ ▼ ┌─────────────┐ ┌─────────────────────────