标签：#Engineering

Hello, DEV Community! 👋 Hi everyone! My name is Sulemana Abdallah , and I'm excited to be part of the DEV Community. I'm passionate about software development and enjoy building modern, responsive web applications using: HTML CSS JavaScript TypeScript React Python My goal is to become a skilled Full-Stack Developer and Software Engineer while continuously learning and building real-world projects. I joined DEV to: Learn from experienced developers. Share my projects and progress. Write about what I learn. Connect with developers from around the world. I'm looking forward to growing with this amazing community. Thanks for reading! 🚀

In Part 1 , we learned that building great AI applications isn't just about writing better prompts. It's about providing the right context . But that naturally leads to another question: How much context can an AI actually understand? If you've used ChatGPT, Claude, Gemini, Cursor or any AI coding assistant for a while, you've probably experienced something like this. 🤔 "Didn't I Already Tell You That?" Imagine you're debugging a production issue with an AI assistant. You start by explaining the architecture. Then you share the API flow. Then database schema. Then logs. Then stack traces. After 30 minutes of conversation, you ask: "So what's causing the bug?" Instead of giving the answer, the AI responds: "Could you share your database schema?" You stare at the screen. "I already did..." Sometimes it even forgets details from earlier in the same conversation. Naturally, people assume: The AI has bad memory. The model is unreliable. The conversation is broken. In reality, something completely different is happening. You're running into one of the most important concepts in modern AI systems: The Context Window. Understanding this concept changes how you interact with AI—and more importantly, how you build AI-powered applications. 🧠 Before We Talk About Context Windows… We first need to understand something much smaller. Tokens. Almost every AI provider mentions them. Pricing is based on them. Context windows are measured using them. Yet many developers still think: 1 token = 1 word That isn't true. 🔤 What Exactly Is a Token? A token is the basic unit of text that an AI model processes. Humans naturally read text as: Characters Words Sentences Large Language Models don't. Before text reaches the model, it's converted into smaller pieces called tokens by a tokenizer. The model never sees your original sentence. It only sees a sequence of tokens. Think of a tokenizer as a translator between humans and AI. You write English ↓ Tokenizer ↓ Sequence of Tokens ↓ LLM The toke

This is my first post here, so I'm going to use it as an introduction. I'm Usman, a software + data engineer who primarily works with data pipelines, backend systems. Not a huge fan of frontend development though. Although I do what I can, projects honestly feel incomplete without them, because at the end of the day you do have to showcase a working end to end system when you build something. I'm here after dozens of incomplete personal projects, and projects that never even got past the design phase, you know the drill. Procrastination and imposter syndrome kept stopping me from taking the next step, but I'm here now, gotta keep myself in check fr. I was scrolling through LinkedIn the past few days, and oh my god, the amount of AI-related brain rot there. Every single post written by AI, telling you how to use AI and how not to use AI. I mean I get it, yeah, the paradigm is shifting and AI is essential to development, but where are your personal anecdotes, stuff you solved, stuff you learned, the challenges you faced, how you overcame them. You know what maybe it's my fault, it's my algorithm after all. Anyway, here I am, looking to interact with like-minded engineers and learn from them. I'm also going to post regularly about my progress and what I am building, even though I have quite a bit of experience, and have built and contributed to large-scale production systems and pipelines, I'm going to start with something small, so I can stay consistent and keep myself in check. Software engineering fascinates me a lot, and there are so many domains that I wish to explore and have explored like game development, data engineering, web/app development. My significant other is graduating in a few days, and I'm thinking of making a small game for her, alongside which I'll be working on a small sales lead enrichment pipeline. Hoping to showcase my work and document it publicly, and hoping to get to know and learn from you all! Also, I'd love to know your thoughts on the am

Introduction Python's testing tools are lightweight enough that it's easy to write a lot of tests without writing good ones. A suite that mocks every collaborator, duplicates the same assertion ten times with different inputs pasted in by hand, or chases a coverage number will pass in CI and still miss real bugs. pytest gives you fixtures, parametrize , and monkeypatch — the tools that make it just as easy to write the right tests as the wrong ones. This post covers how to use them well. Test at the Right Level: the Pyramid Not every test should look the same. The test pyramid is a rough guide to where your effort should go: Unit tests — the bulk of the suite. Pure functions and classes, no I/O, no real database. Milliseconds each. Integration tests — fewer of these. Verify the seams : does your ORM query actually produce correct SQL against a real database, does your HTTP client actually parse a real response. End-to-end tests — a handful. Cover the critical flows through the whole stack, accepting they're slower and more brittle. # Unit — pure logic, no database, no framework def test_applies_ten_percent_discount_for_orders_over_100 (): calculator = DiscountCalculator () total = calculator . apply ( order_total = 150.0 ) assert total == pytest . approx ( 135.0 ) # Integration — the seam that matters: our query against a real database import pytest @pytest.fixture def db_session ( postgres_container ): # real Postgres in a test container, not mocked with postgres_container . session () as session : yield session def test_finds_orders_placed_in_the_last_week ( db_session ): db_session . add ( Order ( id = " ord-1 " , placed_at = datetime . now ( UTC ))) db_session . commit () recent = order_repository . find_recent ( db_session , within = timedelta ( days = 7 )) assert len ( recent ) == 1 A unit suite that never touches a database runs in seconds and catches most logic bugs. A handful of integration tests catch what only shows up at the boundary — the query that's s

An astrolabe doesn’t map every star. It gives you a way to find your position relative to the ones that hold still. That’s the instrument I reach for when someone asks which AI tool they should be using. The honest answer is that the tools will be different in six months. The layers won’t. I spent a week trying to make sense of a handful of names that kept showing up in the same conversations. Tessl . Goose . Archestra . Kestra . Modelplane . RAG , MCP , half a dozen others orbiting nearby. Each one has its own pitch, its own funding round, its own reason it’s the thing you should adopt next. Taken together they read like noise. Taken apart, they sit on different floors of the same building. The agent loop again, the one I keep coming back to. Once you place each tool on a floor, the noise turns into a map. Tessl sits left of the loop , at the intent layer. Turn a spec into something an agent runs against directly. This is the one tool on the list that pushes back instead of going along with it. A well-formed spec is not the same thing as a team that agrees on what the spec means. The Agora produces the second thing as a byproduct of producing the first. Tessl produces the first and assumes the second follows. It doesn’t, automatically. That’s the whole argument. RAG and MCP are plumbing. Protocol, not position. They carry context into the loop and don’t take a side in any argument about who should be in the room when the spec gets written. They’re also the one floor with an actual standard. MCP, A2A , ACP , all under Linux Foundation governance now, joint working groups, cross-protocol commitments. Passing data between systems is a solved problem with decades of precedent behind it, so it standardized almost on contact. Nothing else on this floor plan has that. Governance, orchestration, the harness, the spec layer: every vendor is still building its own version and calling it the obvious one. The standard showed up first at the floor that mattered least to this ar

Cycle recently introduced a separate EU-based control plane, allowing European customers to keep platform management data and telemetry within Europe. The new offering is designed to improve compliance, operational isolation, and responsiveness for European organizations. By Renato Losio

Cloudflare details Town Lake, an internal unified data platform, and Skipper, an AI analytics agent unifying access to operational, billing, security, and business data. The platform processed ~91K billing queries, with billing forming majority usage. Built on a lakehouse architecture using Trino, Iceberg, R2, and DataHub, it enables governed cross-system analytics and natural language access. By Leela Kumili

Here are five architectural lessons we've learned from designing software for modern retailers.* Designing ERP Software for Retail: Five Lessons Every Software Engineer Should Know When people hear the word ERP , they often think of accounting software, dashboards, or inventory management. As software engineers, we see something different. We see distributed systems. Complex business workflows. Real-time data synchronization. Concurrent transactions. Event-driven architecture. And perhaps the biggest challenge of all—representing how real businesses actually operate. At RetailWings , we've learned that building an ERP for retail isn't simply a software engineering challenge. It's a business engineering challenge. Here are five lessons every engineer should understand before designing an ERP platform for modern retail. 1. Retail Doesn't Run in Modules—It Runs as One Business One of the biggest architectural mistakes in business software is treating departments as isolated applications. Many systems separate: Sales Inventory Finance Procurement HR But retailers don't experience their businesses that way. One sale immediately affects inventory. Inventory influences procurement. Procurement impacts finance. Finance drives reporting. Everything is connected. A well-designed ERP should reflect these relationships rather than forcing departments into disconnected silos. 2. Inventory Is More Than a Database Table To many engineers, inventory may appear to be a simple CRUD problem. Create. Read. Update. Delete. Retail quickly proves otherwise. Inventory changes through: Sales Returns Transfers Damages Procurement Stock adjustments Warehouse movements Manual reconciliations Every movement has financial implications. Every movement must be traceable. Designing inventory requires thinking in terms of events, not just records. 3. Real-Time Data Changes Everything Retail managers don't want yesterday's reports. They want answers now. How much stock is left? Which branch is sellin

There is a fascinating, quiet tension happening in the software engineering community right now. If you listen closely to late-night developer chats, team syncs, or tech forums, you will notice that our industry has rapidly split into two distinct schools of thought regarding the rise of AI coding tools like Cursor, Claude Code, and Copilot. On one side, you have the Traditional Developers. They argue that software engineering is a disciplined art form that cannot be replaced by text prompts. To them, unchecked AI coding is a recipe for buggy, unreadable spaghetti code, creating a technical debt nightmare for the future. On the other side, you have the Vibe Coders. This is a fast-moving generation of builders, both technical and non-technical, who believe in shipping fast, prompting quickly, and adjusting on the fly. They do not see a need to obsess over syntax when the AI can translate their intent into a working application in minutes. The reality is that both sides are entirely right. If we stop arguing over who is ruling the current meta and actually look at the core truths each camp holds, we can see exactly where the future of software development is heading. 1. The Traditional Developer is Right: Guardrails Matter The traditional development camp is fundamentally right about structure. Building a beautifully designed UI that works on a surface level is vastly different from building an enterprise-ready, scalable architecture. When you prompt an AI to build a feature, its primary objective is to satisfy the literal words in your core prompt. This is the "as long as it works" mentality. Unless you are practicing strict, spec-driven development and explicitly dictating your architectural doctrines, security protocols, and API patterns, the AI will make assumptions for you. Historically, those assumptions are optimized for speed and not long-term stability. Without deep technical oversight to catch anti-patterns, edge cases, and hidden security flaws, fast-shippe

Hardwood, the project Gunnar Morling kick-started handling of Parquet files in Java, reached version 1. Its multi-threaded approach and zero mandatory external dependencies promise a simpler, more efficient alternative to the Apache Parquet Java implementation. For now, the library supports just reading; writing support is expected in the upcoming versions. By Olimpiu Pop

Introduction As data volumes continue to grow, running aggregation queries directly on raw datasets becomes increasingly expensive. Business dashboards, analytics platforms, and reporting systems often execute the same calculations repeatedly—such as total sales, daily active users, page views, or revenue trends. While ClickHouse® is designed to process analytical workloads at remarkable speed, repeatedly scanning billions of records still consumes valuable CPU, memory, and storage resources. This is where AggregatingMergeTree proves its value. Rather than calculating aggregates every time a query is executed, AggregatingMergeTree stores intermediate aggregation states that are merged automatically in the background. This approach allows analytical queries to read compact, pre-aggregated datasets, resulting in dramatically faster response times and reduced infrastructure costs. In this guide, you'll learn how AggregatingMergeTree works, why aggregate states matter, how to build an automated aggregation pipeline using Materialized Views, and when this engine is the right choice for your ClickHouse® workloads. What is AggregatingMergeTree? AggregatingMergeTree is a specialized ClickHouse® table engine designed to store aggregate function states instead of raw records. Unlike the standard MergeTree engine, which stores every inserted row, AggregatingMergeTree keeps partially aggregated values that ClickHouse combines during background merge operations. This significantly reduces the amount of data that must be processed when generating analytical reports. Because much of the computational work happens during data ingestion, dashboards and reporting applications can retrieve summarized information much more efficiently. Typical scenarios include: Sales reporting Website traffic analytics Financial summaries IoT sensor monitoring Business KPI dashboards Application observability metrics Why Use AggregatingMergeTree? Imagine an online marketplace processing millions of tr

The speakers discuss Agent RFT, OpenAI’s platform for fine-tuning reasoning models via real-time tool interactions and custom reward signals. They explain how reinforcement learning solves complex credit assignment challenges within the context window. They share enterprise success stories, showing how Agent RFT eliminates long-tail token loops and drives extreme efficiency. By Wenjie Zi, Will Hang

One question I get surprisingly often is: "How do you manage thousands of AI prompts without losing track of them?" The answer is simple. I don't treat prompts as conversations. I treat them as reusable software assets. Over the years, I've created prompt libraries across multiple AI projects, books, research initiatives, and client work. That means managing well over 10,000 prompts covering everything from Python development and AI agents to content generation and workflow automation. If you're still storing prompts in random ChatGPT conversations, you're making life much harder than it needs to be. Here's the system that works for me. Stop Thinking of Prompts as Temporary Most people write a prompt, get an answer, and move on. That's fine for casual use. But builders rarely solve the same problem only once. If you find yourself writing: API documentation SQL queries FastAPI endpoints Docker configurations Code reviews Git commit messages ...you're probably solving recurring problems. Recurring problems deserve reusable prompts. My Folder Structure Instead of organizing prompts by AI tool, I organize them by purpose. For example: AI-Prompts/ │ ├── Python/ │ ├── FastAPI │ ├── Django │ ├── Flask │ └── Automation │ ├── JavaScript/ │ ├── React │ ├── Node.js │ └── TypeScript │ ├── DevOps/ │ ├── Docker │ ├── Kubernetes │ └── GitHub Actions │ ├── AI/ │ ├── RAG │ ├── Agents │ ├── MCP │ └── Prompt Engineering │ └── Documentation/ This mirrors how software projects are organized. Finding a prompt takes seconds. Every Prompt Has Metadata A prompt isn't just text. It's documentation. Each prompt in my library includes: Category: Purpose: Model: Input: Expected Output: Version: Last Updated: For example: Category: FastAPI Purpose: Generate CRUD endpoints Model: GPT-4o Expected Output: Production-ready FastAPI code Six months later, I know exactly why that prompt exists. I Version My Prompts Developers version code. Why not prompts? For example: FastAPI_CRUD_v1.md FastAPI_CRUD_v

For decades, developers were judged by one thing: How much code they could write. The best programmers wrote faster. Debugged faster. Built faster. That era is ending. The next generation of developers won't spend most of their time writing code. They'll spend it directing AI. Welcome to the age of the AI Orchestrator. The Evolution of Software Development Software development has always evolved. First, developers wrote machine code. Then came assembly. Then high-level languages. Then frameworks. Then cloud platforms. Then DevOps. Each evolution removed repetitive work and let developers focus on bigger problems. AI is simply the next step. But this time, it isn't replacing a tool. It's becoming a teammate. Coding Is Becoming a Smaller Part of the Job Building software isn't just writing code. A typical project includes: Understanding requirements Researching documentation Designing architecture Writing code Reviewing code Debugging Testing Writing documentation Deploying applications Monitoring production Fixing incidents Only one of those is coding. Everything else is coordination and decision-making. That's where AI is changing the game. From Programmer to Orchestrator Think about how modern teams work. A tech lead rarely writes every line of code. Instead, they: Assign work. Review solutions. Provide feedback. Make architectural decisions. Remove blockers. Developers are beginning to work with AI in much the same way. Instead of writing every function, they'll: Define the goal. Provide the right context. Choose the right tools. Review AI-generated code. Run tests. Improve weak areas. Approve the final result. The value shifts from typing code to guiding its creation. What Does an AI Orchestrator Do? An AI orchestrator doesn't ask one question and accept one answer. They manage a workflow. For example: Break a large project into smaller tasks. Give each AI the context it needs. Decide when to retrieve documentation. Decide when to search the codebase. Ask AI to g

You merge the PR. The build passes. The code does what you expected it to do. You move on. That is review for most engineers. A final read. A feeling that things looked right before the branch closed. I did it the same way for years. Three phases had already run before this one. Think had scoped the work, Plan had written the requirements, Build had shipped a diff that matched the plan exactly. I trusted that the chain held. I had never actually checked. Then I ran the Review phase, and checking turned out to mean something specific: not does this work, but does this requirement hold up, and what is my evidence. I went in expecting to approve it or send it back. The phase gave me three answers instead: covered, partial, missing. I found out what they meant one requirement at a time, starting with the one I almost got wrong. I had been giving impressions, not verdicts The notification scheduler used a queue to manage dispatch. Every call to the external provider went through it. The provider was never exposed directly. The requirement said the provider must be notified. It was notified, exactly the way I had pictured it. I almost called it covered and moved to the next line. The Review phase stopped me there. But the requirement said must be notified , not how. The queue had introduced a call order and a timing the requirement never anticipated. Nothing was broken. Something had changed shape, quietly, and nobody had written that shape down. I sat with that for longer than I expected to. Not because the code was wrong. Because I could not immediately tell you whether the change mattered. The same pass gave the shim from Plan a different verdict on the same page: covered. Mapped to the requirement it existed to satisfy, no gap between what was promised and what was in the diff. One requirement held exactly the shape it was given. The other had quietly grown a new one. Same review. Same pass. Two verdicts. Partial is not a softer word for broken. It is the verdict for

A company shifted from project- to product-thinking after their platform outgrew single-team use. The limitations that they felt with their platform were one-off deliveries, lack of product vision, and weak feedback loops. They have moved toward a self-service, API-driven, multi-tenant infrastructure with clearer ownership and better abstractions. By Ben Linders

I used to waste hours tweaking and re-tweaking my AI model prompts. It was like trying to find a needle in a haystack—I'd make a change, run the code, wait for the results, and then... nothing. The output would be inconsistent, unhelpful, or just plain wrong. I'd try again with tiny modifications, rinse and repeat, until I was about to pull my hair out. It wasn't until I stumbled upon the concept of reusable prompt templates that everything changed. It was like a switch had flipped—my code started producing consistent results, and I finally understood why. No more guesswork, no more frustration. Just good old-fashioned productivity. A simple shift from writing one-off prompt strings to using reusable templates is the key to reducing prompt overhead, increasing consistency, and getting back to doing what we love—building amazing, AI-driven applications. From Chaos to Control: A Simple Example Let's make this tangible. Imagine you're building a feature to generate a short story, but for different characters. Before: The Inconsistent, One-Off Way Without a template, you'd likely write a new prompt each time, introducing small, unintentional differences that lead to wildly different results. Two separate prompts = inconsistent, unpredictable output prompt_for_alex = "Write a short story about a character named Alex who is trying to get to work on time, but keeps getting delayed in a busy city." prompt_for_jordan = "Generate a story about someone named Jordan. They're late for work and stuck in traffic in a big city." See the problem? The tone, wording, and details are different. You have no control over the consistency of the output. After: The Clean, Templated Way Now, let's use a single template. We define the core structure once and simply pass in the parts that change. Now, let's use a single template. We define the core structure once and simply pass in the parts that change. One template = consistent, predictable output story_template = "Write a short story about

The speakers discuss Netflix’s architecture for surviving extreme traffic spikes. They explain the mechanics of prioritized load shedding embedded in their Envoy sidecar proxy, allowing user-initiated requests to steal capacity from non-critical traffic. They share automated platform strategies for continuous chaos load testing, config generation, and retry storm mitigation. By Anirudh Mendiratta, Benjamin Fedorka

📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p

Pressing enter to accept model suggestions now takes less effort than scrolling past it. One keystroke, and the code is yours. Reading it, understanding it, deciding if it's actually right, that part hasn't gotten any faster. That gap, between how fast we can accept code and how fast we can actually understand it, is where things start to go wrong. The new shape of technical debt We used to know where technical debt came from. Tight deadline, cut corner, # TODO: comment that nobody ever revisits. Rushing was the cause, and we could at least point to it. Now you can build up the same kind of debt on a calm Tuesday afternoon, no deadline in sight, just six suggestions in a row accepted because they looked fine and the flow felt good. Nobody rushed you, and the code still ended up just as unexamined. Same debt, just a different excuse. "It works" is not the same as "I understand why it works" Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? — Brian Kernighan, 1974 Fifty years later, the gap got wider. Kernighan was talking about code you wrote. At least you understood it once. A suggestion that compiles, passes the linter, survives code review and even comes with passing tests can still be standing on a wrong assumption that nobody caught, because nobody was reading it as code. They were reading it as output, and output that makes sense tends to get approved. Compiling is a low bar. Passing tests is a slightly higher one, depending on whether you wrote the tests, or its suggestion shaped or created those too. If it's the second, it's like grading its homework with its own answers. None of it tells you the logic is sound, that the edge cases are covered, or that it does what you actually needed, something we already learned every time we trusted code we didn't write. Somehow it's easy to forget it the moment the code appears inline, in our own edito