标签：#dataengineering

CSV files are one of the most common formats for storing and exchanging data. Whether you’re working with logs, analytics data, application exports, or reports, there will likely come a time when you need to load CSV data into ClickHouse®. The good news is that ClickHouse® makes CSV ingestion straightforward and efficient. In this guide, you’ll learn how to create a table, prepare a CSV file, load CSV data into ClickHouse®, and verify that the data has been imported successfully. Why Use CSV Files with ClickHouse®? CSV (Comma-Separated Values) files are simple, portable, and supported by virtually every data platform. Common use cases include: Importing exported application data Loading historical datasets Migrating data from other databases Testing analytics workloads Sharing data between systems Because ClickHouse® is designed for high-performance analytics, it can efficiently process and query large CSV datasets once they are loaded into a table. Sample CSV File Let’s assume we have a file named employees.csv with the following contents: id,name,department,salary 1,Alice,Engineering,75000 2,Bob,Marketing,60000 3,Charlie,Finance,70000 This simple dataset will help demonstrate how to load CSV data into ClickHouse®. Step 1: Create a Table in ClickHouse® Before importing data, create a table that matches the structure of the CSV file. CREATE TABLE employees ( id UInt32, name String, department String, salary UInt32 ) ENGINE = MergeTree() ORDER BY id; This table contains four columns that correspond directly to the columns in our CSV file. Step 2: Load CSV Data into ClickHouse® There are several ways to import CSV data, but one of the most common methods is using the ClickHouse® client. Run the following command: clickhouse-client --query=" INSERT INTO employees FORMAT CSVWithNames" < employees.csv The CSVWithNames format tells ClickHouse® that the first row contains column headers. After executing the command, ClickHouse® will read the CSV file and insert the records

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action

When we started working on Krenalis , we spent a lot of time reviewing how customer data typically flows through a modern data stack. One pattern kept showing up often enough that we started questioning it. In many modern stacks, customer data already lands in a warehouse. Yet we often copy that same data into a CDP before we can start building customer profiles. During one of those discussions, someone asked a question that sounded almost naive: Why are we moving all this data in the first place? Nobody had a particularly strong answer ready. The answer was mostly: Because that's how CDPs work. We expected the question to have an obvious answer. It didn't. The warehouse is no longer just for analytics Over the last few years, the role of the data warehouse has changed significantly. Warehouses are no longer just analytical systems. They're increasingly becoming the place where organizations centralize the context used by applications, AI agents, copilots, and business processes. Customer data from systems like Shopify, Stripe, CRMs, support platforms, and internal applications often ends up there long before anyone starts thinking about segmentation or activation. In many organizations, the warehouse is already the place where teams answer questions about customers, revenue, retention, and product usage. That made us wonder: If the warehouse is already becoming the operational center of the data stack, why does customer identity usually live somewhere else? Consider a customer who buys through Shopify, pays through Stripe, opens support tickets in Zendesk, and uses the product under a different email address. In many organizations, all of those records already end up in the warehouse. Yet building a unified profile often requires exporting that same data into another platform before identity can be resolved. The cost of another copy To be clear, data duplication is not inherently bad. Most software systems rely on some form of replication, caching, or denormalizati

The Term Everyone Uses and Nobody Defines Your CTO came back from a conference and said the team needs to "go agentic." A vendor pitched you an "agentic data platform" last week. LinkedIn is full of posts about agentic workflows transforming everything from customer support to supply chain management. And yet, when you ask three people what "agentic" actually means for your data operations, you get four answers. This is not a vocabulary problem. It is a strategy problem. Organizations are making six-figure decisions about agentic AI without a shared definition of what they are buying, building, or hiring for. That gap between the buzzword and the architecture is where most projects fail -- not because the technology does not work, but because nobody agreed on what it was supposed to do. This guide is a practitioner's attempt to close that gap. No vendor pitch, no hand-waving. Just a clear definition, a real example, and a framework for deciding whether agentic workflow consulting is something your team actually needs. What "Agentic" Actually Means (In Plain Language) Traditional data pipelines are deterministic. You define steps, connect them in order, and run them. Step A feeds step B, which feeds step C. If the input changes shape, the pipeline breaks and a human fixes it. The pipeline does not adapt, reason, or make decisions -- it executes. Robotic process automation (RPA) is slightly smarter but still scripted. It records human actions and replays them. Click here, type there, move this file. When the UI changes or an edge case appears, the bot breaks the same way a pipeline breaks: it stops and waits for a human. Agentic workflows are fundamentally different. An agentic system has components that can reason about their task, make decisions based on context, and take actions without a pre-scripted path for every scenario. Instead of "if X then Y," an agentic node can evaluate ambiguous input, choose between approaches, validate its own output, and route work to

İlk yazıda Event Driven Architecture’ın temel kavramlarını, Kafka üzerinde topic/channel tasarımını, event-command ayrımını, schema contract’ları ve producer-consumer ilişkisini ele aldık. Bu yazıda odağı bir adım ileri taşıyıp event’in platform içindeki yaşam döngüsüne bakacağız. Çünkü EDA tasarımında asıl zorluk yalnızca event üretmek değildir. Asıl mesele, üretilen event’in güvenilir, izlenebilir, tekrar işlenebilir, zenginleştirilebilir ve farklı tüketiciler tarafından kullanılabilir hale gelmesidir. Bu yazıda şu sorulara odaklanacağız: Ham event platforma geldiğinde ne olur? Event nasıl doğrulanır, zenginleştirilir ve tüketilebilir hale gelir? Raw, validated, enriched ve curated topic’ler nasıl konumlandırılmalıdır? Bu yapı modern lakehouse mimarilerindeki Medallion yaklaşımıyla nasıl ilişkilendirilebilir? DLQ ve alert topic’leri ne zaman devreye girer? Replay, idempotency, monitoring, security ve governance nasıl düşünülmelidir? Event Pipeline Nedir? EDA mimarilerinde özellikle data platform projelerinde event’ler genellikle bir yaşam döngüsünden geçer. Bu yaşam döngüsü şöyle modellenebilir: raw -> validated -> enriched -> curated | | v v dlq alert Bu yapı, veri akışının aşama aşama olgunlaşmasını sağlar. Raw topic kaynaktan gelen ham event’i taşır. Validated topic schema ve temel kalite kontrollerinden geçmiş event’leri içerir. Enriched topic event’in referans veriler veya başka veri kaynaklarıyla zenginleştirilmiş halidir. Curated topic ise tüketiciler için güvenilir, normalize edilmiş ve iş anlamı netleşmiş event’leri temsil eder. Event Pipeline ve Medallion Architecture İlişkisi Bu yapı, modern lakehouse mimarilerinde sık kullanılan Medallion yaklaşımıyla doğal bir benzerlik taşır. Lakehouse tarafında Bronze katmanı ham veriyi, Silver katmanı temizlenmiş ve zenginleştirilmiş veriyi, Gold katmanı ise iş tüketimine hazır veri ürünlerini temsil eder. Kafka üzerindeki raw, validated, enriched ve curated topic’leri de benzer bir olgunlaşma mantığını akan veri ü

If you've been calling yourself a "prompt engineer" for the past two years, it's time to update your vocabulary — and your mental model. In 2026, the real leverage when building LLM-powered systems isn't in crafting the perfect sentence. It's in context engineering : designing everything an LLM sees before it ever generates a response. Andrej Karpathy coined the term in mid-2025, and it's since taken over serious AI engineering discussions. This article breaks down what context engineering actually is, why it matters more than prompt writing, and gives you concrete techniques you can apply today. What Is Context Engineering? Context engineering is the discipline of systematically designing the information environment that surrounds a prompt. Where prompt engineering asks "what should I tell the model to do?", context engineering asks "what does the model need to know to do it well?" Think of it this way: a doctor doesn't just answer the question you ask on the spot. They look at your chart, your history, your vitals, and then respond. Context engineering is building that chart for your LLM. The context window is the LLM's working memory — everything it can "see" at once. In 2026, these windows are massive: Claude Opus 4.x : 200K tokens GPT-4o : 128K tokens Gemini 2.5 Flash : Up to 1M tokens But bigger isn't automatically better. More tokens = more cost, more latency, and a real risk of what researchers call the "lost-in-the-middle" problem — where models process information at the beginning and end of the context more reliably than content buried in the middle. Why This Matters for Data Engineers Data engineers are increasingly building pipelines that feed LLMs: RAG systems, AI copilots for data quality, agents that write and review SQL, tools that summarize data lineage. In every one of these systems, the quality of what lands in the context window directly determines output quality. A poorly designed context is like feeding a senior analyst a jumbled mess of raw l

T-SQL on Microsoft Fabric - Episode 1: Mastering Data Retrieval with SELECT, WHERE, and ORDER BY Learning Goals In this lesson, you will learn how to: Read data from tables using SELECT Filter rows with WHERE Sort query results with ORDER BY Get familiar with standard T-SQL syntax Practice directly in Microsoft Fabric Warehouse 1. Understanding Database and Schema In Fabric Warehouse, objects are commonly organized like this: Warehouse | |-- sales | |-- Customers | |-- Orders | |-- hr | |-- Employees | |-- finance |-- Transactions Schemas help you: Group related tables Manage permissions Organize large systems more effectively 2. Create a Schema Create a schema for the sales dataset: CREATE SCHEMA sales ; Check existing schemas: SELECT * FROM sys . schemas ; 3. Create Tables Create the Customers table: CREATE TABLE sales . Customers ( CustomerID INT , CustomerName VARCHAR ( 100 ), City VARCHAR ( 50 ), Country VARCHAR ( 50 ) ); Create the Orders table: CREATE TABLE sales . Orders ( OrderID INT , CustomerID INT , OrderDate DATE , Amount DECIMAL ( 10 , 2 ) ); 4. Insert Sample Data Customers INSERT INTO sales . Customers VALUES ( 1 , 'John Smith' , 'New York' , 'USA' ), ( 2 , 'Emma Brown' , 'Chicago' , 'USA' ), ( 3 , 'David Wilson' , 'London' , 'UK' ), ( 4 , 'Sophia Taylor' , 'Manchester' , 'UK' ), ( 5 , 'Michael Lee' , 'Singapore' , 'Singapore' ); Orders INSERT INTO sales . Orders VALUES ( 101 , 1 , '2026-01-10' , 1200 . 00 ), ( 102 , 1 , '2026-01-15' , 800 . 00 ), ( 103 , 2 , '2026-01-20' , 2500 . 00 ), ( 104 , 3 , '2026-02-01' , 500 . 00 ), ( 105 , 5 , '2026-02-05' , 3200 . 00 ); 5. SELECT Get all columns: SELECT * FROM sales . Customers ; Get specific columns: SELECT CustomerName , Country FROM sales . Customers ; 6. Alias Rename columns in the output: SELECT CustomerName AS Customer , Country AS Nation FROM sales . Customers ; 7. WHERE Filter rows using conditions. Customers in the USA: SELECT * FROM sales . Customers WHERE Country = 'USA' ; Orders greater than 100

Switching focus from Frontend development to Data Engineering means shifting from building user interfaces to architecting robust data pipelines. It’s a completely different mindset, and the learning curve is exciting! As I dive deeper into the world of Data, these are the 4 essential tools and concepts that have become the absolute backbone of my daily learning roadmap: 1️⃣ Python (The Swiss Army Knife): Coming from JavaScript/TypeScript, picking up Python has been a breath of fresh air. From writing custom ETL scripts to data manipulation with Pandas, it's the ultimate language for data manipulation. 2️⃣ Advanced SQL (The Core): It's not just about simple SELECT statements anymore. Mastering Window Functions, CTEs (Common Table Expressions), and query optimization is where the real magic happens when interacting with Data Warehouses. 3️⃣ ETL/ELT Pipelines: Understanding how to efficiently Extract, Transform, and Load data without breaking downstream analytics. Moving from UI state management to Data state management is a game-changer. 4️⃣ Cloud Ecosystems & Modern Stack: Exploring how data flows through modern cloud environments and learning how big data tools manage scale. The transition requires patience, but applying my previous engineering background to these new tools makes the journey incredibly rewarding. 💡 To the Data Engineers in my network: What is the one tool or concept you believe is a "must-have" for someone transitioning into the field today? Drop your advice below!

Introduction In contemporary research, the volume of biological data deposited in public repositories is growing exponentially. The Gene Expression Omnibus (GEO), NCBI Gene, PubMed, and UniProt accumulate thousands of new records daily, including sequences, expression profiles, scientific articles, and functional annotations. On the one hand, this scenario represents a unique opportunity for biomedical research. On the other hand, the diversity of data formats, access protocols, and metadata models creates a significant barrier: each source requires a specific collector, distinct rate-limiting strategies, and its own validation logic. Above all, the lack of standardization in data storage compromises the reproducibility of scientific studies. The need for integrated tools capable of unifying data extraction, curation, and persistence has been widely discussed. In practice, ad hoc solutions such as isolated scripts for individual repositories generate redundant work and make maintenance difficult. First and foremost, it is necessary to establish an architecture that treats data collection as a service rather than a collection of scattered artifacts. This work presents Project 1 of the Integrated Bioinformatics Platform: a containerized Biomedical Data Collector coupled with a Data Lake. Its objective is to provide a REST API capable of triggering asynchronous data collections from the four aforementioned sources, storing immutable raw data in MinIO, and persisting metadata in PostgreSQL, all while ensuring traceability and resilience. Development The system architecture is divided into three main layers. The first is the API and orchestration layer , implemented using FastAPI. Its five endpoints — POST /collections , GET /collections , GET /collections/{id} , GET /collections/{id}/download/{dataset_id} , and GET /health — expose a clean interface for initiating and monitoring collection processes. The second layer is the collector engine , composed of abstract classe

When I first connected to a precious metals WebSocket API, I expected to get a clean stream of prices. What I actually got was a firehose of mixed ticks—gold, silver, platinum—all arriving through the same callback. If you’ve ever tried to build a trading bot or a custom chart, you know this is a recipe for disaster. In this post, I’ll share how I solved the problem with a few lines of Python and a clear mapping strategy. The scenario: You have one WebSocket URL that pushes quotes for multiple metals. You need to separate them so you can update different UI components, run independent strategies, or store them in distinct database tables. The data pain point: every message uses the same JSON structure, and the only differentiator is a field like symbol . If you don’t act on it immediately, everything gets mixed up. Identify Assets via the Symbol Field Start by checking the API docs for the field that carries the instrument code. Usually it’s symbol , but instrumentId or type are also used. Here’s a typical reference table: Field Description Example symbol Asset code XAUUSD, XAGUSD instrumentId Internal platform ID 1001, 1002 type Asset class gold, silver I turn this into a dictionary mapping each symbol to a human-readable category: asset_map = { " XAUUSD " : " gold " , " XAGUSD " : " silver " , " XPTUSD " : " platinum " } Buffer Messages by Type Because these streams are high-frequency, I avoid processing every tick individually. Instead, the WebSocket callback just updates an in-memory store that is already grouped by asset type: # Keep the hot path extremely light def on_message ( msg ): symbol = msg [ ' symbol ' ] price = msg [ ' price ' ] asset_type = asset_map . get ( symbol , " unknown " ) cache [ asset_type ][ symbol ] = price Then, a background timer fetches the latest prices from cache["gold"] and cache["silver"] separately and does the actual work—like computing indicators or rendering charts. The key benefit is complete isolation: your gold logic never t

Over the past decade, the core evolution of data engineering has been the deconstruction and reconstruction of traditional data warehouse architectures through the Modern Data Stack. We separated data ingestion from databases, forming the Data Ingestion layer, using tools like FiveTran, Airbyte, and Apache SeaTunnel to solve ELT / CDC / Reverse ETL problems; We separated compute from storage, forming cloud data warehouse and lakehouse systems such as Snowflake, Databricks, Iceberg, and Hive; We separated orchestration from scripts, leading to orchestration systems like Apache Airflow and Apache DolphinScheduler; SQL development, data modeling, lineage, data quality, BI, and AI analytics were further split into independent tools. This architecture was undoubtedly progress. It moved data engineering away from the primitive era of “a bunch of scripts + Crontab” toward cloud-native infrastructure, elastic computing, engineering governance, and open ecosystems. The greatest contribution of the Modern Data Stack was “decoupling,” and its biggest side effect was also “decoupling.” Tools became more powerful, but data engineers were forced to switch between more systems than ever before: datasources in one place, synchronization configs in another, DAGs somewhere else, logs elsewhere, SQL stored in Git, and Snowflake / Iceberg / cloud warehouse execution results living in yet another environment. As a result, many data engineers spend less time on data modeling, business understanding, metric definitions, architecture design, and cost optimization — and far more time configuring datasources, setting field mappings, dragging DAG nodes, modifying SQL, checking logs, and rerunning tasks. This is the hidden pain created by the Modern Data Stack: data engineers became trapped inside tools. The emergence of engineering-focused AI systems like Codex and Claude Code is now changing the entire software engineering workflow. But how can data engineers truly achieve Vibe Coding? That