标签：#data

İlk yazıda Event Driven Architecture’ın temel kavramlarını, Kafka üzerinde topic/channel tasarımını, event-command ayrımını, schema contract’ları ve producer-consumer ilişkisini ele aldık. Bu yazıda odağı bir adım ileri taşıyıp event’in platform içindeki yaşam döngüsüne bakacağız. Çünkü EDA tasarımında asıl zorluk yalnızca event üretmek değildir. Asıl mesele, üretilen event’in güvenilir, izlenebilir, tekrar işlenebilir, zenginleştirilebilir ve farklı tüketiciler tarafından kullanılabilir hale gelmesidir. Bu yazıda şu sorulara odaklanacağız: Ham event platforma geldiğinde ne olur? Event nasıl doğrulanır, zenginleştirilir ve tüketilebilir hale gelir? Raw, validated, enriched ve curated topic’ler nasıl konumlandırılmalıdır? Bu yapı modern lakehouse mimarilerindeki Medallion yaklaşımıyla nasıl ilişkilendirilebilir? DLQ ve alert topic’leri ne zaman devreye girer? Replay, idempotency, monitoring, security ve governance nasıl düşünülmelidir? Event Pipeline Nedir? EDA mimarilerinde özellikle data platform projelerinde event’ler genellikle bir yaşam döngüsünden geçer. Bu yaşam döngüsü şöyle modellenebilir: raw -> validated -> enriched -> curated | | v v dlq alert Bu yapı, veri akışının aşama aşama olgunlaşmasını sağlar. Raw topic kaynaktan gelen ham event’i taşır. Validated topic schema ve temel kalite kontrollerinden geçmiş event’leri içerir. Enriched topic event’in referans veriler veya başka veri kaynaklarıyla zenginleştirilmiş halidir. Curated topic ise tüketiciler için güvenilir, normalize edilmiş ve iş anlamı netleşmiş event’leri temsil eder. Event Pipeline ve Medallion Architecture İlişkisi Bu yapı, modern lakehouse mimarilerinde sık kullanılan Medallion yaklaşımıyla doğal bir benzerlik taşır. Lakehouse tarafında Bronze katmanı ham veriyi, Silver katmanı temizlenmiş ve zenginleştirilmiş veriyi, Gold katmanı ise iş tüketimine hazır veri ürünlerini temsil eder. Kafka üzerindeki raw, validated, enriched ve curated topic’leri de benzer bir olgunlaşma mantığını akan veri ü

Hyperscalers have come under scrutiny for their impact on water quality and availability.

Columbia admits last year’s data breach exposed victims beyond its students, staff.

We dive again into CSS Pie Charts! This time, Author Antoine Villepreux delivers semantic and flexible charts without a single line of JS. Another Stab at the Perfect CSS Pie Chart… Sans JavaScript! originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

If you've been calling yourself a "prompt engineer" for the past two years, it's time to update your vocabulary — and your mental model. In 2026, the real leverage when building LLM-powered systems isn't in crafting the perfect sentence. It's in context engineering : designing everything an LLM sees before it ever generates a response. Andrej Karpathy coined the term in mid-2025, and it's since taken over serious AI engineering discussions. This article breaks down what context engineering actually is, why it matters more than prompt writing, and gives you concrete techniques you can apply today. What Is Context Engineering? Context engineering is the discipline of systematically designing the information environment that surrounds a prompt. Where prompt engineering asks "what should I tell the model to do?", context engineering asks "what does the model need to know to do it well?" Think of it this way: a doctor doesn't just answer the question you ask on the spot. They look at your chart, your history, your vitals, and then respond. Context engineering is building that chart for your LLM. The context window is the LLM's working memory — everything it can "see" at once. In 2026, these windows are massive: Claude Opus 4.x : 200K tokens GPT-4o : 128K tokens Gemini 2.5 Flash : Up to 1M tokens But bigger isn't automatically better. More tokens = more cost, more latency, and a real risk of what researchers call the "lost-in-the-middle" problem — where models process information at the beginning and end of the context more reliably than content buried in the middle. Why This Matters for Data Engineers Data engineers are increasingly building pipelines that feed LLMs: RAG systems, AI copilots for data quality, agents that write and review SQL, tools that summarize data lineage. In every one of these systems, the quality of what lands in the context window directly determines output quality. A poorly designed context is like feeding a senior analyst a jumbled mess of raw l

AWS disclosed that Resilient Network Graphs, a flat network architecture based on quasi-random graph theory, is now the default for most new data center builds. The design replaces fat-tree hierarchies with direct ToR-to-ToR mesh connections using passive optical ShuffleBoxes, cutting routers by 69%, boosting throughput by 33%, and reducing network power consumption by 40%. By Steef-Jan Wiggers

🚀 Overview — Why Semantic Search Matters Semantic search surpasses simple keyword matching because embeddings place texts in a high‑dimensional vector space where cosine similarity directly reflects intent. A dedicated vector store is therefore required to persist those embeddings and serve nearest‑neighbor queries efficiently. This post demonstrates a pinecone fastapi semantic search tutorial that wires a FastAPI service to Pinecone, showing the full data flow from embedding generation to similarity lookup. 📑 Table of Contents 🚀 Overview — Why Semantic Search Matters 🛠 Environment Setup — How to Install Dependencies 🐍 Python Virtual Environment 📦 Required Packages 📦 Building the FastAPI Service — How to Create the API 🧩 Data Model with Pydantic 🔗 Core FastAPI Application 🔎 Integrating Pinecone — How to Store and Query Vectors 🗂 Index Creation and Configuration 📤 Upserting Documents 🔎 Performing a Semantic Search 📊 Performance & Scaling — How Indexes Influence Latency 🟩 Final Thoughts ❓ Frequently Asked Questions How do I secure the Pinecone API key in production? Can I use a different embedding model? What happens if I need to change the index dimension? 📚 References & Further Reading 🛠 Environment Setup — How to Install Dependencies Creating a reproducible environment guarantees that the tutorial runs identically on any machine. 🐍 Python Virtual Environment $ python3 -m venv venv $ source venv/bin/activate (venv) $ python -V Python 3.11.5 Activating the virtual environment isolates package installations from the global interpreter. 📦 Required Packages $ pip install fastapi[all] uvicorn pinecone-client sentence-transformers Collecting fastapi[all] Downloading fastapi-0.109.0-py3-none-any.whl (48 kB) Collecting uvicorn Downloading uvicorn-0.24.0-py3-none-any.whl (66 kB) Collecting pinecone-client Downloading pinecone_client-2.2.2-py3-none-any.whl (81 kB) Collecting sentence-transformers Downloading sentence_transformers-2.2.2-py3-none-any.whl (1.1 MB) ... Successful

Activists say it’s the first time Big Tech employees have publicly called for regulations governing data center projects.

Google has shared details of its fleet wide large scale A/B experimentation system designed to standardize experiment assignment, exposure logging, and configuration propagation across distributed services. The approach enables consistent measurement across products, reduces experiment conflicts, and improves reliability of data driven decision making at scale. By Leela Kumili

From a massive DOGE data breach and the hacking of critical energy and water systems to the hack of an FBI surveillance system, here are the most damaging security incidents and data breaches of 2026.

Sepehr Khosravi discusses the evolution of developer productivity tools. Evaluating the strengths of tools like Cursor and Claude Code, he explains actionable techniques for senior engineers - including context engineering, custom rules, and Model Context Protocol (MCP) integrations. He shares real-world benchmarks and strategic frameworks for balancing AI adoption with clean code quality. By Sepehr Khosravi

Google, Microsoft, and other hyperscalers have come under scrutiny for their impact on water quality and availability.

CAP Theorem Explained: Choosing Between Consistency, Availability, and Partition Tolerance in Databases Imagine you're trying to book a flight online, and just as you're about to pay, the website crashes. When you try to book again, you find that the flight is now sold out, even though the website initially showed available seats. This frustrating experience is a classic example of a database trade-off between consistency, availability, and partition tolerance. The CAP theorem, first introduced by Eric Brewer in 2000, states that it's impossible for a distributed data store to simultaneously guarantee more than two out of these three principles. In this post, we'll delve into the world of CAP theorem, exploring its fundamentals, real-world database examples, and design implications. Introduction to CAP Theorem Understanding the Basics of CAP Theorem The CAP theorem is based on three primary principles: Consistency : Every read operation will see the most recent write or an error. Availability : Every request receives a response, without guarantee that it contains the most recent version of the information. Partition Tolerance : The system continues to function and make progress even when network partitions (i.e., splits or failures) occur. Importance of CAP Theorem in Distributed Systems In distributed systems, where data is spread across multiple nodes, the CAP theorem plays a crucial role in understanding the trade-offs between these principles. By grasping the CAP theorem, developers can design more resilient and scalable databases that meet the specific needs of their applications. Brief Overview of the Blog Post This post will explore the CAP theorem in depth, using real-world database examples to illustrate the trade-offs between consistency, availability, and partition tolerance. We'll discuss the fundamentals of CAP theorem, examine CA, CP, and AP systems, and provide guidance on designing for each combination. By the end of this post, you'll have a solid un

After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics. By Pranav Bhasker

Hand the same paired before/after dataset (n = 25) to ChatGPT five times. Same prompt: "These are the same subjects measured before and after an intervention. Did their scores change significantly?" Four of the five runs return p = 0.009 from a paired t-test. The fifth run does a Shapiro–Wilk normality check on the differences first, decides they're non-normal, switches to a Wilcoxon signed-rank test, and reports p = 0.000018 . All five reach the same conclusion (significant). But notice what happened: only one run out of five thought to check an assumption you'd want it to check. The other four skipped it. The choice of method — and the test statistic, and the p-value — depended on whether the LLM happened to run an assumption check that time. On borderline data, this is the difference between reject and don't reject. If you're using LLMs for exploratory data analysis on a weekend project, you might shrug. If you're using them for anything that gets cited, gets submitted to a regulator, or gets handed to a clinician, this is a problem. It's a known problem — Cui & Alexander (2026) documented exactly this kind of method-divergence empirically; AIRepr (Zeng et al., 2025) shows the same thing across reproducibility metrics. The current answer in the literature is to constrain the agent so its execution is replayable. But replayability fixes "did we run the same code." It doesn't fix "did we run the right analysis." I've spent the last two months building a different fix. The more interesting half is the architecture. Let me walk through it. The real problem isn't temperature The first reflex is "set temperature=0 ." It's not enough. temperature=0 doesn't make a tool-using agent deterministic across runs. Three reasons: Inference isn't bitwise deterministic, even at temperature=0. Production LLM serving batches requests dynamically, and the attention kernels aren't batch-invariant — so the same input produces different output tokens depending on what other requests it

With shiftbloom studio. I build tools and projects about a variety of experimental approaches to real-world problems. The issue for such use-case often was how most small media systems start out: one big always-on recorder that keeps costing money even when nothing is happening. For live capture you obviously need to stay ready at all times — sometimes you can’t risk losing the first minutes. But for everything else it’s complete overkill. The Core Problem Backfills, VOD downloads, clip imports, repairs and re-encodes are queue work. They can wait a few seconds, run on burst capacity, or even on a regular VPS or laptop. They don’t need the same always-hot infrastructure as the live recorder. That’s why I split the system. Instead of one large monolith, I deployed: Observer cells — only for live streams (time-critical) Harvest cells — for all queue processing (can be delayed) The Three Roles 1. Mothership A small control-plane cron job. It checks queue sizes, currently live channels and running observer tasks, then decides: how many harvest cells should exist right now which channels need an observer cell It’s intentionally simple. The database remains the single source of truth. 2. Observer Cells Each observer cell records exactly one live channel. It receives its assignment through environment variables: +++env OBSERVER_VOD_ID OBSERVER_CHANNEL_ID OBSERVER_CHANNEL_LOGIN OBSERVER_CHANNEL_NAME +++ It starts recording immediately, writes HLS segments to object storage, sends heartbeats, and waits a short standby window after the stream goes offline. This window is important because streams sometimes drop and reconnect quickly. Without it you end up with many small broken VOD fragments. 3. Harvest Cells These handle all background work: downloading VODs, re-encoding, recovering broken files, etc. They can run anywhere Docker is available — AWS tasks, a small VPS, or even a spare laptop. They only need outbound access to Postgres and object storage. What Changed Previous

Hello! I'm Satoshi Kaneyasu, DevOps engineer at Serverworks. In this article, I'll introduce the basic concepts and terminology of vector databases for those who are just starting to learn about them. Target Audience This article is aimed at beginners to vector databases. You may have heard that vector databases are related to LLMs and RAG, but aren't quite sure what they actually are. Think of this as written with that kind of reader in mind. What Is a Vector Database? A vector database is a database that stores data as vectors (arrays of numbers) and searches for data using "distance" or "similarity" between vectors. Traditional relational databases search for data using "exact match" or "partial match" (LIKE queries), but vector databases can search for things that are semantically similar . For example, searching for "weather in Tokyo" might return results like "temperature in Tokyo" or "weather conditions in Kanto" — data that differs as a string but is semantically related. Visualizing Vector Space In a vector database, all data is represented as points in a multidimensional space. When searching, the query is also converted into a vector, and data that is "close in distance" within that space is retrieved. This diagram represents it in two dimensions, but in a real vector database, proximity and distance are defined across many dimensions. Use Cases for Vector Databases Vector databases are used across a wide range of applications: Use Case Description RAG (Retrieval-Augmented Generation) Knowledge base search to provide external knowledge to LLMs. Allows internal documents and up-to-date information to be reflected in LLM responses Semantic Search Searching internal documents or FAQs by meaning rather than keywords. Handles spelling variations and synonyms Recommendation Recommending products and content whose vectors are close to a user's preference vector. Used as an alternative or complement to collaborative filtering Image Search Searching for similar im

A few months ago, I built an AI agent to automate one of the most repetitive parts of my workflow:...

A government committee says that the country’s growing dependence on the data analytics company is a serious liability.

The cybersecurity company is nearing a $300 million round led by Evolution Equity Partners.