标签：#data

DuckDB 1.5.2, PostgreSQL Internal Stats, and SQLite Virtual Table xUpdate Deep Dive Today's Highlights This week brings a stable new patch release for DuckDB, enhancing performance and adding DuckLake support. We also delve into PostgreSQL's internal statistics for better tuning and explore advanced SQLite virtual table implementation via xUpdate . Announcing DuckDB 1.5.2 (DuckDB Blog) Source: https://duckdb.org/2026/04/13/announcing-duckdb-152.html DuckDB has released version 1.5.2, a patch update focusing on stability and performance. This release includes critical bugfixes that improve the reliability of the in-process analytical database, addressing various edge cases and enhancing overall robustness. Key enhancements also target performance bottlenecks, ensuring faster query execution for diverse analytical workloads. A significant new feature in this version is the official support for the DuckLake v1.0 lakehouse format. This integration positions DuckDB as a more robust tool for handling modern data architectures, allowing users to efficiently query and manage data stored in a lakehouse paradigm directly within their applications or analytical workflows. This update makes DuckDB even more compelling for embedded analytics and data pipeline use cases, providing a flexible and high-performance option for developers. Comment: Always good to see performance improvements and bug fixes for an embedded analytics powerhouse like DuckDB. DuckLake v1.0 support is a big step for managing structured data in lakehouse environments directly from DuckDB, enhancing its utility for complex data architectures. pg_stats: How Postgres Internal Stats Work (Planet PostgreSQL) Source: https://postgr.es/p/9mG This article from Planet PostgreSQL delves into the intricate mechanisms behind PostgreSQL's internal statistics, specifically focusing on pg_stats . Understanding how Postgres collects and utilizes these statistics is fundamental for effective database performance tuning and q

Nvidia announced a new cooling system that cuts water use inside the data center. But it does nothing to address AI's biggest water use — fossil fuel power plants.

আমার colleague এখন প্ল্যান দেখতে পারছে। Scan types বুঝতে পারছে। Join types বুঝতে পারছে। Estimate আর actual এর gap দেখতে পারছে। BUFFERS ও দেখছে। কিন্তু সে প্রশ্ন করল। এসব দেখে কি করব? Step by step কোন পথে যাব? আমি বললাম। পাঁচটা step আছে। অর্ডার অনুযায়ী। পর্ব ২ এ আমি বলেছিলাম scan types, join types, estimate আর actual এর gap। BUFFERS কি। এবার আসি সমাধান এ। Diagnostic Workflow আপনার কাছে একটা slow query এসেছে। কিভাবে debug করবেন? এই পাঁচটা প্রশ্ন করুন অর্ডার অনুযায়ী। ৯০% slow query প্রথম বা দ্বিতীয় ধাপেই solve হয়ে যায়। ১. Deepest Seq Scan দেখুন Table বড় কি না? Filter selective কি না? Missing index থাকলে add করুন। আজই শুরু করুন যখন একটা Seq Scan দেখবেন big table এ, প্রথমে WHERE clause টা check করুন। Selective কি না? ৫% এর কম row return হওয়ার কথা? যদি তাই হয়, index missing। CREATE INDEX idx_name ON table(column) run করুন। ২. Join types দেখুন কোনো Nested Loop আছে কিন্তু দুই পাশেই বড় table? Hash Join force করুন বা ডান পাশে index add করুন। আজই শুরু করুন Nested Loop দেখলে ডান পাশের table এ index check করুন। যদি না থাকে, create করুন। Index থাকা সত্ত্বেও planner Nested Loop use করছে? SET enable_nestloop = off temporarily disable করে দেখুন। Hash Join আসবে কি না। ৩. Row estimates দেখুন Estimate vs actual ১০x এর বেশি difference? ANALYZE table দিন বা predicate rewrite করুন। আজই শুরু করুন rows=1 estimate কিন্তু rows=100000 actual দেখলে ANALYZE tablename run করুন। Statistics refresh হবে। তারপর plan আবার দেখুন। যদি তাও না আসে, WHERE clause rewrite করুন। Function call থাকলে remove করুন। Type mismatch থাকলে fix করুন। ৪. BUFFERS add করুন কোনো node এ অনেক disk reads? Caching investigate করুন। আজই শুরু করুন EXPLAIN (ANALYZE, BUFFERS) run করে দেখুন shared read high কোথায়। সেই node টাই bottleneck। Index add করলে reads কমবে। Pre-warm cache করতে পারেন। Data pre-load করতে পারেন। ৫. Sorts আর hashes দেখুন কোনো spill-to-disk আছে? work_mem raise করুন বা sort eliminate করুন। আজই শুরু করুন Plan এ external merge Disk: 421MB দেখলে spill-to-disk হয়েছে। SET work_mem = '256MB' temporarily rais

Alina Krasavina explains how Delivery Hero successfully deprecated Google Analytics and migrated to an internal user tracking platform. She discusses how a simplistic, highly scalable architecture allowed them to handle 10 times more load while capturing 97% of tracking data. By Alina Krasavina

Huntress, HackerOne, Jamf, Recorded Future, and Tanium are among the cybersecurity companies that had data stolen following an earlier breach at market research firm Klue.

In this article, the author explores data poisoning as a threat to machine learning systems, covering techniques such as label flipping, backdoors, clean-label poisoning, and gradient manipulation. The article reviews real-world incidents, discusses the challenges of detecting poisoned data, and presents practical defenses, tools, and operational practices for securing ML training pipelines. By Igor Maljkovic

Big Tech is throwing big money into data center buildouts. As national opposition to the facilities grows, some workers are beginning to question whether it’s worth it.

If you're learning Data Analytics and looking to build a strong portfolio, working on real-world...

Let me confess something a little creepy. I have a habit of peeking at other people's dev posts. Not stealing the writing — relax. I run a tiny read-only job that fetches the public pages on dev.to, Zenn, and Qiita and counts only the boring parts: titles, post times, like counts. Who published what, at what hour, and how far it traveled. Then it tallies the lot. The reason is petty: my own posts weren't landing. The content is already in my hands — so I wanted to know how much the rest, the when and how you publish , actually moves the needle. By the numbers, not by gut. So I counted across three platforms. And the conditions that make a post fly turned out to be roughly mirror images between Japan (Zenn / Qiita) and the English-speaking world (dev.to). Here's the story. First, my most important disclaimer This post is full of numbers, so let me put up a guardrail before any of them. This is correlation, not causation . A result like "weekend posts don't do well" could mean the weekend itself is bad — or it could mean people who post on weekends are just dashing something off on the side. The data can't separate those. Please read it that way. Also, I only keep aggregate numbers I computed myself . I don't store or reuse anyone's article body (read-only GET, count the features, throw the page away). I peek, but only at the overall shape . Nobody gets singled out here. With that out of the way — four findings I enjoyed. 1. The best hour to publish is just your readers' time zone This one came out cleanest. On Qiita , posts published in the morning win (+32pt in the GOOD group). Midday is +14pt. Evening is -32pt, late night -14pt. Zenn likes midday too (+27pt). Late night is -15pt. dev.to is the exact opposite. Late night Japan time scores +7pt — Japanese evening is actually weak. The trick is obvious once you see it. dev.to's readers are English-speaking, mostly US. Late night in Japan is the US working day. Zenn and Qiita readers are in Japan, so the Japanese morni

We’ve all been there: waking up feeling like a zombie despite getting eight hours of sleep. While wearables give us data, they often fail to give us foresight . What if you could predict your stress levels 24 hours in advance? 🚀 In this tutorial, we are going to tackle HRV prediction (Heart Rate Variability) using a state-of-the-art Temporal Convolutional Network (TCN) . By leveraging the Oura Ring API and deep learning, we’ll transform non-stationary biometric time series into actionable insights. Whether you're into time series forecasting or building the next big health-tech app, mastering Temporal Convolutional Networks (TCN) is a game-changer for handling long-term dependencies without the vanishing gradient headaches of traditional RNNs. For those looking for more production-ready examples and advanced biometric signal processing patterns, I highly recommend checking out the deep-dives at WellAlly Blog , which served as a major inspiration for this architecture. The Architecture: Why TCN? Traditional LSTMs are great, but they process data sequentially, making them slow and prone to memory loss over long sequences. TCNs, however, use Dilated Causal Convolutions , allowing the model to look back exponentially further into the past with fewer layers. Data Flow Overview graph TD A[Oura Cloud API] -->|Raw JSON| B(Pandas Preprocessing) B -->|Cleaned HRV/Activity| C{Feature Engineering} C -->|Sliding Windows| D[TCN Model Training] D -->|Dilated Convolutions| E[Stress Trend Prediction] E -->|24h Forecast| F[Dashboard/Alerts] style D fill:#f9f,stroke:#333,stroke-width:2px Prerequisites To follow along, you'll need: Tech Stack : Python, TensorFlow/Keras, Pandas, Scikit-learn. Data : An Oura Cloud Personal Access Token (or use the mock data generator provided). Difficulty : Advanced (Buckle up! 🏎️). Step 1: Fetching Biometric Data First, we need to pull our "Readiness" and "Sleep" data. Oura provides high-resolution HRV samples (usually 5-minute intervals during sleep).

Whether you're brand new to Power BI or just getting started with data analytics, this guide walks you through everything you need to know about data modeling — from how tables connect, to the schemas that make your reports fast and reliable. What Is Data Modeling in Power BI? Imagine you have three spreadsheets: one with your customers , one with your products , and one with your sales transactions . Individually, each table tells you something. But together, they can tell you which customer bought which product, when, and for how much . That's exactly what data modeling is: the process of organizing your data tables and defining how they relate to each other so Power BI can combine them into meaningful reports and dashboards. A data model in Power BI has three core building blocks: Tables — your data sources (Excel files, databases, CSVs, cloud services, etc.) Relationships — the links between tables that tell Power BI how data connects Measures & Calculations — formulas (in DAX) that compute totals, averages, and other insights A well-designed data model is the difference between a report that loads in seconds and one that takes forever. It's also what keeps your numbers accurate and your dashboards easy to use. Why Does Data Modeling Matter? Here's a simple analogy: a city without roads is just a collection of buildings. Data modeling is the road system that lets you travel between your tables. Without a good data model: Your visuals may show incorrect or duplicated numbers Filters in one chart won't affect another Reports will be slow and hard to maintain With a good data model: Clicking on a customer in one visual automatically filters every other visual Calculations are accurate and consistent You can easily add new data sources without rebuilding everything Types of Tables: Facts vs. Dimensions Before diving into schemas and relationships, you need to understand the two types of tables that make up most Power BI models. Fact Tables A Fact table stores the ev

Anthropic recently reported that Claude now handles around 95% of its internal analytics requests, letting employees query business data independently instead of relying on data teams. The company attributes this result less to advances in models and more to data governance, semantic definitions, and operational discipline. By Renato Losio

We all know the grind of working with data, even with AI tools: every experiment starts with re-explaining everything, every iteration needs you to prompt, wait, review, correct, and repeat. And the moment you close the session, everything learned is gone. It makes us the bottleneck, and this hinders human-AI collaboration... So I built 𝐎𝐩𝐞𝐧𝐃𝐚𝐭𝐚𝐒𝐜𝐢, an autonomous agent purpose-built for DS/ML, and tested it on Kaggle. I enrolled in a recent competition, ran the agent with no hints, no guidance, while ironing my shirts. In one shot, it landed AUC 0.95, a top-30% finish out of 3K+ teams and 36K+ submissions using hashtag#Anthropic's Claude Sonnet 4.6. (More on this in README) The top-1 outperformed this agent by merely 0.004, but at the cost of massive manual effort even while using popular AI tools. The needed a dozen model families, deep learning, 400-feature notebooks, AutoML sweeps across many libraries, and 186 models ensembled carefully. Essentially a few weeks worth of effort and time!! OpenDataSci abstracts away all the complexity and has so much to offer for DS/ML automation: → Owns the entire development lifecycle from EDA to final evaluation → Plans, codes, and executes autonomously in a secure local sandbox → Self-reviews and corrects before anything reaches you → Remembers your data across sessions, gets smarter each run → Runs parallel experiments and ensembles → Has advanced context management for token efficiency and quality → Ships with predefined skills for DS/ML, so it knows how to do things right → Bring your own knowledge: out-of-the-box support for custom skills → Works with any major LLM provider (hashtag#Anthropic, hashtag#OpenAI, hashtag#Bedrock, hashtag#VertexAI, hashtag#Ollama, hashtag#vLLM, and any OpenAI-compatible server). This and so much more!! You set the goal. It does the work. No data science knowledge required. 🔗 https://github.com/f4roukb/open-data-sci 📦 pip install open-data-sci Spin it up on your data and see what it achieves!

An App Store rating looks like a verdict. It behaves more like a monument, built over years and slow to move. It says very little about how this month's users feel. I took the 12 most-rated Productivity apps on the US App Store, 32 million ratings between them, and split the headline star into the two numbers it hides: how far recent sentiment has fallen below the lifetime average, and whether the developer replies when users complain. How it is measured Population truth. Lifetime ratings and the star histogram come from Apple's full ratings data, every rating an app has ever received. Recent sentiment. A fixed window of the most recent reviews by date, so an app captured to a depth of thousands is not compared on a multi-year average against an app with a few hundred. Same window for everyone. Developer response. Reply share and median latency over that recent window. Complaints are bucketed with a rule-based taxonomy. It is a heuristic, not a trained classifier, and I treat it as one. What turned up The AI assistants now own this chart, and they reply to almost no one. App Lifetime Recent Reply share ChatGPT 4.8 4.18 0% Claude 4.7 3.06 0% Grok 4.9 3.77 0% Perplexity 4.8 3.60 0% Google Gemini 4.7 3.65 13% Dropbox 4.8 2.75 58% Gmail 4.7 2.40 26% Google Drive 4.8 3.90 23% Microsoft Authenticator 4.7 2.18 1% The older tools are the ones still in the trenches: Dropbox answers 58% of recent reviewers, Gmail 26%, Drive 23%. The steepest recent drops belong to Microsoft Authenticator (4.7 to 2.18), Gmail (4.7 to 2.40) and Dropbox (4.8 to 2.75). Plotted on two axes, backlash against response, every app falls into one of four archetypes: Firefighters, Ghost Ships, Complacent Giants and Resilient Leaders. Eight of the twelve are Ghost Ships, taking a recent hit in near silence. The honest limits Recent reviewers self-select toward the dissatisfied. A person who hits a bug is far more likely to leave a review than a contented one, so a low recent average blends genuine declin

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is...

We opened Chapter 1 with a single line, SELECT * FROM users WHERE id = 1 . For that line to leave the client and come back as a result row, the PostgreSQL backend went through five stages. First it decided which processing path the message should take; then the parser and analyzer turned the text into a tree and gave it meaning from the catalog. The rewriter expanded views and injected policies to transform the tree, the planner weighed the possible execution paths by cost and picked the cheapest one, and the executor followed that plan, pulling up one tuple at a time and sending them back to the client. Chapter 1 was a story about how a query is processed . What tree a given SQL becomes, what plan it turns into, in what order it runs. From start to finish, a chain of logical transformations. But what every one of those stages ultimately deals with is data. The executor pulls up tuples, yet where on disk those tuples lie and in what shape, how they come up into memory, Chapter 1 never asked. When the planner judged an index scan cheaper than a sequential scan, it never opened up what that index physically is. Chapter 1 followed only the logical journey of a query, leaving untouched the substance of the data that journey stands on. Chapter 2, Storage & Access Methods, opens up that substance. In what unit data sits on disk (page), where disk and memory meet (buffer manager), where and how a row survives (heap), and how that row is found quickly (B-tree and the specialized indexes). The very tuple the planner weighed by cost and the executor pulled up in Chapter 1, where it actually came from and how it came to be there, is what Chapter 2 reveals. If Chapter 1 was the logical life of a query, Chapter 2 is the physical dwelling of data. We now look at how the data a query reaches for actually lives on disk.

A scan node sits at the leaf of the tree and pulls rows from a single table. A join node sits in the middle and brings together the rows that its two children send up. It takes one row from users , one row from orders , checks whether they belong to the same user, and if they match, emits the combined row. PostgreSQL has three nodes for this one job: NestLoop, HashJoin, and MergeJoin. The reason a single task splits into three nodes is much like the reason scans did. There is more than one way to find matching pairs from two inputs, and which way is cheapest depends on the size of the inputs and the shape of the join condition. Deciding which way is cheapest, by costing the alternatives, was the planner's job in an earlier chapter. This section looks at what those three nodes actually do when they execute. Given the same two tables, the three find matches in completely different ways, and that difference in approach is exactly what tells them apart. How the three nodes route requests All three join nodes are internal nodes with two children. One child is called the outer, the other the inner. All three run on the Volcano model's pull framework: when the parent asks for the next row, the join node takes rows from its two children, builds one matched row, and sends it up. The only difference is the order and manner in which it routes pull requests to its two children. NestLoop pulls the inner from the start all over again for each outer row it receives. HashJoin slurps the inner in one pass to build an index in memory, then takes outer rows one at a time and probes that index. MergeJoin, on the assumption that both sides are sorted in the same order, advances both sides one step at a time in lockstep. NestLoop: rescan the inner for every outer row The simplest method is NestLoop. As the name says, the loops are nested. The outer loop takes one row from the outer; the inner loop scans the inner from beginning to end, looking for inner rows that match that outer row. Wh

Gelişmiş Veri İşleme (Python) Sıralama, Filtreleme ve Arama – Profesyonel Veri Manipülasyonu Rehberi Python’da veri işleme, sadece döngülerden ibaret değildir. Modern Python yaklaşımı; fonksiyonel programlama araçları , yüksek seviyeli built-in fonksiyonlar ve lambda ifadeleri ile daha kısa, daha okunabilir ve daha performanslı çözümler üretmeyi hedefler. Bu bölümde dört kritik alanı derinlemesine inceleyeceğiz: sorted() ile gelişmiş sıralama lambda ile karmaşık veri yapıları üzerinde sıralama filter() ve map() ile fonksiyonel veri dönüşümü any() ve all() ile toplu doğrulama (validation) Her bölümde gerçek dünya senaryoları ve hands-on örnekler olacak. 1. sorted() Fonksiyonu — Gelişmiş Sıralama Motoru 1.1 Temel Yapı sorted ( iterable , key = None , reverse = False ) Parametreler: iterable: Liste, tuple, set vb. key: Sıralama kriteri (fonksiyon) reverse: True → büyükten küçüğe 1.2 Basit Sıralama ```python id="s1" sayilar = [5, 1, 9, 3, 7] sonuc = sorted(sayilar) print(sonuc) --- ## 1.3 Ters Sıralama ```python id="s2" sayilar = [5, 1, 9, 3, 7] print(sorted(sayilar, reverse=True)) 1.4 Tuple Sıralama ```python id="s3" veri = (10, 5, 20, 15) print(sorted(veri)) --- ## 1.5 String Sıralama (ASCII mantığı) ```python id="s4" kelimeler = ["python", "ai", "data", "backend"] print(sorted(kelimeler)) 2. key Parametresi — Sıralamanın Beyni sorted() fonksiyonunun gerçek gücü burada başlar. 2.1 String Uzunluğuna Göre Sıralama ```python id="k1" kelimeler = ["python", "ai", "veri", "makineöğrenmesi"] sonuc = sorted(kelimeler, key=len) print(sonuc) --- ## 2.2 Sayıların Moduna Göre Sıralama ```python id="k2" sayilar = [10, 3, 7, 21, 14, 9] sonuc = sorted(sayilar, key=lambda x: x % 5) print(sonuc) 2.3 Tuple Sıralama (Gerçek Dünya) ```python id="k3" urunler = [ ("Laptop", 45000), ("Mouse", 500), ("Monitör", 12000) ] sonuc = sorted(urunler, key=lambda x: x[1]) print(sonuc) --- ## 2.4 Çok Katmanlı Sıralama Fiyat → sonra isim ```python id="k4" urunler = [ ("Laptop", 45000), ("Mouse", 500),

By the time 1.4 ends, the planner has produced one PlannedStmt. Inside it is an execution tree built from Plan nodes, frozen into a form you can follow step by step, something like "go into the primary key index on users, fetch the one matching row, then output that whole row." But that is still only a blueprint. Reading actual pages off disk, picking out the rows that match the condition, handing results back to the caller: none of that has happened yet. The stage that takes that blueprint and produces actual rows is the executor. The difference between the planner and the executor is the difference between deciding and doing. The planner was the stage that weighed "which index, in what order, with what join method" by cost and chose . The executor takes the chosen approach and carries it out as is . There is nothing left to choose. It just runs the nodes baked into the plan tree and pulls rows out of them. To run it, the executor takes the Plan tree it received and turns it into a PlanState tree. The Plan tree is the static blueprint the planner made, and it does not change during execution. But to actually run, each node needs state that changes as execution proceeds: which row it is reading now, whether the hash table is fully built, what tuple it has buffered from a child. So when execution begins, a PlanState tree with the exact same shape as the Plan tree is created. The blueprint Plan tree is left untouched, and the running state lives in that PlanState tree instead. How the executor produces result rows is the heart of the stage. The executor does not build the entire result set at once and stack it up. Instead, it asks the topmost node of the tree for "the next row," and that request travels down the tree to the leaves. When a leaf scan node reads one row from a page and passes it up to its parent, that row climbs up one level at a time through joins and filters until it reaches the top. The top sends that single row to the caller (the client, or the targe

Everything from 1.4.1 through 1.4.9 happened inside a single function, standard_planner() . Building paths, costing them, searching for a join order, estimating cardinality from statistics: all of it runs inside that one function. Yet PostgreSQL does not call standard_planner() directly. It puts another function, planner() , one step ahead of it, and has planner() call standard_planner() . And planner() can be made to call some other function instead of standard_planner() . That replacement is what the planner hook enables. When pg_stat_statements measures per-query planning time, or pg_hint_plan rewrites a plan according to hints, it all goes through this hook. Let's look at how PostgreSQL provides a way to observe or change planning behavior without touching a single line of the core, and how external code plugs into it. All planner() does is check the hook The body of planner() is essentially this. if ( planner_hook ) result = ( * planner_hook ) ( parse , query_string , cursorOptions , boundParams ); else result = standard_planner ( parse , query_string , cursorOptions , boundParams ); planner_hook is a global function pointer. Its default value is NULL , in which case standard_planner() is called right away. A plain PostgreSQL build always takes this path: planner_hook is empty, so the incoming query goes straight to standard_planner() . The key here is the type of planner_hook . typedef PlannedStmt * ( * planner_hook_type ) ( Query * parse , const char * query_string , int cursorOptions , ParamListInfo boundParams ); This signature is identical, down to the character, to that of planner() and standard_planner() . It takes the same Query and returns the same PlannedStmt (the execution plan). So external code only has to write a planner function matching this type and store its address in planner_hook . Let's call this function, written by external code to register in planner_hook , a custom planner function. The moment its address is stored, every planning reque