标签：#Data

AI applications are generating more data than ever before. From model telemetry and user interactions to observability events and real-time analytics, modern systems need infrastructure that can ingest, process, and query massive datasets with low latency. That's exactly the problem ClickHouse is targeting with its latest release. The update introduces improvements across query performance, memory management, Kafka integration, lakehouse support, and developer tooling. While many of these changes appear incremental on the surface, together they highlight a much larger shift happening across the industry. One of the most notable additions is improved memory management for large joins. ClickHouse can now automatically spill hash joins to disk when memory usage exceeds configured thresholds. Instead of failing due to memory pressure, queries can continue running using more efficient execution strategies. For teams working with large feature tables, event enrichment, AI telemetry, or observability data, this can significantly improve reliability. The release also expands ClickHouse's Kafka capabilities with Schema Registry integration, AvroConfluent write support, metadata mapping, and zone-aware communication. These improvements make it easier to integrate ClickHouse into real-time event pipelines while reducing latency and unnecessary cross-zone traffic in cloud environments. Another major focus is support for modern lakehouse architectures. Improvements for Apache Iceberg and Apache Paimon strengthen ClickHouse's ability to query data stored in open table formats while maintaining high analytical performance. As more organizations separate storage and compute, ClickHouse is increasingly positioning itself as a high-speed query layer on top of cloud-native data lakes. Performance optimization remains a major theme throughout the release. Improvements include faster JOIN execution, better ORDER BY LIMIT performance, enhanced JSON processing, smarter index pruning, redu

A linked list is an ordered linear data structure where elements are not stored in sequential memory locations, instead they are stored in nodes that are linked together by a pointers. Linked list are used in data intensive application because linked list offer specific benefits for high frequency data manipulation, this benefits include: Efficient insertion and deletion Adding and removing elements from a linked list is highly efficient, unlike arrays which requires shifting all subsequent elements to maintain indexing. A linked list only requires updating the pointer. Dynamic sizing: linked list can grow or shrink during runtime without needing to pre-allocate memory. Memory management: Nodes in a linked list are only allocated when needed which prevents memory wastage. Flexible Traversal: Doubly and circular list allow you to move forward or backward, which makes them helpful for complex navigation The first node in a linked list is called the head which signifies the start of the list, while the last node is called the tail and has a pointer of null except in a circular linked list. Each node in a linked list has two things which are: the actual data the pointer or reference There are three main types of linked list: Singly linked list Doubly linked list Circular linked list Singly Linked List: Singly linked list are lists where each node has a next pointer that points to the next node. Doubly Linked List: Doubly linked list are list where each node has a next and previous pointer that points to the previous and next node. Circular Linked List: Circular linked list are list where the last node points back to the first node, forming a circle. Table of Contents create node class create linked list class isEmpty and getSize Methods prepend and append Method removeHead and removeTail Methods insert and search Methods getIndex and removeIndex Methods clear and print Methods create node class First let's open our code editor and create a new file called singlyLinkedLi

In this article, author Aaditya Chauhan discusses the limitations of RAG pipelines based purely on vector search and how an internal omni-search application using Reciprocal Rank Fusion (RRF) that combines BM25 and vector results, can enhance the search solution. By Aaditya Chauhan

Switching focus from Frontend development to Data Engineering means shifting from building user interfaces to architecting robust data pipelines. It’s a completely different mindset, and the learning curve is exciting! As I dive deeper into the world of Data, these are the 4 essential tools and concepts that have become the absolute backbone of my daily learning roadmap: 1️⃣ Python (The Swiss Army Knife): Coming from JavaScript/TypeScript, picking up Python has been a breath of fresh air. From writing custom ETL scripts to data manipulation with Pandas, it's the ultimate language for data manipulation. 2️⃣ Advanced SQL (The Core): It's not just about simple SELECT statements anymore. Mastering Window Functions, CTEs (Common Table Expressions), and query optimization is where the real magic happens when interacting with Data Warehouses. 3️⃣ ETL/ELT Pipelines: Understanding how to efficiently Extract, Transform, and Load data without breaking downstream analytics. Moving from UI state management to Data state management is a game-changer. 4️⃣ Cloud Ecosystems & Modern Stack: Exploring how data flows through modern cloud environments and learning how big data tools manage scale. The transition requires patience, but applying my previous engineering background to these new tools makes the journey incredibly rewarding. 💡 To the Data Engineers in my network: What is the one tool or concept you believe is a "must-have" for someone transitioning into the field today? Drop your advice below!

PostgreSQL 19 Graph Queries & REPACK; SQLite Advanced SQL Patterns Today's Highlights PostgreSQL 19 introduces major features with SQL/PGQ for graph queries on relational data and a new REPACK command to combat table bloat. Meanwhile, the SQLite community explores advanced SQL patterns for complex 'first match' selection logic, pushing the boundaries of efficient embedded database querying. SQL/PGQ in PostgreSQL 19: Graph Queries Without the Graph Database (Planet PostgreSQL) Source: https://postgr.es/p/9kW PostgreSQL 19 is set to integrate native graph querying capabilities through the introduction of SQL/PGQ (Property Graph Queries) and the GRAPH_TABLE construct. This innovative feature allows users to perform sophisticated graph-like analytics, such as pathfinding and pattern matching, directly on their existing relational datasets without requiring data migration to a specialized graph database. The syntax employed for defining graph patterns closely resembles that of Cypher, making it intuitive for developers familiar with graph query languages. Developers will be able to define 'nodes' and 'edges' from their relational tables and then utilize the GRAPH_TABLE function within standard SQL queries to traverse and extract complex, interconnected information. This effectively transforms PostgreSQL into a multi-model database capable of handling both traditional relational workloads and demanding graph analytics side-by-side, within a single consistent environment. This significant enhancement expands PostgreSQL's utility across a wide range of applications, including social network analysis, fraud detection, and supply chain optimization, where understanding relationships between data points is critical. By embedding graph capabilities, PostgreSQL removes the need for separate graph database infrastructure, simplifying application architectures and leveraging the robustness and familiarity of the PostgreSQL ecosystem for new and existing projects. Comment: This is

This article was originally published on davidohnstad.com . I cross-post here to reach the Dev.to community. { " @context ": " https://schema.org ", " @graph ": [ { "@type": "Person", " @id ": " https://davidohnstad.com/#author ", "name": "David Ohnstad", "url": " https://davidohnstad.com ", "sameAs": [ " https://www.linkedin.com/in/davidohnstad/ ", " https://orcid.org/0009-0007-9023-7456 ", " https://davidohnstad5.mystrikingly.com/ ", " https://github.com/davidohnstad40-netizen ", " https://hashnode.com/@davidohnstad ", " https://davidohnstad.com ", " https://davidohnstad.net ", " https://davidohnstad.info ", " https://david-ohnstad.com ", " https://davidohnstadminnesota.com " ], "jobTitle": "Senior Data Product Manager", "worksFor": { "@type": "Organization", "name": "Veeam Software", "url": " https://www.veeam.com " }, "alumniOf": { "@type": "CollegeOrUniversity", "name": "College of St. Scholastica" }, "address": { "@type": "PostalAddress", "addressLocality": "Duluth", "addressRegion": "MN", "addressCountry": "US" }, "description": "Senior Data Product Manager at Veeam Software, MS and MBA from the College of St. Scholastica, based in Duluth, Minnesota. Specializes in data architecture, AI/ML integrations, and SaaS platform development." }, { "@type": "Article", " @id ": " https://davidohnstad.com/data-product-manager-org-structure-reporting#article ", "headline": "Data Product Manager Org Structure: Reporting Lines That Matter", "description": "David Ohnstad reveals where data product managers actually fit in org charts and why reporting lines determine success. Real insights from a data PM restructure.", "url": " https://davidohnstad.com/data-product-manager-org-structure-reporting ", "datePublished": "2026-05-29T14:06:18Z", "dateModified": "2026-05-29T14:06:18Z", "author": { "@type": "Person", " @id ": " https://davidohnstad.com/#author " }, "publisher": { "@type": "Organization", "name": "David Ohnstad", "url": " https://davidohnstad.com ", "logo": { "@type"

Last week I wrote about why healthcare benefit data is still trapped in PDFs . The response told me something: people in this space know the problem is real. But extraction is only half the story. The harder question is: when an AI system pulls a copay amount from a carrier document, can you prove where that number came from? Not "the AI said so." Not a confidence score with no backing. Can you point to a specific page, a specific table cell, a specific paragraph in the source PDF and say: this value came from here? That is field-level provenance. And in healthcare, it is no longer optional. The Regulatory Floor Just Rose In January 2026, two state laws went into effect that changed the baseline for AI-generated content in healthcare. California SB 942 requires AI systems to disclose when content is AI-generated and to maintain audit trails. Texas HB 149 mandates transparency about AI decision-making processes in regulated industries, with healthcare squarely in scope. These are not theoretical. They are enforceable. And they are just the beginning. CMS transparency mandates tighten every year. Gartner declared digital provenance an enterprise baseline for 2026. The industry is not moving toward provenance. It has arrived. The Problem with Self-Reported Citations Most AI extraction systems today work like this: a language model reads a document, extracts values, and reports where it found them. The model does the extraction AND the citation. It is grading its own homework. This seems fine until you look closer. A language model that hallucinates a copay amount will also hallucinate the page number it came from. The citation and the extraction fail together, silently, in the same direction. In a coverage dispute that ends up in a regulatory proceeding, "the AI told us it found this on page 3" is not evidence. It is hearsay from a statistical model. What Deposition-Grade Provenance Looks Like Field-level provenance means every extracted value carries metadata from an

The company says it needs "significant" water resources to cool its data centers, and that access to abundant, affordable water is a challenge.

Anthropic introduced Dynamic Workflows, a new capability for Claude Code designed to handle complex software engineering tasks by coordinating large numbers of AI agents within a single workflow. The feature allows Claude to dynamically create orchestration scripts, break work into subtasks, run them in parallel, and validate results before presenting a final answer. By Robert Krzaczyński

During maintenance tasks like backup or inspection, you often need to switch a gbase database cluster to READONLY mode and back to NORMAL afterwards. This post shows how to perform the switch online with gcadmin — no downtime required. Tools and Prerequisites Tool : gcadmin , located on any Coordinator node. User : Must be the cluster installation user (default gbase ). Pre‑check : Ensure gcware services are running ( gcware_services status ). Scope : The mode change applies cluster‑wide; no need to repeat per node. Step‑by‑Step (READONLY → NORMAL) Step 1: Check the Current Mode # Cluster‑wide gcadmin showcluster # Specific VC in a multi‑VC environment gcadmin showcluster vc vc1 Look at the VIRTUAL CLUSTER MODE field. Proceed only if it shows READONLY . Step 2: Switch to Normal Read/Write Mode # Single VC or whole cluster gcadmin switchmode normal # Specific VC gcadmin switchmode normal vc vc1 The switch takes effect in seconds. The cluster synchronises the new mode to all nodes automatically — no process restart is needed. Step 3: Verify the Change Run gcadmin showcluster again. When VIRTUAL CLUSTER MODE shows NORMAL , the cluster is fully writable again. Production Notes Business impact : An online switch does not interrupt running read queries. Pending write operations queued during READONLY mode are executed automatically after the switch. Node to run on : Any single Coordinator node is enough. Privileges : Only the gbase user can switch modes. Reverse switch : To return to read‑only mode, use gcadmin switchmode readonly [vc vc_name] . Troubleshooting : If the command fails, check gcware service health and node status first, then retry. Companion Commands # Check all cluster processes gcluster_services all info gcware_services all info # View detailed VC information gcadmin showvc The mode‑switch mechanism in GBase 8a is built for high availability. Following the "verify → switch → re‑verify" workflow lets you change cluster modes safely and transparently in a g

BadHost is a high-severity authentication bypass vulnerability in the widely used Python web framework Starlette, with 325 million weekly downloads. The flaw allows attackers to use malformed HTTP Host headers to bypass path-based access controls and access sensitive AI agent infrastructure, among other systems. By Sergio De Simone

Crescent Island is an air-cooled chip that uses LPDDR5 memory.

Introduction In contemporary research, the volume of biological data deposited in public repositories is growing exponentially. The Gene Expression Omnibus (GEO), NCBI Gene, PubMed, and UniProt accumulate thousands of new records daily, including sequences, expression profiles, scientific articles, and functional annotations. On the one hand, this scenario represents a unique opportunity for biomedical research. On the other hand, the diversity of data formats, access protocols, and metadata models creates a significant barrier: each source requires a specific collector, distinct rate-limiting strategies, and its own validation logic. Above all, the lack of standardization in data storage compromises the reproducibility of scientific studies. The need for integrated tools capable of unifying data extraction, curation, and persistence has been widely discussed. In practice, ad hoc solutions such as isolated scripts for individual repositories generate redundant work and make maintenance difficult. First and foremost, it is necessary to establish an architecture that treats data collection as a service rather than a collection of scattered artifacts. This work presents Project 1 of the Integrated Bioinformatics Platform: a containerized Biomedical Data Collector coupled with a Data Lake. Its objective is to provide a REST API capable of triggering asynchronous data collections from the four aforementioned sources, storing immutable raw data in MinIO, and persisting metadata in PostgreSQL, all while ensuring traceability and resilience. Development The system architecture is divided into three main layers. The first is the API and orchestration layer , implemented using FastAPI. Its five endpoints — POST /collections , GET /collections , GET /collections/{id} , GET /collections/{id}/download/{dataset_id} , and GET /health — expose a clean interface for initiating and monitoring collection processes. The second layer is the collector engine , composed of abstract classe

Strava will charge a flat monthly fee from developers to access its API

Modern search engines do more than match keywords. When you search for "cozy mystery set in Paris" and get results for "atmospheric detective novel in France" that's vector search at work: documents and queries are converted into lists of numbers, called embeddings, and the search engine finds the documents whose numbers are closest to the query's. Manticore Search supports this natively. Under the hood, it uses a data structure called HNSW: a graph that connects nearby vectors, so it can find nearest neighbors quickly without scanning every document. That makes vector search fast enough to run on millions of documents in milliseconds. But HNSW has an inefficiency. Early in the traversal, almost every distance computation finds a better candidate than the ones already in the result set. As the search goes on, those improvements become rarer, but the algorithm keeps traversing the graph until it exhausts its exploration budget. By that point, the result set has often already converged, and the remaining work does little or nothing to improve it. Early termination fixes this by detecting that point and stopping early. The effect becomes more noticeable as k grows, where k is the number of nearest neighbors the query asks Manticore to return. Returning more neighbors requires more graph exploration, and much of that extra work happens after the result set has already stabilized. That also makes early termination more valuable, because it has more unnecessary work to cut. This gets more pronounced with vector quantization . Quantization compresses stored vectors to save memory, which slightly lowers search precision. To recover it, Manticore uses oversampling : it fetches 3x more candidates than requested, then rescores them using the original full-precision vectors. With the default 3x oversampling, HNSW explores many more candidates per query. Large k values often come from this kind of candidate expansion: an application may ask the vector index for hundreds or thous

0A000 feature not supported 는? PostgreSQL 에러 코드 0A000 은 현재 사용하려는 기능이 PostgreSQL에서 지원되지 않거나, 특정 컨텍스트에서는 사용할 수 없음을 의미합니다. 주로 트랜잭션 내부에서 허용되지 않는 명령을 실행하거나, 해당 버전의 PostgreSQL에서 아직 구현되지 않은 SQL 표준 문법을 사용할 때, 또는 복제(Replication) 환경의 제약으로 인해 발생합니다. 실무에서는 특히 CREATE DATABASE , VACUUM , CLUSTER 같은 명령을 트랜잭션 블록 안에서 실행하거나, Logical Replication 슬롯과 관련된 작업을 수행할 때 자주 마주치는 에러입니다. 주요 발생 원인 1. 트랜잭션 블록 내에서 허용되지 않는 DDL 명령 실행 PostgreSQL은 일부 DDL 명령을 트랜잭션 블록( BEGIN ... COMMIT ) 내에서 실행하는 것을 허용하지 않습니다. CREATE DATABASE , DROP DATABASE , CREATE TABLESPACE , DROP TABLESPACE , VACUUM , CLUSTER 등의 명령은 트랜잭션 컨텍스트 밖에서 단독으로 실행되어야 하며, 이를 무시하고 트랜잭션 내부에서 호출하면 0A000 에러가 발생합니다. 2. 특정 PostgreSQL 버전에서 지원하지 않는 문법 또는 기능 사용 SQL 표준에는 정의되어 있지만 PostgreSQL의 해당 버전에서 아직 구현되지 않은 기능을 사용할 때 이 에러가 발생합니다. 예를 들어, 구버전 PostgreSQL에서 LATERAL JOIN , MERGE 문, GENERATED ALWAYS AS (expression) STORED 컬럼 정의 등을 사용하거나, 특정 윈도우 함수 옵션 조합을 사용하는 경우 해당 에러를 만날 수 있습니다. 3. 논리 복제(Logical Replication) 또는 스트리밍 복제 환경에서의 제약 위반 Standby 서버나 Logical Replication 구독자(Subscriber) 측에서 쓰기 작업이나 특정 관리 명령을 실행하려 할 때 0A000 에러가 발생합니다. Hot Standby 상태의 서버에서 DDL을 실행하거나, Logical Replication 슬롯이 활성화된 상태에서 지원되지 않는 방식으로 복제 슬롯을 조작하려는 경우 이 에러를 마주치게 됩니다. 해결 방법 원인 1: 트랜잭션 블록 내 허용되지 않는 DDL 실행 트랜잭션 블록을 제거하고 해당 명령을 단독으로 실행하는 것이 핵심입니다. 아래는 잘못된 예제와 올바른 예제를 비교한 것입니다. ❌ 잘못된 예 (0A000 에러 발생) BEGIN ; CREATE DATABASE myapp_db WITH OWNER = myapp_user ENCODING = 'UTF8' LC_COLLATE = 'ko_KR.UTF-8' LC_CTYPE = 'ko_KR.UTF-8' ; COMMIT ; -- ERROR: 0A000: CREATE DATABASE cannot run inside a transaction block ✅ 올바른 예 (트랜잭션 블록 밖에서 실행) -- 트랜잭션 블록 없이 단독 실행 CREATE DATABASE myapp_db WITH OWNER = myapp_user ENCODING = 'UTF8' LC_COLLATE = 'ko_KR.UTF-8' LC_CTYPE = 'ko_KR.UTF-8' ; VACUUM 명령도 동일한 원칙이 적용됩니다. -- ❌ 잘못된 예 BEGIN ; VACUUM ANALYZE public . orders ; COMMIT ; -- ✅ 올바른 예: 트랜잭션 밖에서 단독 실행 VACUUM ANALYZE public . orders ; -- ✅ 특정 테이블만 선택적으로 VACUUM VACUUM ( VERBOSE , ANALYZE ) public . orders ; 애플리케이션 코드(예: Python psycopg2)에서 자동 커밋을 끈 상태로 VACUUM을 호출하는 경우도 흔한 실수입니다. import psycopg2 conn = psycopg2 . conn

Environmental activist Erin Brockovich has a new mission.

How a Product Sync Automation Project Transformed Customer Onboarding When people think about impactful engineering work, they often imagine distributed systems, high-scale infrastructure, or complex algorithms. One of the most impactful projects I worked on wasn't any of those. It was solving a seemingly simple problem: Keeping product data in sync across multiple retail systems. Years later, our CEO still remembers how much smoother customer onboarding became after this project. The Context: What is Commerce Connect? At Casa Retail AI, we have an internal platform called Commerce Connect (CC) . Commerce Connect acts as the central Product Information Management (PIM) system and serves as the source of truth for product information. Under the hood, it is built on top of a customized version of the open-source e-commerce platform Spree Commerce , extended with multi-vendor and multi-tenant capabilities. Its primary responsibility is simple: Collect product information from multiple retail ecosystems and distribute it to every Casa product that needs it. Once product data enters Commerce Connect, it is synchronized to multiple downstream systems. Why Product Data Matters Many applications inside Casa depend on product information. Product Consumers Once product data enters Commerce Connect, it is distributed to multiple systems across the Casa ecosystem. Customer-Facing Applications Several products rely on product information to provide context and improve customer experience: Lead management applications use product information during customer interactions. Ticket management systems link customer issues to specific products. Digital receipts display product names, images, and related details. Analytics & Reporting Product data powers business dashboards and reports, helping retailers answer questions such as: Which categories perform best? Which products attract the most attention? Which products generate the most complaints? It is also used for filtering and segme

Now that almost everyone has thought about or is actively integrating AI workflows into their projects, some might ask is this all worth the cost? Many think the current economics of the AI space don't scale and that there will be upward price movement. Others still might not be comfortable with sending their data to remote services for processing. Then there is the crowd that wants to deploy models in small spaces with limited compute. Are there ways we can deploy small models locally and run at a lower cost? Yes with Knowledge Distillation . Knowledge distillation can get a bad rap due to it's questionable use in training some Large Language Models (LLMs). But it's a perfectly valid way to transfer performance from a larger model to a smaller one. Especially when both models are yours and/or open. This article will explore progressive distillation which is a technique to incrementally transfer knowledge from a series of larger teacher models into a smaller student. Install dependencies Install txtai and all dependencies. pip install txtai [ pipeline - train ] datasets Setup the Training Pipeline The first step we need to do is setup up the training pipeline. We'll use the Hugging Face Training framework to build a series of models. The following code establishes a train method, test method and loads the classification training data. from datasets import load_dataset from transformers import AutoModelForSequenceClassification , AutoTokenizer from txtai.pipeline import HFTrainer , Labels def train ( teacher , student , distillation , ** kwargs ): trainer = HFTrainer () model = AutoModelForSequenceClassification . from_pretrained ( student , trust_remote_code = True ) tokenizer = AutoTokenizer . from_pretrained ( student , trust_remote_code = True ) return trainer ( ( model , tokenizer ), ds [ " train " ], columns = ( " sentence " , " label " ), maxlength = maxlength , teacher = teacher , distillation = distillation , ** kwargs ) def test ( model ): labels = Labels (

The Theorem That Changed How We Think About Databases In 2000, Eric Brewer stood at a conference and proposed a conjecture that would reshape distributed systems forever: "You can only guarantee two of these three properties at the same time: Consistency, Availability, and Partition Tolerance." Two years later, Seth Gilbert and Nancy Lynch proved it mathematically. It became known as the CAP Theorem — and every distributed system architect since has had to wrestle with it. It sounds abstract. But once you understand it, you'll never look at a database choice the same way again. You'll understand why Amazon DynamoDB and Google Spanner make opposite architectural choices. You'll know why your bank uses PostgreSQL while Twitter uses Cassandra. Let's break it down from first principles. The Three Properties C — Consistency Every read receives the most recent write, or an error. There's only one version of the truth — all nodes agree. Not the same consistency as ACID . CAP consistency (linearizability) means every read reflects the latest write across all nodes. ACID consistency means transactions don't violate database constraints. Different concepts, same confusing word. A — Availability Every request receives a non-error response — though it might not be the most recent data. The system is always up and answering. Note: "Available" in CAP doesn't mean "fast." It means "responds without error." A system that always returns a (possibly stale) answer is Available. P — Partition Tolerance The system continues operating even when network messages between nodes are lost or delayed. A partition is when part of your distributed system can't communicate with another part. The Unavoidable Truth: P Is Not Optional Here's the insight that makes CAP actually useful: In any real distributed system, partitions will happen. Networks fail. Cables get cut. Data centers lose connectivity. AWS regions go down. Since you must tolerate partitions (or have a single-server system, which does