标签：#data

This isn't a syntax error or a bug in the query — it's Power Query's Formula Firewall refusing to combine data from more than one source until it knows whether that's actually safe. Combining a private/organizational source with a public one (an internal database and a public web API, for example) can leak data from one into the other; the firewall blocks it by default rather than guessing. Why This Exists Every data source in Power Query has a privacy level — Public, Organizational, or Private — set the first time it's connected to. When a query's steps end up needing to send data from one source into a call against a different source, Power Query checks whether the privacy levels involved allow that combination. Originally published on PBIDocs — Power BI documentation covering DAX, Power Query, data modeling, and Microsoft Fabric.

This is the data appendix for Posts 1–4 . The narrative and takeaways live in the main posts. This page is pure measurement. Eval set: 22 questions (P1) → expanded to 31 questions (P2 onward) Corpus evolution: P1/P2: ~3,700 docs (raw threads + code chunks) P3/P4: 16,315 docs (distilled threads + bursts + code) Quick comparison (same 31-question set) Metric Vector P2 Hybrid P2 Vector P3 Hybrid P3 Hybrid + Rerank (P4) recall@1 0.68 0.61 0.52 0.39 0.87 recall@3 0.84 0.65 0.71 0.65 0.94 recall@10 0.90 0.90 0.81 0.94 0.94 MRR 0.77 0.67 0.63 0.57 0.90 Takeaway: Hybrid alone never beat pure vector on this corpus. Hybrid + LLM rerank is the first clear win. P1 — Naive vector baseline Corpus: 3,000 raw issue threads + 687 code chunks Embeddings: BGE-M3 (1024d), max_seq_length=1024, HNSW cosine Numbers (22 questions) Metric Score recall@10 1.00 (22/22) recall@3 0.95 recall@1 0.77 (17/22) Main k=1 misses Exact error pastes ( TypeError: Object of type int64... , AttributeError: 'Depends'... ) — ranked 4–5 instead of 1 jsonable_encoder code chunk outranked by issues about the function API key header implementation (code vs similar issues) Paraphrase questions (dependency injection outside routes, custom 404) Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal. Ops notes Ingest wall time ~40 min (GitHub API is the bottleneck) BGE-M3 OOM on Apple Silicon fixed by capping max_seq_length=1024 Python 3.13 + uv editable install issue fixed by pinning 3.12 P2 — Hybrid (vector + FTS + RRF) Corpus: Same size as P1, with better comment pagination and symbol-based code IDs Eval set: Expanded to 31 questions (added exact error pastes + rare identifiers) Numbers Metric Vector FTS Hybrid recall@1 0.68 0.42 0.61 recall@3 0.84 0.48 0.65 recall@10 0.90 0.65 0.90 MRR 0.77 0.47 0.67 Headline: Hybrid is not a strict win over vector-only. Where hybrid helped Exact error pastes (e.g. TypeError: int64 is not JSON serializable ) → moved from rank 5 → 1 Near-d

You deleted the row. The secret is gone from the app, the queries return nothing, and the dashboard is clean. In SQLite — the database behind most session stores, browser profiles, and agent state files — that delete is a fiction. The bytes are still in the file. Three ways deleted data survives 1. Free pages. SQLite doesn't zero out the space a deleted row occupied. The page is marked free and added to the freelist; the old bytes stay until they're overwritten by a future write. A file that's been deleted-from is a forensics goldmine: recover the freelist pages and the "deleted" rows come back. 2. FTS virtual tables. If the database uses SQLite's full-text search (FTS5), the FTS index keeps its own copies of the indexed text, maintained separately from the source tables. Delete the row from the source table and the FTS index still contains the tokens — searchable. This is the one that catches people: their app shows the secret is gone, and the FTS index still has it. 3. WAL and journal files. In WAL mode, recent writes live in the -wal file; transactions in the -journal file. Both can retain pre-delete content until checkpointed or cleaned. "Deleted" in SQLite means "no longer referenced", not "no longer present". What erasure actually requires Making a secret physically disappear from a SQLite database takes three operations, in order: Replace the value everywhere it lives. Known secret values get replaced across all tables; pattern matches (API key formats) get masked. Two layers, because you can't enumerate every secret that leaked. Rebuild the FTS indexes. INSERT INTO t(t) VALUES('rebuild') style rebuilds, or drop/recreate the virtual tables — so the index no longer contains the old tokens. Run VACUUM. VACUUM rewrites the entire database file, copying only live data into a fresh file — free pages with old bytes are discarded in the process. After VACUUM, the file's raw bytes no longer contain the secret. (Note: VACUUM doesn't shrink WAL files; those need a chec

Nikolay Samokhvalov has developed PGSimCity, an open-source educational tool that visualises PostgreSQL mechanics as a 3D spatial simulation in the browser. It assists backend developers and site reliability engineers in understanding SQL and the dynamics of kernel execution. The project is available on GitHub and aims to enhance understanding of database architecture through interactive elements. By Olimpiu Pop

Adapted from the SQL Essentials Companion Guide . You write a query to find every customer with no phone number on file. WHERE phone = NULL looks obviously correct — and it returns zero rows, even though you can see NULL sitting right there in the column. Nothing crashes. No error. The query just quietly lies to you about what's in the table. This isn't SQL being broken. It's SQL being consistent about something most languages don't force you to think about: NULL doesn't mean "nothing," it means "unknown." And you can't compare something to unknown with = and expect a real answer. What's actually happening Take this table: -- customers | id | name | phone | | ----|-------------|------------| | 1 | Jordan Lee | 555 - 0142 | | 2 | Sam Rivera | NULL | | 3 | Alex Chen | 555 - 0198 | SELECT name FROM customers WHERE phone = NULL ; -- returns 0 rows SQL doesn't evaluate conditions as just true or false — it has a third result: unknown . phone = NULL asks "does this unknown value equal this other unknown value?" There's no way to answer that, so SQL returns UNKNOWN for every single row, including Sam Rivera's. And WHERE only keeps rows where the condition is TRUE . UNKNOWN doesn't qualify, so the row gets filtered out — the exact same as if it had evaluated to FALSE . This is true even for the row that "should" match. NULL = NULL isn't TRUE — it's also UNKNOWN . NULL never equals anything, not even another NULL . That's the whole rule, and it applies uniformly, which is why = can't be patched into working here — it's not almost right, it's answering a different question than the one you're asking. The fix, step by step Recognize the symptom : a query that runs cleanly but returns fewer rows than it should — especially zero rows when you can see matching data — with a NULL column somewhere in the WHERE clause. Swap = for IS NULL (or != for IS NOT NULL ). These are dedicated operators built specifically to test for absence, not comparison operators being asked to do somethin

I audit catalog migrations for a living. Every disaster I've seen was preventable — not by hiring better, but by agreeing in writing what "done" means before work starts. Copy this list. Send it to whoever is doing your migration. Ask them to commit to each line — and note the italics: every promise comes with a way you can check it yourself in about two minutes, no tools, no trust required. Every product made it across — none lost, none duplicated. Compare row counts in both files. Every variant made it across. Pick any product, count its rows in both files. No handle silently renamed ( -1 , -2 suffixes). Search the new file for -1 , -2 . SKUs unchanged and unique. Pick 5 SKUs from your export, find them in the new file. Prices and compare-at prices identical. Pick any product, compare both price fields. Inventory identical. Same spot-check. No missing titles, vendors, types, or prices. Sort each column, look for blanks at the top. Images attached to the right variant , not dumped at product level. Open a product with colors; each color shows its own image. Every image link loads. Click any 5 image URLs. Collections intact. Pick a collection, compare its product count. Custom fields (metafields) survived. Open a product that had them. Every old URL redirects. Try 5 URLs from your old sitemap. No garbled characters. Search the file for †. No description empty or cut short. Read 5 descriptions in both files. Description formatting survived (bullets, tables). Same 5 products. Option names still meaningful ("Size", not "Option1"). Open any product with options. Option values mean what they meant. Compare the value lists. Two more things worth writing down: What happens if a check fails — fix at no charge? partial refund? Agree now, not later. Anything already broken in your source data — list it upfront so nobody argues about whose fault it was. If your provider hesitates to commit to a list like this, that hesitation is information. (I keep a pre-filled version of t

If you work with external APIs, you know the problem. There is no dev.github.com , no staging.api.companieshouse.gov.uk , no test endpoint for the thing you actually depend on. Production is the only source. So how do you get a development environment without re-fetching everything you already have? The usual answer is to copy: duplicate the warehouse, or keep a separate dev database and sync it periodically. Both are slow, both drift, and both cost storage in proportion to the number of people on the team. Interlace does something else. An environment is not a copy of your data, it is a set of views over it. Fingerprints first Every model gets a fingerprint: a hash of its canonical SQL — or its Python source — together with its strategy configuration and its upstream fingerprints. A build writes an immutable physical table named after that fingerprint. interlace__main.orders__a1b2c3 That table never changes. If the model's definition changes, the new version gets a new fingerprint and a new table, and the old one stays exactly where it is. An environment is then just a set of views pointing at fingerprinted tables. Production is the unprefixed namespace; every other environment prefixes its schema. Environment View for main.orders prod main.orders dev dev__main.orders pr-142 pr-142__main.orders Consumers and BI tools connect to main.orders and never learn that a fingerprint exists. There is no environment list to configure, either — an environment exists once something has been promoted to it. Why the sandbox is free Here is where the re-fetching problem disappears. Applying to a sandbox does not rebuild models whose fingerprint already exists. It points the sandbox's views at the tables production already built. interlace apply --env dev Change one model out of forty and the sandbox builds one model. The other thirty-nine are reused — not copied, reused, the same physical tables production is reading. The expensive source extract that ran this morning is the table

Real transaction data is never clean — and the worst part is that it looks clean. This is a short story from a real dataset (UCI Online Retail: 541,909 e-commerce transactions) about the quietest way to destroy data: silent type coercion. All numbers below come verbatim from an executed notebook. The head looks perfect Peek at the first rows of the file and InvoiceNo parses as clean integers — 100% parse rate, full confidence. Any type-inference step, mine included, would call it int64 and move on. Measure the whole file instead of the head, and the number drops to ~98%. The other 2%: invoice numbers starting with "C" — which in this dataset marks a cancellation . Coerce the column to numeric and every one of them becomes NaN : Invoice numbers destroyed by numeric coercion: 9,291 DextraLoaderWarning: load: ambiguous decision(s): column 'InvoiceNo': ambiguous - float64 at parse_rate=0.98 An entire class of business events — silently gone. No exception, no crash. That's what makes coercion the quietest bug in data work: the pipeline succeeds . Why those 9,291 rows matter They are not noise. They are the returns side of the business : cancelled orders worth 8.4% of everything sold. Lose them and every revenue number downstream is quietly wrong. One example of what they catch: the dataset's apparent #1 bestseller, "PAPER CRAFT, LITTLE BIRDIE" (168,470 GBP), is a phantom — a single 80,995-unit order entered at 09:15 and fully cancelled at 09:27 the same morning. Only the preserved cancellation rows expose it. The genuine bestseller is a cake stand. The fix: identifiers are labels, not quantities No library can know that "InvoiceNo" is an ID — that's domain knowledge. What a tool can do is disclose its guess and hand you a replayable plan you can correct: naive , plan = dx . load ( CSV_PATH , return_params = True ) # warns: ambiguous at 0.98 plan [ " columns " ][ " InvoiceNo " ][ " dtype " ] = " object " # invoices are labels plan [ " columns " ][ " StockCode " ][ " dtype

There is no manual for TronSoft. No API reference, no schema diagram, no forum thread explaining why a comanda refuses to close. If you want to understand it, you open the database and start pulling threads until something makes sense. That's exactly what I did — for months, on top of my actual job. The problem nobody wrote down I'm the Operations Manager at a restaurant in Itaúna, a mid-sized town in Minas Gerais, Brazil. I'm also the only person there who writes software. Not because I was hired to — because the restaurant runs on a Brazilian ERP called TronSoft, built on a Firebird database, and Firebird doesn't come with the kind of ecosystem you get around Postgres or MySQL. No Stack Overflow flood of answers. No official docs beyond a thin operator manual. Vendor support exists, but it's slow, and it doesn't scale to "I want to automate this specific internal workflow at 11pm on a Tuesday." So when I needed to automate payment reconciliation, close out comandas without touching the vendor's fragile UI, and trigger fiscal document emission (NFC-e) reliably, I didn't have a spec to follow. I had a live production database and a lot of curiosity. Learning a system by watching it think I started the way you'd expect: opening tables, guessing at relationships, breaking things in a test environment until I understood why they broke. Over time that turned into something more systematic — I ended up documenting 390 tables and 514 foreign keys across roughly 40 functional modules, entirely from observation. No vendor documentation, no source code access. Just structure, inference, and a lot of trial and error. Some of what I learned only reveals itself under pressure: Firebird's SQL dialect has its own quirks — FIRST 1 instead of LIMIT , for one. Small thing, but it breaks every query you copy-paste from a Postgres tutorial. Primary keys aren't auto-incrementing in the way you'd assume. They're driven by generators ( GEN_ID ), and if you write a record without syncing

"We don't have access to the data on the hardware/servers," Iron Mountain told Ars.

Meta released Glimmer this week, an open-weight AI model anyone can download and run on their own hardware — a contrast to Muse Spark, the company’s more powerful model that stays locked behind its own APIs. The release landed alongside a letter from Mark Zuckerberg arguing AI should be “for everyone” rather than controlled by a handful of labs, but as Equity’s […]

Last week we kicked off a series covering the (data/storage/network) announcements that, in my view, had the most impact. This week I'm bringing 5 more recent launches to run through the fine-tooth comb. Description, the pain point it solved, and my critical take on each one. 1. Aurora DSQL lands in five more regions What it is: Aurora DSQL, AWS's distributed, serverless SQL database, is now available in Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Singapore), Europe (Stockholm), and South America (São Paulo). With this launch, it now covers 19 regions total. Official link: https://aws.amazon.com/about-aws/whats-new/2026/05/amazon-aurora-dsql-five-additional-aws-regions/ How it used to be (the pain point): Until now, if you wanted to run Aurora DSQL with decent latency for users in Brazil, there was no regional option here. You either accepted the latency of hitting us-east-1 (an ugly phrase for a transactional application to hear) or gave up on DSQL entirely and went with a traditional Aurora PostgreSQL, purely for geographic proximity. My take: region expansion is the kind of announcement that isn't sexy, but it's what decides whether a service actually goes into production or stays a showcase item. With São Paulo on the list, the main adoption blocker for DSQL around here (latency) drops significantly. That said, it's worth remembering DSQL is still a relatively young product, with compatibility limitations against full PostgreSQL (extensions, certain data types, certain transactional features). Regional presence solves latency, it doesn't solve feature-set maturity. Those are two separate conversations. 2. Aurora DSQL gets a native PHP connector What it is: launch of the Aurora DSQL connector for PHP (PDO_PGSQL), which handles IAM authentication natively: generates a token per connection, guarantees the valid token gets used, and keeps full compatibility with the PDO_PGSQL that the PHP ecosystem already knows. It also brings retry with exponent

A massive new gas plant in Texas will be built with much less efficient technology than regular gas plants. It’s far from the only data center power project to rely on dirty turbines.

When a language-learning product says it teaches “the sounds of French,” one deceptively simple question appears immediately: how many sounds are there? There is no useful answer without first defining the job the inventory is meant to do. A phonological analysis, a pronunciation dictionary, a speech-recognition system, and a beginner curriculum can all model French sound structure differently without one of them necessarily being careless. They have different users, evidence, and failure costs. Our team encountered this while turning Parle's internal pronunciation inventory into a public CSV. We needed a list that could connect IPA symbols to French spelling patterns, example words, and short mouth cues for English-speaking beginners. We also needed to avoid presenting one product's learning model as the only correct account of French phonology. The result is a bounded dataset of 35 practical sound entries. This article explains the design decisions, the schema, and the limits we published with it. A teaching inventory is a model, not a census The International Phonetic Alphabet gives us a shared notation for describing speech sounds. It does not require every analyst or teacher to draw identical category boundaries for every language variety. The official IPA chart is a notation system; selecting a French inventory still requires linguistic and pedagogical decisions. Counts can change when an inventory treats any of the following differently: a contrast that is maintained by some speakers but merged by others; a marginal or loan sound that appears mainly in borrowed words; schwa, whose realization and deletion depend heavily on context and variety; a historical contrast that remains visible in spelling but not in every speaker's production; a phonetic realization versus a contrastive phoneme; a glide represented separately from its related vowel. For a curriculum, the important question is not “What number wins?” It is “What distinctions and cues help this audienc

Meta AI Research has introduced Muse Glimmer, a 30-billion-parameter open-weight model under the Apache 2.0 license, designed for local workflows. It enables autonomous agents and complex task execution on consumer GPUs without relying on cloud APIs. The model employs a multi-stage training approach for efficient performance and supports multimodal inputs, enhancing coding and automation tasks. By Olimpiu Pop

We all know the Big-O complexity of basic data structures. Arrays are O(n) for search. Hash maps are O(1). Linked lists are... well, complicated. But when I set out to build hashbrowns — a C++17 benchmarking suite comparing arrays, linked lists, and hash maps — I discovered that theory and practice are very different beasts. Here's what I learned building this project from scratch, and why you should probably benchmark before you optimize. 🎯 The Goal Was Simple (Ha!) I wanted a clean, educational project that would: Implement dynamic arrays, linked lists, and hash maps from scratch Benchmark insert, search, and remove operations Find the "crossover points" where one structure beats another Export everything to CSV for analysis Sounds straightforward, right? Four months later, I had written a custom memory tracker, implemented multiple hash map strategies, added statistical bootstrapping for confidence intervals, and learned more about CPU caches than I ever wanted to know. 📚 Lesson 1: Polymorphism Has a Price (But It's Worth It) My first architectural decision was creating a common DataStructure interface: class DataStructure { public: virtual void insert ( int key , const std :: string & value ) = 0 ; virtual bool search ( int key , std :: string & value ) const = 0 ; virtual bool remove ( int key ) = 0 ; virtual size_t memory_usage () const = 0 ; virtual std :: string type_name () const = 0 ; // ... }; This made benchmarking elegant — I could write generic code that tested any data structure: for ( auto & structure : structures ) { timer . start (); structure -> insert ( key , value ); timer . stop (); } But virtual function calls have overhead. In tight loops, that vtable lookup adds up. I spent a whole weekend convinced my hash map was slower than expected... until I realized I was measuring the cost of polymorphism, not the data structure itself. The fix? I kept the clean interface for the benchmarking harness but used templates internally where performance-cri

Analytical Insights: The Mechanisms Driving Career Growth in Data Engineering In the rapidly evolving field of data engineering, career progression is not merely a product of time served but a result of deliberate, adaptive strategies. A 7-8 year trajectory to a Senior Data Engineer role, marked by multiple successful contracts, underscores the critical role of adaptability and continuous learning. This analysis dissects the mechanisms that propel career growth, highlighting their interdependencies and the consequences of their neglect. 1. Continuous Learning and Skill Development Impact: The pace of technological advancement in data engineering demands constant upskilling. Internal Process: Engaging with new tools, methodologies, and industry trends through online courses, certifications, and hands-on practice ensures relevance. Observable Effect: Enhanced technical proficiency translates into the successful delivery of complex projects and the attainment of senior-level roles. Instability: Skill Stagnation occurs when learning efforts are inconsistent or outdated, leading to reduced competitiveness. This gap between current skills and industry demands can halt career progression, making individuals less attractive to employers seeking cutting-edge expertise. Intermediate Conclusion: Continuous learning is not optional; it is a survival mechanism in a field where obsolescence is a constant threat. 2. Client Relationship Management Impact: Diverse client needs and expectations across multiple contracts require tailored approaches. Internal Process: Implementing tailored communication strategies, proactively aligning project goals, and establishing iterative feedback loops foster trust and collaboration. Observable Effect: High client satisfaction leads to repeat contracts and positive referrals, which are critical for career advancement. Instability: Client Misalignment arises from inadequate communication or misunderstanding of client requirements, resulting in pro

Python's value comes not only from handling a great deal of data; its biggest asset comes from translating that data into meaningful business insight, and that business insight is used to make better business decisions. For businesses striving to increase customer satisfaction, enhance sales figures, and make smarter choices, a deep understanding of customer behavior is essential. Valuable business data includes customer transaction histories, website visits, product reviews, and responses to marketing efforts. When data such as this is analyzed, companies can effectively identify trends, understand preferences, and predict what their customers will do in the future. Python is the most popular when it comes to customer behavior analysis due to its comprehensive set of libraries, ranging from data cleaning, analysis, visualization, and machine learning; its flexibility makes it useful for new as well as seasoned data analysts. Why Analyze Customer Behavior? Customer behavior analysis assists businesses in answering key business questions such as: What are the products a customer buys most frequently? What spending figures do different customer groups have? Which customers are most likely to discontinue their service/products? What factors influence the customer's decision to purchase? Which marketing channels seem to receive the highest engagement? With answers like these, companies can implement targeted marketing campaigns, improve their product and services, customize experiences, and retain more customers. Key Python Libraries Some Python libraries that business data analysts use most frequently are: Pandas: Used for data cleaning, organizing, filtering, and manipulating datasets. NumPy: Provides a collection of high-level mathematical functions to perform numerical operations and work with arrays efficiently. Matplotlib: Enables users to create and plot static, animated, and interactive visualizations. Seaborn: An excellent library for plotting statistical graph

A contact table is easy to build. A usable CRM is an operations system. The difficult part is not storing a name and email address. It is preserving context as work moves between people, organizations, deals, tasks, notes, imports, reports, and follow-up. That requires deliberate data modeling and product decisions. Model relationships, not isolated records A customer may belong to an organization. An organization may have several contacts. A deal can involve multiple people, tasks, notes, and status changes. If those relationships are flattened into one spreadsheet-like table, duplicate data and contradictory updates appear quickly. Define stable identifiers and explicit relationships early. Treat activity history as a product feature Users need to know what changed, when it changed, and who changed it. Audit history supports troubleshooting and accountability. It also makes bulk operations safer: after an import or mass edit, an administrator should be able to understand the result rather than guessing which rows moved. Decide which actions deserve history, how long it is retained, and who can see it. Avoid collecting sensitive data simply because the schema allows it. Design imports for failure CSV import is where clean demos meet messy reality. A useful import flow should provide: column mapping; required-field validation; duplicate-handling rules; a preview before committing; clear row-level errors; an exportable error report; idempotent or recoverable behavior where practical. Never assume the first row contains perfect headers or that dates, phone numbers, and booleans use one format. Make views part of the workflow Saved filters and views let different roles focus on their work without changing the underlying data. A sales view may emphasize open deals and next actions. An operations view may emphasize overdue tasks. An administrator may need import history and permission context. This is more than UI convenience: it is a way to keep one shared system useful

By the end of this page you can look at a folder of eight finished sheets and say which two or three belong on the dashboard, which one goes in the upper-left corner, and which of Tableau's three sizing options to pick. You'll also have a one-sentence test that decides every one of those calls. It's about fifteen minutes. Here's the move to make today. Open your busiest dashboard and write the single question it answers, in one sentence, for one named person. Then remove every view that isn't part of answering it. Most people delete half, and the half that survives lands harder than the whole thing did. The short version: Tableau's own guidance is two or three views on a dashboard. Crowding is what happens when one dashboard is asked to serve several audiences at once. Where the surviving views sit is the second decision, and it has a known answer, so that gets the picture. The original carries a diagram here. In words: A single dashboard rectangle divided into three panes. One large pane occupies the whole upper-left area and spans most of the width. Two smaller panes sit below it, side by side. A curved arrow enters at the top-left corner of the large pane, travels right across it, then drops down and moves left to right across the two smaller panes, showing the order a reader takes them in. A small numeral one sits on the large pane, two and three on the smaller panes. The drawing shows that the first thing a reader meets is whatever occupies the upper left, so the most important view belongs there and the supporting views belong underneath. 1. Why two or three, and where that number comes from Before the explanation: you have eight finished sheets and one dashboard. How many of them would you put on it? Two or three. That's not a taste call, it's Tableau's published guidance: "In general, it's a good idea to limit the number of views you include in your dashboard to two or three." The reason is about attention rather than about screen space. A dashboard is read,