标签：#Data

One of the most common questions I hear is: "My API is slow. Where do I start?" The first instinct is usually: Upgrade the server Increase CPU Add more RAM But in many cases, the database query is the real bottleneck . Whenever I investigate a slow Laravel application, I follow the same checklist. It helps me identify performance issues before making unnecessary infrastructure changes. Let's go through it. 1️⃣ Find the Slow Queries First Don't start optimizing random queries. Start with the queries that are executed the most or take the most time. Useful tools: Laravel Telescope Laravel Debugbar (development) MySQL Slow Query Log Application Performance Monitoring (APM) You can't optimize what you haven't measured. 2️⃣ Stop Using SELECT * One of the easiest improvements. ❌ Instead of: SELECT * FROM users WHERE id = 10 ; Use: SELECT id , name , email FROM users WHERE id = 10 ; Why? Less data transferred Lower memory usage Faster response Easier for MySQL to use covering indexes Only fetch the columns your application actually needs. 3️⃣ Always Check the Execution Plan Before changing anything, run: EXPLAIN SELECT id , name FROM users WHERE email = 'john@example.com' ; Things I usually look for: Is MySQL scanning the whole table? Is an index being used? How many rows are examined? Is there a temporary table? Is filesort being used? EXPLAIN often tells you exactly why a query is slow. 4️⃣ Verify Your Indexes Indexes are one of the biggest performance improvements you can make—but only when they match your queries. Example: SELECT * FROM orders WHERE customer_id = 100 ; Create an index: CREATE INDEX idx_customer_id ON orders ( customer_id ); Now MySQL can jump directly to the matching rows instead of scanning the entire table. 5️⃣ Look for Composite Index Opportunities Suppose your query is: SELECT id , total FROM orders WHERE customer_id = 10 AND status = 'paid' ; Instead of two separate indexes: customer_id status A composite index is often better: CREATE INDEX idx_cu

7 MongoDB Query Mistakes That Return the Wrong Results VisuaLeaf VisuaLeaf VisuaLeaf Follow Jul 14 7 MongoDB Query Mistakes That Return the Wrong Results # mongodb # coding # software # database 3 reactions Add Comment 5 min read

MongoDB queries look simple. You type a field, give it a value, hit run, and you get your data back. But just because a query runs without throwing an error doesn't mean it worked right. Sometimes you get a blank screen. Sometimes you get way too many records. Other times, the data looks fine at first glance, but it doesn't actually match what you asked for. Most of these slip-ups happen for one basic reason: the query structure doesn't match the way the data actually sits in the database. To show you what we mean, we’ll use a clinic database with a collection called visits . Here is what a typical document looks like: JSON { "_id": "6871b6f9c3f1d1a4c2a10001", "status": "completed", "visitDate": "2026-07-01T09:30:00.000Z", "patient": { "name": "Anna Keller", "age": 34 }, "doctor": { "name": "Dr. James Carter", "specialty": "Cardiology" }, "symptoms": ["cough", "fever"], "prescriptions": [ { "name": "Ibuprofen", "active": false }, { "name": "Paracetamol", "active": true } ], "invoice": { "paid": true, "method": "card", "total": 250 } } You can run these examples right in the VisuaLeaf MongoDB Shell . Using visual tools makes a big difference because you can see exactly what MongoDB is returning in real time. 1. Forgetting the Curly Braces This is just a quick typo, but it breaks things right away. The Mistake: db . visits . find ( status : " completed " ) The Correct Query The find() tool always expects an object. Even if you are only looking for one specific thing, you still need to wrap that condition in curly braces {} . 2. Treating $or Like a Regular Object This one trips a lot of people up because the broken version looks like it should work. The Mistake: db.visits.find({ $or: { status: "completed", "invoice.paid": false } }) What is wrong: $or expects an array of conditions, but this query gives it one object. The error will usually be something like: MongoServerError: $or must be an array The Correct Query The first query is wrong because $or needs an array, n

Drawing from the enduring adaptability of HTML and HTTP, Seph Gentle proposes embedding self-contained schemas directly into file headers, ensuring data remains readable without external definitions. His experimental format prioritises forward, backwards, and sideways compatibility, enabling data format evolution without central coordination or data loss By Olimpiu Pop

TL;DR Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this model by running the data plane inside the customer’s cloud while the vendor manages the control plane. This keeps data within the enterprise perimeter while still providing a managed platform. Condense builds on this model with AI-driven automation, unified monitoring, and marketplace deployment, enabling private, compliant, and cost-efficient real-time data streaming. The enterprise data landscape is currently defined by a conflict between real-time AI data streaming utility and the strict requirements of data sovereignty . For years, the standard SaaS model forced a compromise. To access premium analytics, companies had to export sensitive telemetry to a vendor cloud. This created massive cloud egress costs and introduced significant security vulnerabilities. Bring Your Own Cloud (BYOC) for data streaming platforms has emerged as the professional solution to this dilemma. It allows a business to keep data within its own perimeter while benefiting from a fully managed, high-performance ecosystem. The BYOC Architecture: Privacy by Design An experienced analyst views BYOC as a clean separation of concerns. The architecture splits the environment into two distinct layers to ensure raw data never leaves the authorized environment. SaaS Control Plane: This is the management layer hosted by the provider. It handles the brain of the operation. It manages orchestration, user access, and pipeline configuration without ever seeing the actual data packets. Private Data Plane: This is the muscle. The managed Kafka clusters , Kubernetes (K8s) nodes, and storage engines like ClickHouse live inside the customer Virtual Private Cloud (VPC) . By keeping the data plane inside the customer perimeter, telemetry collection remains private. This architecture is the most direct path to satisfying internal security audits and global regulatory

DuckDB Iceberg MERGE, PostgreSQL GUCs, SQLite Optimization Checklist Today's Highlights This week's highlights include powerful new Iceberg data manipulation features in DuckDB v1.5.3 and a deep dive into an obscure PostgreSQL GUC. Plus, the SQLite community discusses a practical optimization checklist for embedded databases. New DuckDB-Iceberg Features in v1.5.3 (DuckDB Blog) Source: https://duckdb.org/2026/05/29/new-iceberg-features.html The latest DuckDB v1.5.3 release significantly enhances its integration with Apache Iceberg, introducing a suite of powerful new features for data engineers and analysts. Key among these are the support for MERGE INTO and ALTER TABLE statements, allowing for more robust data manipulation directly within DuckDB for Iceberg tables. This update enables complex operations like upserting data based on conditions, schema evolution (e.g., adding/dropping columns), and modifying table properties, all achievable through a familiar SQL environment. This capability is crucial for maintaining data integrity and adapting schemas without complex external tooling. Furthermore, DuckDB-Iceberg now supports partition transforms, making it easier to manage and query partitioned Iceberg datasets efficiently by defining how data is distributed across files. The release also brings support for Iceberg V3, ensuring compatibility with the latest features of the Iceberg format, including new manifest list and manifest file layouts which offer performance improvements. These additions position DuckDB as an even stronger tool for building performant data pipelines and performing complex analytics directly on large-scale Iceberg data lakes, fully leveraging DuckDB's in-process analytical capabilities and the flexibility of the Iceberg table format. Comment: This update is a game-changer for working with Iceberg tables directly in DuckDB. MERGE INTO support means simplified ETL for incremental loads, and V3 compatibility ensures we're ready for future Iceberg

The ScyllaDB PHP driver is not a C++ extension anymore. As of 1.4.0 it's pure C23, the ZendCPP template layer we leaned on for the object embed and allocate pattern is deleted, and the build no longer needs a C++ compiler at all. Every hand-written .cpp file is a .c file now (71 of them), the descriptor generator emits .c , and CMake builds with c_std_23 and nothing else. That's the biggest change to how this extension is built since we forked it for PHP 8.0. This is also the release where a plan I opened back in 2023 finally landed. PR #50 laid it out: rewrite the src/Cluster directory to be more maintainable, use Zend Fast Argument Parsing, remove some memory allocations, and add .stub.php files that generate the C headers so nobody has to hand-maintain Zend arginfo by hand. 1.4.0 is that plan finished, and a lot more that grew out of it. The things you'll actually feel: persistent session connect() doesn't allocate a 200-character key string on every call anymore, and the minimum PHP is now 8.3 (8.2 is gone). Nothing in your application code changes, this is almost all under the surface. The .stub.php Build The idea from PR #50 was small. Instead of writing ZEND_BEGIN_ARG_INFO_EX blocks by hand and keeping them in sync with the actual method bodies, write the signature once in a .stub.php file and generate the C arginfo from it. In v1.4.0 that's the whole build. There are 75 .stub.php files now, and each one is just the PHP signature of the class: // src/Keyspace.stub.php interface Keyspace { public function name (): string ; public function replicationClassName (): string ; /** @return array<string, mixed> */ public function replicationOptions (): array ; public function hasDurableWrites (): bool ; /** @return Table|false */ public function table ( string $name ): Table | false ; public function aggregate ( string $name , mixed ... $types ): Aggregate | false ; } At build time CMake runs gen_stub.php (vendored from PHP 8.5's build/gen_stub.php , with two small p

Apple would not comment on the "security breach," which allegedly allowed a former employee to download sensitive files from Apple's network long after he departed the company for rival OpenAI.

How I Built CodeGraph: A Living Knowledge Graph That Tells You What Breaks Before You Break It Built for HACKHAZARDS '26 — powered by Neo4j AuraDB, tree-sitter, Groq LLaMA, and Next.js The Problem That Frustrated Me Every developer knows this feeling. You join a new codebase. There are 50,000 lines of code. Your manager says "just fix this small bug in the authentication module." You make the change. You push. And suddenly three completely unrelated features are broken — a payment flow, a notification system, and a dashboard widget you've never even looked at. You spend the next four hours tracing function calls manually, reading code you've never seen, trying to understand why changing one function in auth.py broke something in notifications.py on the other side of the codebase. This is not a rare experience. According to JetBrains' developer survey, engineers spend 58% of their time reading and understanding code — not writing it. One wrong change in a large codebase can cost hours of debugging, failed deployments, and frustrated users. I built CodeGraph to solve this. Not with another AI chatbot that guesses at your code. With a real, queryable knowledge graph that actually understands how your codebase is connected. What CodeGraph Does CodeGraph takes any public GitHub repository URL and within seconds: Parses every function in the codebase using tree-sitter Maps every call relationship between functions as a directed graph Stores everything in Neo4j AuraDB as a live knowledge graph Lets you ask questions in plain English — answered by AI grounded in real graph data The result: paste a GitHub URL, see your entire codebase as an interactive graph, click any function, and instantly know what breaks if you change it. The Tech Stack Here's what I used and why each choice mattered: Backend: Python + FastAPI (REST API server) Neo4j AuraDB (graph database — the core of everything) tree-sitter (AST parser for Python, JS, TS, TSX) Groq API with LLaMA 3.3 70B (free-tier L

Responding to Musk accusing him of being a scammer, Altman said, "homeboy you're the one sellling [sic] public market investors on short-term space datacenters."

The Cybersecurity and Infrastructure Security Agency (CISA) has issued a postmortem on a data leak in which a contractor published dozens of internal CISA credentials -- including AWS Govcloud keys -- in a public GitHub repository for almost six months before being notified by KrebsOnSecurity. Experts say the gaps identified in the agency's initial response provide important lessons that all security teams should absorb.

A panel on data ownership challenged the definition of "ownership," arguing it must extend beyond simple account control to include structural independence, interoperability, and community governance. Speakers like Zenna Fiscella, Paul Frazee, Boris Mann, and Robin Berjon emphasised the need for shared standards, unbundled platforms, and better tools to support user sovereignty. By Olimpiu Pop

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Hackathons are intense. You've got a brilliant idea, a tight deadline, and often, limited sleep. The last thing you want is to spend half your precious time wrestling with database boilerplate, ORM setup, or SQL query syntax. This guide will walk you through building a functional Node.js backend for your hackathon project, focusing on speed and minimal friction, so you can spend more time on your core idea. The Hackathon Backend Challenge Typically, setting up a database and its interaction layer involves several steps: Schema Definition: Deciding on tables/collections, fields, types, and relationships. ORM/Driver Setup: Installing and configuring your database driver or ORM (e.g., Mongoose, Sequelize). Model Creation: Translating your schema into code, often with verbose syntax. Query Writing: Crafting SELECT , INSERT , UPDATE , DELETE statements or ORM methods for every data operation. Debugging: Fixing typos, schema mismatches, and complex join logic. This process, while fundamental, eats up valuable time that could be spent on features, UI, or even sleep. For a hackathon, you need to iterate rapidly, and database interactions should be the least of your worries. Strategy 1: Embrace Simplicity For many hackathon projects, you don't need highly optimized, production-grade queries from day one. You need functional queries that work quickly. Focus on getting data in and out reliably. Strategy 2: Natural Language for Data Modeling Instead of writing verbose schema definitions, think about how you'd describe your data to a non-technical person. For example, if you're building a task management app, you might say: "We need a collection of tasks. Each task has a title, a description, a due date, and a status (like 'pending' or 'completed'). Each task belongs to one user." This natural language description contains all the essential information for a data model, including relationships and field types. Strategy 3: Expressive Querying Similarly, when you need to fetch dat

Jake Lazaroff discussed the AT Protocol as a framework for distributed applications beyond social networking. He emphasised a local-first architecture where users maintain data in PDSs while leveraging shared infrastructure for synchronisation and updates. The presentation included experiments showcasing collaborative tools and highlighted the benefits of reduced reliance on app-specific backends. By Olimpiu Pop

Introdução Validar dados é uma responsabilidade que pode ficar na aplicação, no banco de dados, ou em ambos. Deixar tudo na aplicação é arriscado: diferentes sistemas podem acessar o mesmo banco, migrações podem rodar diretamente, um bug pode deixar passar um valor inválido. Constraints são regras definidas no próprio banco de dados — uma camada de proteção que age independente de quem está escrevendo os dados. PRIMARY KEY A chave primária identifica cada linha de forma única. Ela combina duas restrições implicitamente: NOT NULL e UNIQUE . Nenhuma linha pode ter o mesmo valor de chave primária, e nenhuma pode tê-la nula. CREATE TABLE clientes ( id INT PRIMARY KEY , nome VARCHAR ( 100 ) NOT NULL ); Quando a chave primária envolve mais de uma coluna, ela é declarada separadamente: CREATE TABLE matriculas ( aluno_id INT , curso_id INT , PRIMARY KEY ( aluno_id , curso_id ) ); Na maioria dos bancos, é comum usar uma chave primária auto-incremental para não precisar gerenciar os IDs manualmente: -- PostgreSQL id SERIAL PRIMARY KEY -- MySQL id INT AUTO_INCREMENTPRIMARY KEY -- SQL padrão (suportado por ambos) id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY FOREIGN KEY A chave estrangeira garante integridade referencial : um valor só pode existir numa coluna se ele existir como chave primária na tabela referenciada. É o que torna os relacionamentos entre tabelas confiáveis. CREATE TABLE pedidos ( id INT PRIMARY KEY , cliente_idINT REFERENCES clientes ( id ) ); Tentar inserir um pedido com cliente_id = 99 quando não existe cliente com esse id resulta em erro imediato. O banco rejeita a operação antes mesmo de ela chegar ao disco. O comportamento quando o registro referenciado é deletado pode ser configurado: CREATE TABLE pedidos ( id INT PRIMARY KEY , cliente_id INT REFERENCES clientes ( id ) ON DELETE CASCADE -- deleta os pedidos junto com o cliente ON UPDATE CASCADE -- atualiza o cliente_id se o id do cliente mudar ); As opções disponíveis são: Opção Comportamento RESTRICT

Introdução Consultas individuais respondem perguntas como "qual o email do cliente 42?". Mas as perguntas mais valiosas em qualquer sistema são de outro tipo: "qual o produto mais vendido?", "qual a receita média por pedido?", "quantos clientes se cadastraram esse mês?". Para responder isso, o SQL oferece as funções de agregação — operações que recebem um conjunto de linhas e devolvem um único valor resumido. Para os exemplos a seguir, considere esta tabela: pedidos: | id | cliente | produto | categoria | quantidade | valor | |----|------------|-------------|--------------|------------|--------| | 1 | Ana Lima | Notebook | Eletrônicos | 1 | 3500.00| | 2 | Ana Lima | Mouse | Periféricos | 2 | 80.00| | 3 | Bruno Melo | Teclado | Periféricos | 1 | 150.00| | 4 | Bruno Melo | Notebook | Eletrônicos | 1 | 3500.00| | 5 | Carla Nunes| Monitor | Eletrônicos | 2 | 1200.00| | 6 | Carla Nunes| Mouse | Periféricos | 1 | 80.00| As Funções de Agregação COUNT Conta o número de linhas — ou de valores não nulos em uma coluna específica. -- Total de pedidos SELECT COUNT ( * ) AS total_pedidosFROM pedidos ; -- Resultado: 6 -- Clientes distintos que fizeram pedidos SELECT COUNT ( DISTINCT cliente ) AS clientes_unicosFROM pedidos ; -- Resultado: 3 COUNT(*) conta todas as linhas, incluindo as que têm nulos. COUNT(coluna) conta apenas as linhas onde aquela coluna não é nula. COUNT(DISTINCT coluna) conta valores únicos — útil para saber quantos clientes, produtos ou categorias distintos aparecem no resultado. SUM Soma os valores de uma coluna numérica. -- Receita total SELECT SUM ( valor ) AS receita_total FROM pedidos ; -- Resultado: 8510.00 -- Total de itens vendidos SELECT SUM ( quantidade ) AS itens_vendidos FROM pedidos ; -- Resultado: 8 AVG Calcula a média aritmética dos valores. -- Valor médio por pedido SELECT AVG ( valor ) AS ticket_medio FROM pedidos ; -- Resultado: 1418.33 AVG ignora valores nulos automaticamente — calcula a média apenas sobre os registros que têm valor preenchid

Cargar 112.647 filas en una tabla de hechos debería tardar segundos. A mí me tardaba más de 25 minutos, y acababa cancelando la query. Los datos estaban bien, el SQL estaba bien, las dimensiones se poblaban sin problema. El culpable era otro, y descubrirlo fue la parte más instructiva de todo el proyecto. Todo esto surgió construyendo un Data Warehouse en estrella sobre datos reales de e-commerce: no una tabla bonita para hacer un SELECT * , sino un modelo dimensional completo, reproducible desde cero, capaz de responder preguntas de negocio de verdad. El dataset Trabajé con el Brazilian E-Commerce Public Dataset by Olist : pedidos reales de un marketplace brasileño entre septiembre de 2016 y octubre de 2018. Son 9 CSV relacionados entre sí: 99.441 pedidos y 112.650 líneas de venta 103.886 pagos y 104.719 reseñas 32.951 productos, 3.095 vendedores 1.000.163 registros de geolocalización Y con trampas de datos reales que hay que ver antes de que te muerdan: Un pedido puede tener varios pagos y varias reseñas. Si los unes tal cual a la tabla de hechos, duplicas ventas . Es el error clásico y silencioso: los totales salen inflados y nadie se entera. customer_id no es un cliente. Olist crea uno por cada pedido; la persona real es customer_unique_id . Contar mal aquí te cambia el KPI: hay 99.441 cuentas frente a 96.096 personas. El CSV de productos trae una errata en la cabecera ( product_name_lenght , con "lenght"). Si tu esquema la escribe bien y cargas por interfaz gráfica (que empareja por nombre ), esas columnas se quedan vacías sin que nadie avise. El proceso Monté una arquitectura en capas: CSV → staging → modelo dimensional → vistas → análisis , todo en cuatro scripts ejecutables en orden y idempotentes (el esquema se recrea desde cero, se puede relanzar mil veces). El modelo es un star schema : una tabla de hechos fact_sales al grano de línea de producto dentro de un pedido , y cinco dimensiones (cliente, producto, vendedor, pago y fecha), con claves sustitutas,

No hacking — a passive scan only looks at what your browser already downloads when it opens a page. Here's what I found: 6 of 15 load their Supabase database directly client-side. The public API key sits in the page source. That's fine if Row-Level Security is configured right — but it's one wrong setting away from "anyone can read the whole table." 14 of 15 ship no Content-Security-Policy — a simple, high-value hardening against script injection, almost always missing. Is this theoretical? No. Two apps I audited with the owner's permission: A social app: the profiles table — user names, cities, and a password hash — readable by a logged-out stranger. Closed in an afternoon. A paid learning app: 155 paid study sheets and 4,872 answers were readable by anyone, with no account and no subscription — its entire paid catalogue, a single API call away. The paywall lived only in the front-end; the database served everything to everyone. Loading Supabase in the browser isn't the mistake. Not enforcing access in the database (RLS) is. And the tools you build with won't tell you — they'll happily ship it. If you built something on Lovable / Bolt / Replit with real users (or paying ones), it's worth 60 seconds to check what a stranger can already see. I made a free tool that runs the surface check (passive, no signup): sealdy.dev Happy to answer questions on how RLS leaks happen and how to lock them down.

If you’ve ever struggled with CALCULATE() or wondered why SUMX() behaves differently from SUM() , this guide is for you. DAX (Data Analysis Expressions) is the language that powers Power BI , Analysis Services , and Power Pivot — enabling dynamic calculations, filtering, and time intelligence. Below is a categorized cheat sheet of essential DAX functions , plus examples showing how to use each in real-world Power BI scenarios. Filtering & Context These functions control how filters are applied and evaluated in your calculations. Function Example Description CALCULATE() CALCULATE(SUM(Sales[Amount]), Region[Name] = "Nairobi") Changes filter context to calculate total sales for Nairobi. FILTER() FILTER(Sales, Sales[Amount] > 10000) Returns a table filtered by condition. ALL() CALCULATE(SUM(Sales[Amount]), ALL(Region)) Ignores filters on Region. REMOVEFILTERS() CALCULATE(SUM(Sales[Amount]), REMOVEFILTERS(Region)) Removes filters from Region. VALUES() VALUES(Customer[City]) Returns unique list of cities. SELECTEDVALUE() SELECTEDVALUE(Product[Category], "All") Returns selected category or “All” if none. TREATAS() TREATAS(VALUES(Temp[City]), Customer[City]) Applies one table’s values as filters on another. KEEPFILTERS() CALCULATE(SUM(Sales[Amount]), KEEPFILTERS(Product[Category] = "Electronics")) Keeps existing filters and adds new ones. ALLSELECTED() CALCULATE(SUM(Sales[Amount]), ALLSELECTED(Region)) Respects user selections in visuals. ALLEXCEPT() CALCULATE(SUM(Sales[Amount]), ALLEXCEPT(Sales, Sales[Year])) Removes all filters except Year. Aggregation Summarize or aggregate data across rows or columns. Function Example Description SUM() SUM(Sales[Amount]) Adds all sales amounts. AVERAGE() AVERAGE(Sales[Amount]) Calculates mean value. COUNT() COUNT(Customer[ID]) Counts non-blank entries. COUNTROWS() COUNTROWS(Sales) Counts rows in a table. DISTINCTCOUNT() DISTINCTCOUNT(Customer[ID]) Counts unique customers. MIN() MIN(Sales[Amount]) Finds smallest sale. MAX() MAX(Sales[Amo