标签：#data

lakehouse has two storage areas ; Files and Tables Files Store structured, queryable data by sql Supports schema definitions and ACID transactions Tables Stores Raw or semi-structured data(CSV, parquet, JSON) No schema support Flexible for data explorations Schema allows for logical ordering of data on business functions or domain (sales,marketing etc) A dbo schema is enabled by default once a lakehouse is created Schema-enabled lakehouses also support schema-level permissions and cross-workspace queries using the four-part namespace Lakehouse mode : Lakehouse Explorer and SQL analytics endpoint Lakehouse Explorer: Allows managing, Update, create, upload of data.You can switch between tables in the lakehouse SQL anlytics endpoit : Does not allow modifying of the underlying data. You can query using TSQL at read only mode. Loading data into lakehouse: Upload data into files/ folders on the explorer Load into delta tables (no code) Transform using power query in dataflow gen2 INgest into notebooks using apache spark (programmatically) Use Copy data to move data into differnt sources using data factory pipelines -Shortcuts allow you to reference external data reducing copies. Access is managed by One Lake. Schema shortcuts map an entire schema to a folder of Delta tables in another lakehouse. SQL analytics endpoint provides read-only access to lakehouse tables using T-SQL queries. SQL USE CASES : adhoc queries, BI connections to power bi or azure data studio, Data validation You can use SQL views to store reusable query logic. Views are useful when you need to apply business rules, simplify complex joins, or provide curated data for downstream consumers. You can use Spark SQL for SQL-like queries or PySpark for programmatic data manipulation in Notebooks. Spark SQL works well for familiar SQL patterns. PySpark provides greater flexibility for complex transformations and integration with Python libraries. Power BI is the business intelligence and reporting layer in Fabr

Data Visualizer Live Demo 🌐 Try it live: https://datavisualizer.urlmediainspector.dev/ What It Is Data Visualizer is a visual workspace where developers can explore, transform, execute, and understand data using interconnected nodes on an infinite canvas. Instead of jumping between API tools, JSON viewers, spreadsheets, code editors, schema inspectors, and visualization platforms, everything happens inside a single interactive environment. Each node represents a specific capability and can be connected together to create powerful workflows for data exploration, processing, automation, and analysis. Key Features Infinite Visual Workspace Work on an unlimited canvas where data, code, APIs, documents, and visualizations can be organized as connected workflows instead of isolated files and tabs. API Exploration Connect to APIs, inspect responses, analyze payloads, and build reusable visual pipelines for data processing. JSON & YAML Visualization Navigate deeply nested structures through interactive visual representations that make complex data easier to understand. JavaScript & TypeScript Execution Run JavaScript and TypeScript directly inside workflow nodes to transform, filter, and manipulate data in real time. Browser-Based Python Runtime Execute real Python entirely in the browser without requiring local installations or external servers. CSV & Dataset Analysis Import and explore tabular data visually, making it easier to inspect records, understand relationships, and process large datasets. Schema Exploration Visualize schemas and nested structures to quickly understand how data is organized and connected. PDF, Image & Video Support Work with documents and media assets directly inside the workspace without constantly switching applications. Visual Data Pipelines Create workflows by connecting nodes together, allowing data to flow naturally between APIs, transformations, code execution, schemas, and visualizations. Interactive Data Transformation Modify and reshape

DuckLake Spec, pg_background 2.0, and pgsql_tweaks 1.0.3 Enhance Database Ecosystem Today's Highlights This week's highlights include DuckDB's new DuckLake specification for simplified dataframe integration with data lakes, alongside key updates from the PostgreSQL community. We cover pg_background 2.0 for safer asynchronous SQL execution and the release of pgsql_tweaks 1.0.3 for enhanced monitoring and performance tuning. The DuckLake Spec Is so Simple, Even a Clanker Can Build One for Dataframes (DuckDB Blog) Source: https://duckdb.org/2026/05/04/ducklake-dataframe.html The DuckDB team has unveiled the DuckLake v1.0 specification, a significant step towards simplifying data lake interactions with dataframes. This specification aims to provide a robust yet straightforward framework for reading and writing dataframes directly from and to data lake storage, emphasizing ease of implementation. The announcement highlights the specification's simplicity, so much so that even AI can be leveraged to generate compatible dataframe reader/writer tools. This initiative promises to democratize data lake access, allowing developers and data engineers to integrate DuckDB's powerful analytical capabilities with their data lake architectures more seamlessly. By defining a clear standard, DuckLake facilitates the creation of a vibrant ecosystem of tools and connectors, enabling efficient data processing directly within the data lake context without complex ETL pipelines. This development positions DuckDB as an even more versatile tool for analytical workloads, bridging the gap between local data processing and large-scale data lake environments. The ability to easily build data lake connectors, potentially even with AI assistance, marks a notable shift towards more accessible and integrated data workflows. This could streamline operations for data scientists and analysts who frequently work with large datasets stored in various data lake formats, allowing them to leverage DuckDB's

Microsoft shut down dozens of GitHub code repositories for Azure and AI coding tools after a reported hack.

If you work with Brazilian companies — as an accountant, credit analyst, or anyone processing PJ clients at scale — here's a practical automation approach using free public data. What you can verify automatically For any CNPJ, public data gives you: Situação cadastral : ATIVA, BAIXADA, INAPTA, SUSPENSA — critical for invoice validation Razão social : legal name for contract matching CNAE : is this company allowed to do what they claim? QSA : who are the actual partners/directors? Data abertura : how old is the company? The data 65M+ CNPJs from Receita Federal, indexed and searchable at Jurídico Online . Free. Also available as a Python package: pip install juridico-online from juridico_online import empresa_url , buscar_url # Get company page URL for a CNPJ url = empresa_url ( " 00.000.000/0001-91 " ) print ( url ) # https://juridicoonline.com.br/empresa/00000000000191 # Search by company or partner name search = buscar_url ( " Magazine Luiza " ) print ( search ) Checks worth automating 1. Situação ATIVA before accepting any invoice INAPTA or BAIXADA companies cannot legally issue NF-e. 2. CNAE vs service being billed A company with CNAE "comércio de alimentos" billing for software development is a red flag. 3. Company age vs contract value A 3-month-old company offering a R$500k contract deserves extra scrutiny. 4. Shared partners across suppliers If two suppliers share directors, that's a conflict of interest. Search partner names at juridicoonline.com.br to see all companies they control. Integration patterns ERP/AP : validate CNPJ status before releasing payment Onboarding : auto-fill razão social when client enters CNPJ Batch audit : cross-check your vendor list quarterly Monitoring : alert if a key supplier's CNPJ changes status The data is public, free, and updated regularly. No excuse to check manually at scale.

Short answer first: use Supabase if you want Postgres plus auth, realtime, storage, and a dashboard as one managed bundle. Self-host Postgres – or use a managed Postgres – if you mostly need a database and your app already handles its own auth and logic. The choice is not really "Postgres vs Supabase". It's whether you need the extra layers Supabase puts on top of Postgres. Supabase is not a database Supabase runs on PostgreSQL, but it's a stack of services around it: Postgres – the actual database Auth – user signup, login, JWT tokens Realtime – live updates over websockets Storage – an S3-style file store Edge Functions – serverless functions Studio – dashboard + auto-generated REST/GraphQL API So "self-host Postgres or use Supabase" compares a plain database to a full backend. The honest question: do you need those extra layers, or just the database underneath them? A quick test: You use Supabase Auth, Storage, and Realtime → Supabase earns its place. You use one of them → it's replaceable. You use none and treat it as "Postgres with a nice dashboard" → you want plain Postgres. Side-by-side comparison Factor Supabase (managed) Self-hosted Supabase Plain Postgres (managed or self-hosted) Database engine PostgreSQL PostgreSQL PostgreSQL Built-in auth Yes Yes No (bring your own) Realtime / websockets Yes Yes No File storage Yes Yes No Dashboard + auto API Yes Yes No (use any SQL client) Backups Managed (limits by plan) You manage Managed or you manage Cost shape Metered, grows with usage Server cost + your time Database only Self-host effort None High (many containers) Low–medium Lock-in Medium–high Medium Very low The lock-in point decides it for many teams. Your data is standard Postgres in every option ( pg_dump portable). The lock-in is everything else: Auth tokens, Storage paths, Supabase-specific RLS policies, Edge Function code. The more Supabase-specific features you adopt, the harder the exit. When each option wins Pick managed Supabase when: You're startin

The bill is expected to blanket-ban companies and startups from selling people's precise location data across the state.

Google says Gemma 4 12B is "designed to bring agentic, multimodal intelligence directly to your laptop", further noting that the new model can be combined with Google AI Edge to "build and experiment locally, on everyday machines". This integration allows for a wide range of capabilities, from autonomous data processing to generating visual insights and even building webpages or executing tools. By Sergio De Simone

The era of passive data analytics is over. Today, the most forward-thinking data teams aren't just...

In this article, the author examines how AI is transforming phishing from a manual, targeted activity into an automated and scalable attack model. The article breaks down each stage of the phishing lifecycle, showing how AI improves reconnaissance, profiling, content generation, delivery, and interaction, while outlining layered defenses that combine controls, processes, and user awareness. By Marco Rizzi

Senior Solution Architect Viktor Vedmich shares how engineering leaders can maximize application performance using Valkey. He discusses the open-source Redis fork's 100% API compatibility, explores advanced caching strategies like lazy loading, and explains how to implement powerful data structures for real-time analytics, rate limiting, and session stores to solve the thundering herd problem. By Viktor Vedmich

CSV files are one of the most common formats for storing and exchanging data. Whether you’re working with logs, analytics data, application exports, or reports, there will likely come a time when you need to load CSV data into ClickHouse®. The good news is that ClickHouse® makes CSV ingestion straightforward and efficient. In this guide, you’ll learn how to create a table, prepare a CSV file, load CSV data into ClickHouse®, and verify that the data has been imported successfully. Why Use CSV Files with ClickHouse®? CSV (Comma-Separated Values) files are simple, portable, and supported by virtually every data platform. Common use cases include: Importing exported application data Loading historical datasets Migrating data from other databases Testing analytics workloads Sharing data between systems Because ClickHouse® is designed for high-performance analytics, it can efficiently process and query large CSV datasets once they are loaded into a table. Sample CSV File Let’s assume we have a file named employees.csv with the following contents: id,name,department,salary 1,Alice,Engineering,75000 2,Bob,Marketing,60000 3,Charlie,Finance,70000 This simple dataset will help demonstrate how to load CSV data into ClickHouse®. Step 1: Create a Table in ClickHouse® Before importing data, create a table that matches the structure of the CSV file. CREATE TABLE employees ( id UInt32, name String, department String, salary UInt32 ) ENGINE = MergeTree() ORDER BY id; This table contains four columns that correspond directly to the columns in our CSV file. Step 2: Load CSV Data into ClickHouse® There are several ways to import CSV data, but one of the most common methods is using the ClickHouse® client. Run the following command: clickhouse-client --query=" INSERT INTO employees FORMAT CSVWithNames" < employees.csv The CSVWithNames format tells ClickHouse® that the first row contains column headers. After executing the command, ClickHouse® will read the CSV file and insert the records

​We just deployed a major update to our SQL Server web sandbox, moving from an individual tool to a collaborative environment: ​✅ Group Rooms (Beta): Create private rooms using security tokens to chat and write code together in real time. ✅ Simulated Linked Servers (Beta): Connect with your friends' sandboxes to run cross-queries strictly in read-only mode to preserve performance. ✅ Gamification & XP: Earn experience points and unlock technical badges as you run queries or build database schemas. ✅ Social Layer: Manage your friends list and customize your public developer profile to showcase your achievements. ​Currently, all features are completely free to use during this phase of the platform. Try it out with your colleagues at tsqlsandbox.online and leave your feedback below!

PostgreSQL Error 2200D: invalid escape octet The 2200D: invalid escape octet error occurs in PostgreSQL when a bytea value contains an invalid escape sequence. This typically happens with the legacy escape format for binary data, where octet values must be represented as three-digit octal numbers in the range \000 to \377 . If the escape sequence falls outside this range or uses non-octal digits, PostgreSQL raises this error immediately. Top 3 Causes 1. Out-of-range octal values in bytea escape literals The bytea escape format only accepts octal values from \000 to \377 (decimal 0–255). Using values like \400 or non-octal digits like \9 will trigger this error. -- BAD: \400 exceeds valid octal range (max is \377) SELECT E ' \\ 400' :: bytea ; -- ERROR: invalid escape octet -- BAD: \9 is not a valid octal digit SELECT E ' \\ 9AB' :: bytea ; -- ERROR: invalid escape octet -- GOOD: valid octal escape sequences SELECT E ' \\ 377' :: bytea ; -- decimal 255 SELECT E ' \\ 101' :: bytea ; -- 'A' character SELECT E ' \\ 000' :: bytea ; -- null byte 2. Using escape format strings with hex output format Since PostgreSQL 9.0, the default bytea_output is hex . Applications that mix hex -format output back into escape -format input processing can generate malformed escape sequences. -- Check current bytea output format SHOW bytea_output ; -- GOOD: Use hex format (recommended for all new projects) SELECT ' \x DEADBEEF' :: bytea ; SELECT ' \x 48656C6C6F' :: bytea ; -- 'Hello' -- GOOD: Use encode/decode for safe conversions SELECT encode ( ' \x DEADBEEF' :: bytea , 'hex' ); -- output as hex string SELECT encode ( ' \x DEADBEEF' :: bytea , 'base64' ); -- output as base64 SELECT decode ( 'deadbeef' , 'hex' ); -- hex string → bytea SELECT decode ( 'SGVsbG8=' , 'base64' ); -- base64 → bytea 3. Incorrect escaping during data migration or manual SQL When migrating binary data from other databases (Oracle, MySQL) or writing raw INSERT statements manually, developers often confuse octal and

Introduction Excel is a powerful spreadsheet program developed by Microsoft that is used to calculate, organize and analyze data. It provides a way of turning raw data into meaningful insights through handling large datasets more efficiently from tracking sales and expenses to analyzing trends. Various Excel applications. Decision making: One of the major ways Excel is used in real-world data analysis is to support decision making. Companies collect large volumes of raw data everyday ranging from customer information, sales records to log records. This data is organized and cleaned by Excel into tables, charts and reports making it easier to derive insights and identify trends that help in decision making. Financial reporting: Excel is also widely used for financial reporting and budgeting. Businesses use it to record income and expenses, calculate profit margins and create financial predictions. By analyzing financial data, organizations are able to monitor their performance over time and plan better for future growth. Marketing performance: In addition to that, Excel can be used in market analysis. Marketing teams utilize Excel to track campaign and social media performance, customer engagement and product popularity. Insights derived from this data helps companies improve their marketing strategies and better understand consumer behavior. This past week I was introduced to several data cleaning features and formulas used in Excel to make analysis less nerve-wracking. For example, in stead of editing data cell by cell in the case of duplicate values, you can use the Find and Replace filter. Also, conditional formatting makes it easier to highlight specific cell ranges and erase duplicate values. Functions and formulas make it easy to obtain statistical and mathematical data. Learning Excel helps you look at data differently. Instead of data being just a bunch of texts, numbers or logs, data becomes something you can use to gain insights, make decisions, reveal pat

How I got here On principle, you will never catch me parading myself as a some sort of expert data scientist. Technically, that's what I do in my day job, but I know I still have so much to learn because the field is broad, and to truly become expert requires dangerously ambitious levels of work ethic. I think I'm a functional data scientist who learns more as I encounter new problems daily. I'm writing this piece because in the last week or two, precisely three people have asked me questions related to transitioning into data science. As such, I thought to unify my thoughts around the topic so that I can refer anyone else who asks here--if anyone else ever asks. This article assumes you're already familiar with some of the data science entails such as data analysis, model training, prediction, etc, so I will not be doing a lecture series, just addressing some of the disconnects I have observed in conversation with people looking to transition to the field. Initial Excitement In 2026, it's easy to see what claude or chatGPT is doing and go "What sorcery is this? I must learn this trick!" and then reach out to the closest person you know who has ever mentioned anything about data or machine learning to find out how you can transition into AI. First of all, transitioning into "AI" is such a broad way to look at it. It is analogous to saying "I want to emigrate to Africa, show me how". But that's forgivable too. To cut short your initial excitement, or maybe redirect it, playing with a locally hosted LLM or making API calls to the DeepSeek endpoint is not data science, or machine learning or "AI". It's coding. And if you want to go down that route, you're better of focusing on software engineering. I say this because when you work with LLMs, the finished models to be specific, it's like using any other SaaS API out there. The difference being that you're interacting with a much less deterministic interface. But the rest of the work you do around it is pretty much a det

When I started learning Excel as part of my Data Science & Analytics course, I assumed it was just a tool for creating tables and performing basic calculations. After spending a week exploring its features, I quickly realized that Excel is much more powerful than I thought. Almost every organization generates data. Businesses track sales, schools monitor student performance, hospitals manage patient records, and marketers analyze campaign results. Before data can be analyzed, it needs to be organized, cleaned, and summarized—and that's where Excel comes in. In this article, I'll share some of the Excel concepts I've learned so far and how they're used in real-world data analysis. Understanding the Excel Workspace Before working with data, it's important to understand the basic structure of Excel. When you open Excel, you're working inside a workbook . A workbook can contain multiple worksheets (often called sheets), which help organize different sets of data. At the top of the screen is the Ribbon , which contains tabs such as Home, Insert, Page Layout, Formulas, Data, and View. The Ribbon acts like a control center where you can access Excel's tools and features. Rows run horizontally and are identified by numbers, while columns run vertically and are identified by letters. The intersection of a row and column is called a cell , where data is entered. At first, all these parts seemed overwhelming, but after using Excel regularly, navigating through them has become much easier. The Different Types of Data in Excel One of the first things I learned is that not all data is the same. Excel commonly works with: Text data (names, product categories, locations) Numeric data (sales figures, quantities, prices) Date and time data (order dates, deadlines) Logical data (TRUE or FALSE values) Understanding data types is important because Excel treats each type differently when performing calculations and analysis. Number Formats Matter More Than I Expected Another concept that

Last year, I examined RUM indexes within this series on multi-key indexing, demonstrating that they cannot substitute MongoDB's compound indexes for sorted queries. A year later, Microsoft has fixed this in the DocumentDB extension for PostgreSQL with an Extended RUM index that preserves the ordering of the keys, allowing an ordered scan rather than a bitmap scan. Let's revisit our pagination query to see how it performs now. I start a container with the latest DocumentDB (version v0.112-0 from May 26, 2026): docker run -d --name documentdb-local -p 10260:10260 -p 9712:9712 ghcr.io/documentdb/documentdb/documentdb-local:latest --username franck --password franck --start-pg I can connect to PostgreSQL on port 9712, where many extensions are installed, including the extended RUM index: docker exec -it documentdb-local psql -p 9712 postgres psql ( 17.10 ( Debian 17.10-1.pgdg13+1 )) Type "help" for help. postgres = # \dx List of installed extensions Name | Version | Schema | Description ------------------------- +---------+------------+------------------------------------------------------------ documentdb | 0.112-0 | public | API surface for DocumentDB for PostgreSQL documentdb_core | 0.112-0 | public | Core API surface for DocumentDB on PostgreSQL documentdb_extended_rum | 0.112-0 | public | DocumentDB Extended RUM index access method pg_cron | 1.6 | pg_catalog | Job scheduler for PostgreSQL plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language postgis | 3.6.3 | public | PostGIS geometry and geography spatial types and functions tsm_system_rows | 1.0 | public | TABLESAMPLE method which accepts number of rows as a limit vector | 0.8.2 | public | vector data type and ivfflat and hnsw access methods ( 8 rows ) postgres = # I can also connect to the MongoDB-compatible API: docker exec -it documentdb-local mongosh -u franck -p franck 'mongodb://localhost:10260/?tls=true&tlsAllowInvalidCertificates=true' Current Mongosh Log ID: 6a0b3b537d2a1c3471d1a7ba Connecting to: mo

From a massive DOGE data breach and the hacking of critical energy and water systems to the hack of an FBI surveillance system, here are the most damaging security incidents and data breaches of 2026.

Before this week, I thought Excel was just a fancy calculator with boxes. But after three days of my Data Science & Analytics course, I realise I was wrong. Really wrong. Excel is a spreadsheet tool used by millions of people from small business owners to data analysts at giant companies. And the best part? You don’t need to be a programmer to use it. You just need to know a few tricks. Here’s how Excel helps solve real-world problems using exactly what I learned in Week 1. 3 Real-World Ways Excel Is Used Business decisions with logic Managers use IF() statements to answer yes/no questions. Example: =IF(Sales>1000, "Bonus", "Needs Improvement"). One cell can decide who gets paid more. Cleaning messy data Real data is never clean. Marketing teams use Remove Duplicates, Find & Replace, and Text to Columns to fix hundreds of messy rows in seconds. No manual typing. Tracking deadlines and ages HR teams use DATEDIF() to calculate employee ages or years of service. TODAY() and NOW() keep reports automatically updated. No more “oh, I forgot to update the date.” 3 Excel Features I Learned This Week Remove Duplicates – One click, and Excel deletes repeated rows. Saved me from sending the same customer email twice. IFERROR() – Hides ugly errors like #DIV/0! and shows something friendly instead (e.g., “Check data”). Your boss will thank you. Sort & Filter – With AutoFilter, I can find all sales above $500 in one second. Then Custom Sort lets me sort by date and region together. My personal reflection Honestly? Learning Excel has changed how I see data. I used to look at a messy spreadsheet and feel lost. Now I see Remove Duplicates, Text to Columns, and TRIM() as tiny tools that bring order to chaos. Data isn’t scary anymore. It’s just a puzzle and Excel gives me the pieces. I’m only one week in. But I already feel like a junior data analyst in training.