今日已更新 230 条资讯 | 累计 24187 条内容
关于我们

标签:#Data

找到 500 篇相关文章

AI 资讯

Bringing MongoDB Atlas and Voyage AI to Dify: Build RAG Workflows and Data Agents Without Heavy Glue Code

AI applications are moving quickly from simple chatbots to systems that can search, reason, recommend, summarize, and act on live business data. For developers, that usually means wiring together databases, embedding models, vector search, rerankers, orchestration logic, and application code. For no-code AI builders, it often means waiting for those integrations to exist before an idea can become a working prototype. The MongoDB extensions for Dify help close that gap. With the new MongoDB Atlas and Voyage AI extensions, Dify builders can visually compose AI workflows and agents that connect directly to MongoDB data, perform semantic retrieval with Atlas Vector Search, improve result quality with Voyage AI embeddings and reranking, and optionally interact with operational documents through controlled database tools. The result is a practical path from idea to working AI application: less custom orchestration code, more reusable building blocks, and a smoother experience for both developers and no-code builders. Why Dify and MongoDB Belong Together Dify provides a visual environment for building AI apps, workflows, and agents. It makes it easy to connect user input, model calls, tools, prompts, and outputs into a working application. MongoDB Atlas provides the data foundation: flexible documents, operational queries, aggregation, full-text search, and vector search in one platform. Together, they create a powerful pattern: Dify orchestrates the AI experience — workflows, agents, prompts, tools, and user interactions. MongoDB Atlas stores and retrieves the data — documents, application records, knowledge sources, and vector embeddings. Voyage AI improves retrieval quality — embeddings for semantic search and reranking for precision. For a no-code builder, this means you can assemble a retrieval-augmented generation workflow visually. For a developer, it means the integration points are packaged as reusable Dify tools rather than one-off glue code. Meet the Extensions

2026-05-31 原文 →
AI 资讯

CONFIGURING SEMANTIC MODEL IN POWER BI

INTRODUCTION Configuring a Power BI semantic model involves refining data structures, creating relationships, and setting up calculations. Semantic model is the last stop in the data pipeline before reports and dashboards are built. It is the end product of the raw data that has been extracted, transformed, loaded, modeled, built relationship, and written calculation. The Semantic model consist of Data connections to one or more data sources, Transformations that clean and prepare the data for reporting, Defined calculations and metrics based on business rules to ensure consistent reports and Defined relationships between tables. Key words to note in Semantic Modelling are; 1. Fact table and Dimension table: The Fact table records the quantitative and numerical data. It is where every single details are recorded. The Dimension table act as the descriptive companion to the fact table, containing the attributes or characteristics that provide context to the data. 2. Primary and Foreign Key: Primary Keys are unique identifier assigned to a specific record with a database table ensuring that no two rows are identical or repeated. foreign Keys are columns or group of columns in one table that provides a link between data in two tables by referencing the primary key of another. 3. Star Schema Star Schema is a data modeling technique where a central fact table is surrounded by several dimension tables that provide descriptive content. 4. Cardinality Cardinality defines the kind of relationship between two tables. They are; One to Many (1.*) Many to one (*.1) One to One (1.1) Many to Many ( . ) The cardinality of a relationship is described by the "one" (1) or "many" (*) icons located at the ends of the relationship line. 5. Cross Filter Direction The direction determine how filters propagate. Possible cross filter options are dependent on the relationship cardinality type. One to Many - Single or Both sides One to One - Both sides Many to Many - Single to either table or b

2026-05-31 原文 →
AI 资讯

Great Stack to Doesn't Work Bonus: SQL vs NoSQL: Which One in 2026?

The honest decision framework, not another flame war. The SQL vs NoSQL debate has been running for 15 years and it still generates more heat than light. Here's the framework that actually helps you decide. The Real Question It's not "SQL or NoSQL." It's: what does your access pattern look like? If your application is mostly reading and writing related data through well-defined queries — orders with line items, users with addresses, products with categories — relational databases are purpose-built for this. JOINs are not expensive when they're indexed. Transactions are not slow when they're scoped correctly. PostgreSQL handles 50 million rows comfortably on a single node. If your application is reading and writing self-contained documents with predictable access by a primary key, and you rarely need cross-document queries — user profiles, product catalogs, content management — a document database simplifies your code. No ORM mapping hell. No migration files for adding a field. If your application writes massive volumes and reads by partition key with eventual consistency — time-series data, IoT telemetry, activity feeds at scale — wide-column stores like Cassandra were built for this specific workload. The 2026 Reality PostgreSQL has eaten NoSQL's lunch in many areas. JSONB support means you can store and query unstructured data inside PostgreSQL with GIN indexes. You get the document model flexibility without giving up transactions, JOINs, and a 30-year ecosystem. For 80% of startups and mid-size companies, PostgreSQL is the only database you need. MongoDB has gotten more relational. Multi-document ACID transactions (since 4.0), schema validation, aggregation pipelines that look suspiciously like SQL. It's converging toward what PostgreSQL already does, but with a different starting point. DynamoDB dominates serverless. If you're in AWS and your access pattern is simple key-value with known query patterns, DynamoDB's pricing model (pay-per-request) and operational s

2026-05-31 原文 →
AI 资讯

I created a fork of GunDB and rewrote it in TypeScript using Vibe Code

Inspired by a similar project called GenosDB and Cloudflare’s initiative to rebuild Next.js, I decided to rebuild GunDB with a modern coding style, incorporating improvements and addressing shortcomings in the original technology. I used the OpenCode tool with the Big Pickle model to rewrite the project in a new graph database called Garfo (the Portuguese word for “fork”), and I was impressed with the results and its practical applications. In this article, I’ll explain the technology and its improvements over GunDB. Introduction Garfo is a modern, browser-first fork of GUN.js — the decentralized, offline-first graph database. A fork of the original project that keeps the familiar GUN graph API while bringing meaningful improvements to the modern JavaScript ecosystem. Why a Fork? GUN.js is a revolutionary technology — a graph database that syncs in real time, works peer-to-peer, resolves conflicts automatically, and runs in the browser. However, the JavaScript ecosystem has evolved. TypeScript has become the standard, ES modules are the norm, and new transport layers like Nostr have emerged as promising decentralized protocols. Garfo was born to fill these gaps: a GUN rewritten with modern typing, designed with the browser as a first-class citizen, and with native support for the Nostr protocol. Key Features Familiar Graph API If you've used GUN before, you'll feel right at home. Garfo exposes the same chainable API: import Garfo from ' garfo ' ; const db = new Garfo ({ localStorage : true }); db . get ( ' users ' ). get ( ' alice ' ). put ({ name : ' Alice ' , status : ' online ' }); db . get ( ' users ' ). get ( ' alice ' ). on ( profile => { console . log ( ' Update: ' , profile ); }); All the classic methods are there: get() , put() , set() , on() , once() , map() . Optional Nostr Transport This is one of the most exciting additions. Garfo can use Nostr relays as a transport layer, allowing peers to exchange graph messages through public or private relays: const

2026-05-31 原文 →
AI 资讯

Releasing HeliosProxy, The programmable Postgres data-plane

Happy to announce HeliosProxy !! Far beyond a pooling tool, HeliosProxy ** is a next-gen programmable Postgres data-plane. **Works with PostgreSQL-compatible databases , not only HeliosDB. It starts as a PgBouncer-compatible wedge, then adds the operational surface teams usually build from multiple tools: connection pooling failover and transaction replay shadow execution anomaly detection edge cache controls admin REST API embedded admin UI signed WASM plugins OCI-style plugin artifacts Kubernetes operator Terraform and Pulumi providers 22 installable Claude/Codex operator skills Install operator skills: heliosdb-proxy install skills PostgreSQL #DevOps #SRE #Database #AIcoding

2026-05-30 原文 →
AI 资讯

Append-only doesn't mean what you'd hope

Event sourcing gets sold on immutability. You don't update, you don't delete, you only append, so the history is permanent. It mostly isn't. The events are immutable because your code agrees not to touch them, not because anything actually stops it. Underneath they're still rows in Postgres, and rows have a DBA with write access. A migration that "cleans up" old data. A 2 a.m. query run against the wrong connection. A backup restored with slightly different bytes in it. Change one of those rows and a replay won't blink. The aggregate rebuilds, the projections rebuild, everything looks fine. Usually the first person to notice is a customer whose balance is off, and by then the trail is cold. Chain each event into the next The trick is small. Give every row two extra columns: a hash of its contents, and the hash of the row before it. #1 AccountOpened prev=00000… hash=70be4f… │ ▼ #2 AmountDeposited prev=70be4f… hash=796018… │ ▼ #3 AmountWithdrawn prev=796018… hash=6a0260… The hash is SHA-256(previousHash || json(payload)) . Nothing exotic. The point is that each hash depends on the one before it. Edit a payload and its hash stops matching. Rewrite that hash to cover for the edit, and now the next row's pointer is wrong. You can't fix one without breaking the next. About forty lines of it Appending an event hashes it together with the previous one: public HashChainedEntry Append ( object payload ) { var previousHash = _entries . Count == 0 ? GenesisHash : _entries [^ 1 ]. Hash ; var hash = ComputeHash ( previousHash , payload ); var entry = new HashChainedEntry ( _entries . Count + 1 , payload , previousHash , hash ); _entries . Add ( entry ); return entry ; } internal static byte [] ComputeHash ( byte [] previousHash , object payload ) { var payloadJson = JsonSerializer . SerializeToUtf8Bytes ( payload , payload . GetType ()); var combined = new byte [ previousHash . Length + payloadJson . Length ]; Buffer . BlockCopy ( previousHash , 0 , combined , 0 , previousHash .

2026-05-30 原文 →
AI 资讯

UUID v4 vs UUID v7 — Lequel choisir pour PostgreSQL en 2026 ?

Si vous utilisez PostgreSQL, vous avez probablement déjà dû choisir entre une clé primaire BIGSERIAL et un UUID. Depuis des années, la version 4 (aléatoire) est le choix par défaut quand on veut un identifiant unique et distribué. Mais en 2026, une alternative plus récente s’impose : UUID v7, qui intègre un timestamp et promet de meilleures performances pour les index. Dans cet article, je vous explique concrètement ce qui change, avec des benchmarks PostgreSQL et des exemples de code, pour que vous puissiez décider en connaissance de cause. UUID v4 : le standard aléatoire et son problème d’index Un UUID v4 est constitué de 122 bits aléatoires. Cette absence totale de tri est sa force pour l’unicité, mais elle devient un handicap dans un index B‑tree, qui est la structure utilisée par PostgreSQL pour les clés primaires. Lorsque vous insérez un nouvel UUID v4, il a autant de chances de se retrouver au début de l’index qu’à la fin. Résultat : l’index se fragmente, les pages se remplissent mal, et les performances d’écriture se dégradent à mesure que la table grossit. J’ai reproduit un test simple sur PostgreSQL 16 avec 10 millions de lignes, en utilisant une table dont la seule différence est la colonne id : -- Table UUID v4 CREATE TABLE events_v4 ( id UUID DEFAULT gen_random_uuid () PRIMARY KEY , payload JSONB , created_at TIMESTAMPTZ DEFAULT now () ); -- Table UUID v7 (généré côté application, voir plus bas) CREATE TABLE events_v7 ( id UUID PRIMARY KEY , payload JSONB , created_at TIMESTAMPTZ DEFAULT now () ); Après insertion, voici les mesures : Type de clé Taille de l’index Fragmentation Latence moyenne d’insertion BIGINT ~214 Mo 0 % ~0,8 ms/ligne UUID v4 ~428 Mo (2×) 99 % ~4,8 ms/ligne UUID v7 ~428 Mo (2×) ~2 % ~1,1 ms/ligne Ce qui frappe, c’est la fragmentation quasi nulle de l’UUID v7. L’index reste compact et les insertions sont presque aussi rapides qu’avec un BIGSERIAL. L’UUID v4, lui, est plus de quatre fois plus lent à l’insertion sur ce volume. UUID v7 :

2026-05-30 原文 →
AI 资讯

How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability

The engineering team at Meta recently outlined how the company migrated a data ingestion platform that transfers several petabytes of MySQL social graph data daily to improve reliability and operational efficiency. The team used techniques like reverse shadowing and continuous checksum monitoring to ensure zero downtime during the transition. By Renato Losio

2026-05-30 原文 →
AI 资讯

SQL Pattern Series #1: The Presence Pattern

Thinking in terms of existence instead of lists SQL Pattern Series #1 of 21 A collection of practical SQL patterns that help developers recognize common solutions to recurring database problems. What You'll Learn In this article you'll learn: When EXISTS and IN solve the same problem The difference between set membership and existence Why the underlying mental model matters When I typically reach for EXISTS Most SQL developers write a query like this at some point: SELECT c . CustomerID , c . CustomerName FROM Customers c WHERE c . CustomerID IN ( SELECT o . CustomerID FROM Orders o ); And it works. But sometimes it isn't the best way to think about the problem. The Question Behind the Query Many SQL problems can be framed in two different ways. Set Membership Is this value in a set? WHERE CustomerID IN (...) Existence Does at least one matching row exist? WHERE EXISTS (...) Both approaches often return the same result. But they represent different mental models. The Presence Pattern The Presence Pattern is useful when you do not actually care about the values being returned from a related table. You only care whether a matching row exists. For example: Customers who have placed an order Users who have logged in Employees assigned to a project Products that have sales In these cases, the question is often: Does a related row exist? rather than: What values are contained in this list? Example Using EXISTS SELECT c . CustomerID , c . CustomerName FROM Customers c WHERE EXISTS ( SELECT 1 FROM Orders o WHERE o . CustomerID = c . CustomerID ); The subquery is correlated to the outer query. Conceptually, SQL asks: For this customer, does at least one matching order exist? As soon as the answer becomes true, the condition is satisfied. Why This Pattern Matters Many SQL developers initially learn syntax. Over time, they discover that query writing is really about choosing the right mental model. The Presence Pattern encourages you to think in terms of: existence relationshi

2026-05-30 原文 →
AI 资讯

nbwipers: Setup and Troubleshooting

What is nbwipers? nbwipers is a CLI tool that strips outputs and metadata from Jupyter notebooks before git commit. Written in Rust - faster than nbstripout Supports git clean filter Works with .ipynb files Why use it? Jupyter notebooks store cell outputs inside the .ipynb file (JSON). This causes problems: Noisy diffs - output changes pollute every commit Repo size - images and large outputs bloat the repo Security - sensitive data can leak in outputs (API keys, query results) The solution: strip outputs automatically on git add via a clean filter. Why not nbstripout? nbstripout is written in Python. It is slow - git status , git diff , and git add all became noticeably slow on this repo because nbstripout was invoked for every .ipynb file. The main cause is Python startup time. With 100+ notebooks, nbstripout can take 40+ seconds where a Rust-based tool takes ~1 second. Faster alternatives: Tool Language Notes nbstripout-fast Rust Up to 200x faster; no git filter install support nbwipers Rust Inspired by nbstripout-fast; adds git filter + pyproject.toml config nbwipers is essentially nbstripout-fast with better git integration. Switching to nbwipers fixed the slowness. Setup 1. Install felixgwilliams/nbwipers is now in the aqua registry as of v4.517.0 . Using aqua , add to aqua.yaml : packages : - name : felixgwilliams/nbwipers@v0.6.2 Then run: aqua install 2. Configure git filter Run once per repo (writes to .git/config ): git config filter.nbwipers.clean "nbwipers clean -" git config filter.nbwipers.smudge cat git config filter.nbwipers.required true Or edit .git/config directly: [filter "nbwipers"] clean = nbwipers clean - smudge = cat required = true required = true makes the commit fail if nbwipers is not installed. This prevents accidentally committing outputs. 3. Add .gitattributes In the repo root, add .gitattributes : *.ipynb filter=nbwipers **/.ipynb_checkpoints/*.ipynb !filter **/.virtual_documents/*.ipynb !filter The !filter lines exclude checkpoint an

2026-05-30 原文 →
AI 资讯

DuckDB 1.5.3 Iceberg updates, PostgreSQL TDE extension & AI index tuning

DuckDB 1.5.3 Iceberg updates, PostgreSQL TDE extension & AI index tuning Today's Highlights Today's highlights include DuckDB's enhanced Iceberg integration with new DML and schema evolution features, alongside a deep dive into PostgreSQL's new open-source Transparent Data Encryption. Additionally, we explore AI-driven strategies for automating PostgreSQL index tuning, offering practical performance improvements. New DuckDB-Iceberg Features in v1.5.3 (DuckDB Blog) Source: https://duckdb.org/2026/05/29/new-iceberg-features.html The latest DuckDB v1.5.3 release introduces significant enhancements for working with Apache Iceberg tables, a critical component in modern data lake architectures. Key additions include full MERGE INTO support, allowing users to efficiently update, insert, and delete rows in Iceberg tables based on a source query. This release also brings ALTER TABLE commands for schema evolution, enabling operations like adding, renaming, or dropping columns, crucial for adapting to changing data requirements. Furthermore, DuckDB now supports partition transforms within Iceberg, providing more control over data organization and query optimization. Compatibility has been extended to Iceberg V3, ensuring support for the latest table format specifications, and improved handling for Iceberg REST Catalogs streamlines metadata management. These features position DuckDB as an even more powerful embedded analytical database for processing large-scale, evolving datasets directly in a data lake environment, making complex ETL/ELT operations more accessible and performant. Comment: The MERGE INTO and ALTER TABLE additions are game-changers for using DuckDB in production data pipelines with Iceberg, enabling robust upserts and schema changes. Open-Source TDE for PostgreSQL: What pg_tde Is, and Whether You Need It (Planet PostgreSQL) Source: https://postgr.es/p/9kM This article introduces pg_tde , PostgreSQL's new open-source Transparent Data Encryption (TDE) option, a l

2026-05-30 原文 →
AI 资讯

Presentation: Building Evals for AI Adoption: From Principles to Practice

Mallika Rao discusses the hidden risk of evaluation debt in production AI systems, drawing on her experience at Twitter, Walmart, and Netflix. She explains why traditional metrics fail modern architectures, breaks down a five-layer evaluation stack spanning infrastructure and UX, and shares a diagnostic maturity model to help engineering leaders eliminate silent semantic failures. By Mallika Rao

2026-05-29 原文 →
AI 资讯

How to Route Real-Time Gold and Silver Prices from a Unified WebSocket Stream

When I first connected to a precious metals WebSocket API, I expected to get a clean stream of prices. What I actually got was a firehose of mixed ticks—gold, silver, platinum—all arriving through the same callback. If you’ve ever tried to build a trading bot or a custom chart, you know this is a recipe for disaster. In this post, I’ll share how I solved the problem with a few lines of Python and a clear mapping strategy. The scenario: You have one WebSocket URL that pushes quotes for multiple metals. You need to separate them so you can update different UI components, run independent strategies, or store them in distinct database tables. The data pain point: every message uses the same JSON structure, and the only differentiator is a field like symbol . If you don’t act on it immediately, everything gets mixed up. Identify Assets via the Symbol Field Start by checking the API docs for the field that carries the instrument code. Usually it’s symbol , but instrumentId or type are also used. Here’s a typical reference table: Field Description Example symbol Asset code XAUUSD, XAGUSD instrumentId Internal platform ID 1001, 1002 type Asset class gold, silver I turn this into a dictionary mapping each symbol to a human-readable category: asset_map = { " XAUUSD " : " gold " , " XAGUSD " : " silver " , " XPTUSD " : " platinum " } Buffer Messages by Type Because these streams are high-frequency, I avoid processing every tick individually. Instead, the WebSocket callback just updates an in-memory store that is already grouped by asset type: # Keep the hot path extremely light def on_message ( msg ): symbol = msg [ ' symbol ' ] price = msg [ ' price ' ] asset_type = asset_map . get ( symbol , " unknown " ) cache [ asset_type ][ symbol ] = price Then, a background timer fetches the latest prices from cache["gold"] and cache["silver"] separately and does the actual work—like computing indicators or rendering charts. The key benefit is complete isolation: your gold logic never t

2026-05-29 原文 →
AI 资讯

Building a Custom API Using PL/SQL with ORDS

In modern application development, exposing database logic as REST APIs is a powerful way to integrate systems. Oracle REST Data Services (ORDS) makes it easy to turn PL/SQL into RESTful APIs without needing a separate backend service. In this blog, we’ll walk through how to create a simple POST API using ORDS and PL/SQL to insert data into a table. Pre-requisites A cloud-based ATP wallet (I prefer) Let's start how we create the APIs on the top of any custom table which relies on databases Create a table in the oracle SQL Developer and followed by create an ORDS Module 1.Create an ORDS Module A module is a logical container for related REST endpoints. What this Module does ?? Creates a module named nj_api Defines base URL: http://server_name/ords/table_Schema/nj_api/ 2: Define a Template (Endpoint Path) A template represents the API endpoint path. It defines how Endpoint URL:/ords/table_schema/nj_api/insert_data 3: Define the Handler (Business Logic) The handler contains the logic executed when the API is called. Key Concepts: p_method => 'POST': Defines HTTP method p_source_type => ORDS.source_type_plsql: Uses PL/SQL block Bind variables (:name, :num, etc.) map directly to JSON request body parameters 4: Testing the API Using Tools like Postman,cURL,ORDS REST Workshop I tested with Postman FYR Let's call same in Oracle VBCS in new blog. .. Try other methods like Delete, PATCH & GET

2026-05-29 原文 →
AI 资讯

Oracle ORA-00031 Error: Causes and Solutions Complete Guide

ORA-00031: Session Marked for Kill — What It Means and How to Fix It ORA-00031 occurs when a DBA issues ALTER SYSTEM KILL SESSION but Oracle cannot terminate the target session immediately. Instead, Oracle marks the session as "KILLED" and waits for it to reach a safe termination point — typically after completing a rollback or releasing OS-level resources. This is less of a hard error and more of a transitional state that every Oracle DBA will eventually encounter. Top 3 Causes 1. Large Transaction Rollback in Progress When you kill a session mid-transaction, Oracle must roll back all uncommitted changes to preserve data integrity. The larger the transaction, the longer the session stays in KILLED status. -- Check rollback progress for KILLED sessions SELECT s . sid , s . serial # , s . username , t . used_ublk AS undo_blocks , t . used_urec AS undo_records FROM v $ session s JOIN v $ transaction t ON s . taddr = t . addr WHERE s . status = 'KILLED' ; 2. Unresponsive or Disconnected Client If the client network connection is broken or the client process has hung, Oracle cannot deliver the kill signal. The session lingers in KILLED state until the OS-level connection finally times out. -- Find the OS process ID (SPID) for stuck KILLED sessions SELECT s . sid , s . serial # , s . username , s . status , p . spid AS os_pid , s . machine , s . program FROM v $ session s JOIN v $ process p ON s . paddr = p . addr WHERE s . status = 'KILLED' ; 3. OS-Level I/O or Resource Wait Sessions blocked at the OS level (disk I/O stall, memory pressure, storage issues) cannot respond to Oracle's internal kill signal. In these cases, only an OS-level process termination will resolve the problem. -- Identify what the session was waiting on before being killed SELECT sid , serial # , status , event , wait_class , seconds_in_wait FROM v $ session WHERE status = 'KILLED' ; Quick Fix Solutions Option 1 — Use the IMMEDIATE keyword (recommended first step) -- Standard kill (asynchronous) AL

2026-05-29 原文 →
AI 资讯

The Paradox of Democratized Software

Everyone can build it. Almost no one can afford to run it at scale. And the companies selling the picks and shovels are about to get undercut by the same forces they unleashed. by VEKTOR Memory — 20 min read How This Article Started: 20 Forums, 40 Headlines, and a Growing Sense That Everyone Was Confused I woke up to clear skies and the sun finally shining, and I set out to understand this idea, the truth behind it, and the nagging suspicion that the narrative around AI and software costs had become so loud, so uniform, and so confidently confusing that someone needed to sit down and actually go through it. No tweets, or are they now X's? No LinkedIn thought leader infomercials, no Substack hype, just actual research and deep thoughts. So I spent time reading, collating data. Forums, whitepapers, LinkedIn posts, Hacker News threads, VC essays, Reddit arguments. I went looking for the real signal underneath the noise. What I found instead was the full spectrum of human overconfidence, lots of moat real estate. On one end: the hype machine at full throttle. “Software is going to zero.” “A solo dev can now build what a 50-person team built in 2021.” “The era of the $500/month SaaS subscription is over.” “Vibe coding will replace your entire engineering org.” These headlines were everywhere. Breathless. Confident. Shared tens of thousands of times, this angle gets views, of course, the algorithm loves being fed claps, shares, comments, and reposts. Most were written by people who had a very good Tuesday with Codex, Windsurf, Claude and Cursor and decided that instant dev, open source to Github and getting oodles of stars, maybe even roping in a celebrity, was now the permanent condition of software development. “We are now famous on GitHub!" Very hipster, very vibes, see you on the playa.. On the other end: the backlash. Experienced engineer, people with 15 to 25 years in production systems are pushing back hard. “Show me the vibe-coded app that survived its first real

2026-05-29 原文 →
开发者

How I built an Ofsted school data API on Apify (without scraping a single webpage)

Most scraping projects start by finding a website to scrape. This one started from the opposite direction: I knew the data existed as official government downloads, and my job was to make it accessible via a clean API. The data source Ofsted (the UK school inspections body) publishes monthly management information as CSV files on GOV.UK. The file covers all 22,000+ state-funded schools in England with their latest inspection grades, local authority, postcode, phase, and size data. It's 16 MB, published under the Open Government Licence v3.0 — explicitly permitting commercial use. No scraping needed. No authentication. Just a CSV download and some parsing logic. The architecture The actor is deliberately simple: Fetch the GOV.UK stats page to find the current month's CSV URL (the URL hash changes with each release) Download the CSV (~16 MB from assets.publishing.service.gov.uk ) Parse it with csv-parse Apply the user's filters (name, local authority, region, postcode prefix) Push matching records to the Apify dataset No Crawlee. No browser. No proxy. Just fetch() and a CSV parser. const match = html . match ( /href=" ( https: \/\/ assets \. publishing \. service \. gov \. uk \/[^ " ] +latest_inspections_as_at [^ " ] + \. csv ) "/ ); That one regex does the URL discovery. The GOV.UK page lists files in reverse chronological order, so the first match is always the latest release. The interesting part: Ofsted changed their grading system mid-build I built this in May 2026. In November 2025, Ofsted scrapped their 20-year-old four-word judgement system (Outstanding / Good / Requires Improvement / Inadequate) and replaced it with a report card format — six separate grade areas, each on a five-point scale: Exceptional Strong Expected standard Needs attention Urgent improvement Plus a standalone Safeguarding verdict (Met / Not met). The April 2026 CSV reflects this change entirely. There's no "Overall effectiveness" column. Schools inspected before November 2025 have null gr

2026-05-29 原文 →