标签：#ai

What is an LLM evaluation harness? A deep dive into lm-eval-harness You fine-tuned a 7B model. It aced your smoke tests, your colleague ran a few prompts and shrugged approvingly, and the README is now full of cherry-picked outputs that look great in a screenshot. Then someone asks: how good is it, really? — and you realize you have no number to point at. No MMLU score. No HellaSwag. Nothing reproducible, nothing you can defend in a PR review, nothing you can compare to last week's checkpoint. That's the gap an evaluation harness fills. It turns "vibes-based evaluation" into something with a score, a stderr, and a config file you can re-run next Tuesday. Why evaluate LLMs at all? Two reasons that actually matter: Comparability. If you can't put a number on a model, you can't compare it to anything else — not the previous checkpoint, not the open-source baseline, not the commercial API you're trying to replace. Leaderboards are noisy and gaming-prone, but a local leaderboard with the tasks you care about is one of the most useful artifacts a team can build. Regression detection. Most model regressions are silent. A 0.3-point drop on MMLU won't show up in a chat session, but it will show up in CI. People who ship models for a living treat evals the way backend engineers treat unit tests: mandatory, run on every PR, and blocking on regressions. You don't need a hundred benchmarks. You need the three to five tasks that map to your actual use case , plus one or two general capability anchors (MMLU, HellaSwag) so you can sanity-check that you didn't accidentally destroy basic reasoning while you were tuning for your domain. What is an "evaluation harness"? An evaluation harness is the software that sits between a model and a benchmark. It handles the boring-but-critical parts: loading the model weights, tokenizing prompts in the way the benchmark expects, running inference, extracting the answer from a longer generation, scoring it against a ground-truth key, aggregating

Swap debugging war stories\n\nI have been living in webhook + PHP + email land (Vapi, OpenAI, PHPMailer, CSV logs).\n\nDrop your stack in a comment (even one line). I will reply with the first three places I would look for a silent production failure.\n\nNo sales pitch — trying to meet dev friends who ship unglamorous integration work.\n\nMy builds: elionmusic.com · prayerauthority.com

Amazon's Stargate series greenlit last year has been cancelled.

Sepehr Khosravi discusses the evolution of developer productivity tools. Evaluating the strengths of tools like Cursor and Claude Code, he explains actionable techniques for senior engineers - including context engineering, custom rules, and Model Context Protocol (MCP) integrations. He shares real-world benchmarks and strategic frameworks for balancing AI adoption with clean code quality. By Sepehr Khosravi

Researchers are using machine learning algorithms to decrypt historical pencil-and-paper ciphers.

OpenAI outlines a blueprint for U.S. governance of frontier AI, proposing a federal framework for safety, resilience, and national security.

OpenAI outlines its public policy agenda for AI, including safety, youth protection, workforce transition, and global standards to ensure AI benefits society.

A client asked me for a simple thing. Not ChatGPT. Not an agent. Not a multimodal assistant that can explain invoices, generate React components, and write poetry in three languages. Just a small classifier embedded into a website. The job sounded boring in the best possible way: take some text, classify it, return a result, keep it fast. So I started looking at the usual solutions. And then I had one of those moments where you stop reading documentation, lean back, and ask: Are we seriously doing this? Because the answer I kept running into looked like this: download a huge runtime download a huge model initialize a big ML stack then classify one small piece of text In one setup, the path was getting close to something like 250 MB per user . For a simple classifier. On a website. From a server. Every time. No. Sorry. That is insane. The problem The web has a strange habit now. You ask for one small AI feature, and the answer is often: bring the entire construction company. But sometimes I do not need a construction company. I need one person on the construction site. One task. One tool. One result. This is especially true for simple classification, embeddings, semantic search, routing, filtering, ranking, small local decisions. Not every AI problem needs an LLM. Not every website needs a full inference engine. Not every user should pay a 250 MB download tax because we were too lazy to think smaller. So I started digging I wanted something simple: runs in the browser does not require a server for inference small enough to actually ship works with transformer-style models can tokenize text can run BERT-like forward inference can produce embeddings or classification input does not bring ONNX Runtime, Candle, ndarray, or half the internet with it At first I thought: “Surely someone already made the tiny version.” There are great tools out there. Transformers.js is powerful. ONNX Runtime Web is powerful. Candle is powerful. But that was exactly the problem. They are pow

I'll be honest, when I first heard about Astro, I was skeptical. Another JavaScript framework? I already had React, Next.js was doing fine. Why bother? Then I actually used it. I built the website for Zimnovate, my AI-native digital product studio based in Harare, Zimbabwe, with Astro, ran it through Google PageSpeed Insights, and got scores I'd never seen before on a site I actually built myself. That changed everything for me. Let me tell you what Astro is, why it's architecturally different, and why it's worth learning, especially if you care about performance and SEO. What Even Is Astro? Astro is a web framework built around one radical idea: ship zero JavaScript by default . Most modern frameworks (React, Vue, Svelte) are component-based and hydrate the entire page on the client. Even if your page is mostly static content, the user's browser still downloads and runs JavaScript to render it. Astro flips this. It renders your components to pure HTML at build time. JavaScript only runs in the browser when you explicitly need it, and only for the specific components that need it. This isn't just a config option. It's the core architecture. The Architecture: Islands Astro uses a pattern called Islands Architecture . Think of your page as a static ocean with interactive "islands" floating in it. The ocean (static content, headings, text, images) ships as plain HTML. The islands (a navbar with a dropdown, a contact form, a live counter) are the only parts that hydrate with JavaScript. --- // This runs only at build time zero runtime cost const services = await fetch('/api/services').then(r => r.json()) --- <html> <body> <!-- Pure static HTML, no JS needed --> <h1>Zimnovate</h1> {services.map(service => <ServiceCard service={service} />)} <!-- This island hydrates only when visible --> <ContactForm client:visible /> </body> </html> The client:visible directive tells Astro: "only load this component's JavaScript when it scrolls into the viewport." You get full interacti

At GTC 2026, Jensen Huang said something that made a lot of people pause: the PC is being reinvented. He and Microsoft launched RTX Spark with the N1X chip, cramming petaflop-level AI compute into a desktop form factor. On the surface it looks like another hardware upgrade, but this time the use case is genuinely different. Previous PC performance gains served humans: faster rendering, faster compiling, smoother gaming. This round of compute improvement is largely aimed at AI agents. Agents need to run vision-language models locally, understand screen content in real time, and execute GUI operations. These workloads demand sustained compute resources with a load profile completely different from human computer use. Agents Need Different Hardware Than Humans Humans use computers in bursts: typing, clicking, waiting for responses. The load is pulsed. Agents use computers continuously: constantly capturing screenshots, interpreting the display, making decisions, executing operations. The load is steady-state. This means agents need memory bandwidth and energy efficiency more than peak compute. This explains why Apple's M-series chips perform well in on-device AI scenarios. The unified memory architecture lets GPU and CPU share the same memory pool without data transfers between them, which is highly efficient for model inference that frequently accesses large parameter sets. M-series energy efficiency also suits long-running agent workloads without thermal throttling. NVIDIA's RTX Spark takes another path: more GPU compute and more memory (128GB unified) to handle on-device AI demands. The N1X chip has higher total compute than M-series, better suited for heavy workloads. Different tradeoffs, same destination: AI agents running on the device in front of you. There's Already a Complete Agent Stack on Mac What's worth noting is that the on-device AI agent stack on Apple's ecosystem is already fairly complete. M-series chips at the hardware layer. MLX at the framework lay

StudyQuiz has moved forward since the first frontend MVP release. This update focuses less on adding major new features and more on making the app smoother to use, safer to change, and more reliable in production. What’s New Logout functionality Call-to-action section on the user dashboard Edit and delete functionality for questions and answers Guest Mode restrictions for editing and deleting questions and answers Integration tests for core backend workflows Improved Enhanced quiz user experience and feedback Improved quiz creation user experience Improved navigation across the application More reliable page reload behavior for protected routes Fixed Fixed 500 errors when reloading protected frontend pages Fixed production database session configuration issues Improved reliability around authentication-protected routes Why This Release Matters This release is mainly about stability and maintainability. I added integration tests and a GitHub Actions workflow so future changes are checked automatically before they silently break existing behaviour. StudyQuiz now feels closer to a maintainable product rather than just a working prototype. Coming Next The next major focus is slide upload and AI-assisted quiz generation, allowing users to generate structured quizzes more directly from their study materials. Repo: github.com/aissa-laribi/studyquiz Live app: https://www.studyquiz.co

Today I started Week 3 of the Machine Learning Specialization and learned about Classification. Until now, most of my learning focused on regression, where models predict numerical values. Today I discovered that many real-world problems involve predicting categories instead. Some examples include: Detecting spam emails Predicting whether a customer will leave or stay Identifying whether a tumor is malignant or benign I also learned about Logistic Regression. Despite its name, it is used for classification tasks. The model predicts probabilities that help determine which class an example belongs to. Another important concept was the Decision Boundary, which is used to separate different classes based on predicted probabilities. To reinforce my understanding, I completed the graded assignment for this section. This week feels like an important step because classification is widely used in real-world machine learning applications. 🚀 Looking forward to learning more about classification models and improving my understanding of machine learning. MachineLearning #AI #DataScience #Python #LearningJourney

Two Hypotheses In the contemporary discussion about artificial intelligence, two distinct hypotheses intersect and are often conflated. The first hypothesis describes AI as a thin client between intention and result. Historically, a chain of translators existed between a concept and an artifact. A person formulated a task for a programmer, the programmer wrote code, the code became a program. A screenwriter passed an idea to a studio, the studio hired a VFX team, the team produced a film. A composer worked with musicians and a studio to record a track. AI shortens this chain, allowing a result to be obtained directly from a natural language prompt. The second hypothesis is more radical. It asserts that AI washes out not only performers but also apprentices. The main function of many professions was not the production of the current result, but the reproduction of knowledge. A junior was needed not because he is useful today, but because in five years he will become a senior. A student was needed not to create value now, but to become an engineer. A doctoral candidate was needed not for brilliant papers, but to undergo the school of scientific thinking. The Destruction of the Apprenticeship Mechanism The classical model of competence growth was built on review. A junior wrote code, a senior dissected it, extracted the substrate of experience, and transmitted professional intuition. Each review was an act of knowledge transfer. The new model looks different. A person formulates a prompt, AI generates the result. If code of acceptable quality appears immediately, the economic need for a junior declines. Along with it, the mechanism through which knowledge was transmitted disappears. A structural question arises that goes beyond the labor market. Where will the next seniors come from if the intermediate link does not undergo the path of learning through mistakes and reviews. This is a problem of competence reproduction, not simply automation. The Transformation of Educa

The AI world is full of old infrastructure with stochastic organs. That sentence probably explains...

Researchers created an AI-powered worm that that can spread through networks with no human intervention.

After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics. By Pranav Bhasker

In the face of widespread backlash to the AI data center buildout throughout the US, Google is touting its efforts to minimize the environmental impact by actually increasing water for local communities. The company laid out five commitments around water use in a new blog post published Wednesday, including a goal to replenish more water […]

Online publishers are getting more control over whether their websites appear in Google's AI Search features, thanks to a UK regulatory ruling. The new conduct rule imposed by the Competition and Markets Authority (CMA) requires Google to let website owners keep their content out of features like AI Overviews and prevent it from being used […]

TL;DR Day 2 of AI Native DevCon shifted from agent capability to operating discipline. The strongest sessions focused on how teams can run AI-native delivery with clearer context pipelines, measurable agent behavior, safer execution boundaries, and better organizational ownership. The scale showed up in the numbers too. Across the two days, DevCon brought together 650+ in-person registrations, around 2,000 online registrations, and a packed mix of sessions, workshops, hallway conversations, and practical lessons. Day 2 leaned into workshops. That shift mattered because the second day was less about proving agents can do useful work and more about showing how teams can make that work repeatable. Hey there, welcome back. Rohan Sharma here again continuing the devcon series. Day 1 gave us the framing, including Guy Podjarny ’s core point that skills should be treated like real software assets. Day 2 picked up from there and moved into the operating details. Once agents are inside daily engineering work, platform and product teams need to decide what changes first, who owns those changes, and how the results are measured. Talks that shaped Day 2 Harness engineering beyond code Marc Sloan from Tessl focused on the next gap many teams are hitting. Code context is increasingly structured, but product and design context still lives in external systems such as Figma, Notion, and Linear. Pulling that context live can reduce staleness, but it introduces drift in evals, versioning, and reproducibility. The practical lesson was to stop treating external product and design context as random reference material. Teams need a defined layer between the repository and those external systems, with clear versioning so evaluations can be replayed against known context snapshots. Without that, agents can produce work that looks technically correct while missing the product constraint that actually mattered. That is a very expensive kind of almost-right. From vibes to metrics Simon Obstbau

AI Coding Agents in 2026: From Pair Programming to Autonomous Teams Slug: ai-coding-agents-2026-stack-comparison 1. The Three Categories That Actually Matter The 2024‑2025 hype cycle treated every AI coding tool as a single‑dimensional “best‑of‑list.” 2026 data shows that professional developers now average 2.4 tools per workflow (Stack Overflow Survey 2025). The real decision is architectural: Layer Goal Typical Agent Type Line‑level editing Speed, low latency Editor assistants Repo‑level planning Context depth, multi‑file changes Autonomous agents Enterprise governance Isolation, audit, CI/CD integration Platform agents Choosing a “one best tool” ignores the trade‑off between context window size (how many tokens the model can see) and execution speed (how fast the tool returns a suggestion). A narrow‑window editor assistant excels at instant autocomplete, while a wide‑window autonomous agent can rewrite an entire microservice in a single run. The three‑tier framework aligns the tool’s strengths with the architectural layer where they matter most. 2. Tier 1: Editor Assistants — Speed at the Line Level Tool Market Position Key Feature (2026) Pricing (per developer) Cursor $500 M+ ARR, fastest growth in Q1 2026 Parallel agents update git worktrees; 2‑second latency on 8‑core laptops $15 /mo (individual) – $120 /mo (team) GitHub Copilot 4.7 M paid subscriptions, 75 % YoY growth Agent Mode with multi‑agent workflows; deep VS Code integration $10 /mo (individual) – $100 /mo (enterprise) Windsurf 1.2 M active users, strong UI polish Real‑time code‑style enforcement; limited to 4‑file context Free tier up to 5 k lines, $30 /mo premium Tabnine Enterprise‑only after 2026 pivot Air‑gapped deployment; NVIDIA Nemotron 4‑bit models for on‑prem inference $200 /mo per seat (minimum 10 seats) When to choose each Cursor – prioritize raw typing speed and git‑aware suggestions. Ideal for startups that need rapid iteration without heavy IDE lock‑in. Copilot – best for teams already on