标签：#MachineLearning

How AI Helps Us Explore the Universe Modern telescopes and space missions generate more data in a single night than a team of human astronomers could review in a lifetime. The Vera C. Rubin Observatory in Chile, for instance, is expected to produce up to seven million alerts every night once it reaches full operational cadence, each one flagging something in the sky that changed since the last image. No group of humans can look at that stream and make sense of it in real time. Machine learning can, and increasingly does. This is the quiet story behind most recent breakthroughs in astronomy: it is not just bigger telescopes, but bigger telescopes paired with models that can filter, classify, reconstruct, and predict faster than any manual pipeline. Here is a tour of where AI is actually doing that work, and why it matters to anyone who writes code. The Data Problem Comes First Space science has quietly become a big data problem. The Rubin Observatory's ten-year Legacy Survey of Space and Time will produce roughly 60 petabytes of raw imagery and catalog around 20 billion galaxies and a similar number of stars. Every image the telescope takes is compared, pixel by pixel, against previous images of the same patch of sky, and any meaningful difference (a moving asteroid, a brightening supernova, a flaring galactic nucleus) triggers an alert within about two minutes of the exposure being taken. That alert stream is too large and too fast for manual triage. So astronomers built software "brokers": machine learning classifiers that sit between the telescope's raw output and the scientists, deciding in near real time which alerts are worth a second look. This is a pattern you will see across almost every domain of modern astronomy: instruments generate more signal than humans can parse, and a model is inserted into the pipeline to do the first pass of filtering. Finding Planets in a Sea of Noise Exoplanets are found mostly through the transit method: a planet passes in front

I created a simple text to image benchmark. I curated 192 prompts that are difficult for T2I models in various ways: text rendering, spatial reasoning, human realism, negations, etc... I then asked a VLM to judge every output against a pre-specified binary question with the ground truth baked in. I'm publishing all the results including the images. (Most public T2I leaderboards don't publish the actual images and that's a shame IMO) There is currently 52 model tested! more than 9k images have been generated and analysed! Full methodology: https://imagebench.ai/methodology-v1 Hugging face dataset: https://huggingface.co/datasets/dh7/imagebench (it contains the prompts to reproduce the results AND the results) Github: https://github.com/dh7/image-bench-ai Gallery to inspect the results: https://imagebench.ai/gallery Leaderboard: https://imagebench.ai/imagebench-v1 Limitations: it's text to image only, and VLM are not perfect as a judge. Let me know what could be useful from there! submitted by /u/dh7net [link] [留言]

Author here. Ibteda Digital Library is a private community archive in Pakistan — for ten years we digitized rare Urdu books (lithographs, dictionaries, periodicals) on a DIY camera rig, finishing every page by hand in Photoshop. When we wound down daily operations, I realized those 575,729 finished pages across 1,765 books recorded a decade of crop decisions, so I registered them back to their raw photos (SIFT + MAGSAC with conservative acceptance gates) and used the recovered geometry as supervision. The negative results are probably the most interesting part for this sub. Scaling from 378 to 572 training books didn't move unseen-book pass@80 . Neither did ResNet-50 (better training fit, flat held-out, worse after calibration), 1024px inputs, or a spatial head. Per-book error analysis showed why: the failures were near-constant offsets per volume — our operator's preferred margin inset, which simply isn't present in the pixels of a new book. Ten operator-corrected crops per book (element-wise median residual) took pass@80 from 0.71 to 0.83 on held-out volumes. Ten labels beat every scaling lever we tried. For retouching (stain/stamp removal), we kept the neural net to detection only — a U-Net proposes removal support, classical OpenCV reconstructs the paper, and everything outside the mask is byte-identical to the original. Labels used REMOVE/KEEP/IGNORE states, and any erased Urdu diacritic vetoed deployment regardless of IoU. The stricter label cut both improved mark IoU (0.56 → 0.60) and got diacritic false positives to zero. Two things I'd genuinely like input on: (1) has anyone modeled document boundaries that depend on an invisible human preference rather than visible structure — is there prior work on per-instance residual calibration like this? Our own next step is conditioning the model on the calibration examples directly (few-shot inset inference) instead of a post-hoc median. (2) Is there any constrained diffusion/inpainting setup you'd trust to guarant

I've been working on an open-source project called Millwright , an attempt to explore what an end-to-end machine learning workflow could look like in Rust. https://millwright-rs.dev/ This started while I was learning and building ML tooling in Rust. I kept finding capable individual libraries, but also gaps between them. Training a model was rarely the problem. Building the workflow around it — preprocessing, model selection, evaluation, explainability, deployment and monitoring — often meant integrating several unrelated crates and data representations. I initially started implementing some of those missing pieces as smaller independent crates. Eventually I realized I was more interested in the integration problem itself. That became Millwright. The current idea is to cover the classical ML lifecycle: ingest → explore → preprocess → select → fit → assess → explain → export → serve → monitor without trying to reimplement every ML algorithm. Instead, Millwright provides a common abstraction layer over existing Rust libraries and uses adapters for different ML backends. One architectural decision I'm experimenting with is having the framework own a small 2D data boundary ( Frame ) rather than exposing a particular backend's ndarray/dataframe representation throughout the API. That allows models and components backed by different libraries to participate in the same pipeline, at the cost of conversions at backend boundaries. The project currently includes work around: preprocessing and composable pipelines cross-validation and hyperparameter optimization multiple ML backends ensembles regression diagnostics SHAP-based explainability ONNX export model serving and registry drift monitoring time-series workflows incremental learning AutoML There are also Python bindings. I'm not building this on the assumption that Rust should replace Python for ML. Python's ecosystem is enormously more mature, and there would be little value in simply recreating scikit-learn in another l

sklearn 1.9 fixed a bug in how BayesianRidge computes its uncertainty. We traced predict on 1.8 and 1.9 and compared the two formulas it actually computes, see if you can spot what changed before the notebook tells you. https://github.com/aadya940/scikit-verify/blob/master/examples/sklearn_bug_hunting.ipynb submitted by /u/Lost-Dragonfruit-663 [link] [留言]

When someone asks whether to use Azure OpenAI Service or the direct OpenAI API, the starting point is this: the models running on both platforms are identical. GPT-4o, GPT-5, and the o-series models you deploy on Azure have the same weights, the same capabilities, and the same output quality as the ones you call from platform.openai.com, and what changes between the two platforms is the infrastructure where they run, the authentication mechanism, and the compliance guarantees the provider can offer on those requests. What changed in 2026 Azure AI Foundry was renamed Microsoft Foundry on January 1, 2026, and Azure OpenAI Service now lives inside that unified platform alongside the model catalog, development tooling, and agents. References to Microsoft Foundry in new documentation point to what used to be Azure AI Foundry. In July 2026, the GPT-5.6 family arrived with Sol, Terra, and Luna available on Azure the same day as on the direct OpenAI API. Historically Azure lagged four to eight weeks behind new model releases because Microsoft validates them within their compliance frameworks before making them available, and while that gap still exists for some specific features and APIs, for the main models in the GPT-5 family availability is converging. Where data is processed When you call GPT-4o from the OpenAI API, the request goes to OpenAI's own infrastructure, which is centralized and gives you no control over which region processes your data. For most use cases that doesn't matter, but for organizations with data residency requirements, regulatory compliance needs, or industries like healthcare, banking, or government, that detail can determine whether the service is usable at all. Azure OpenAI runs the same models within the boundary of your Azure tenant, so the data you send in prompts doesn't leave to OpenAI's infrastructure but processes in the Azure regions you choose. That's what makes it possible to meet HIPAA, SOC 2, EU data residency, and other certificati

Foundation models are expensive. A trading agent that calls GPT-4 for every price prediction burns budget fast. Lightweight forecasters are cheap to run but expensive to train, especially when you only have a handful of examples. MetaCaster introduces a meta-harness architecture where agents don't forecast directly. Instead, they train specialized lightweight models on-demand from few-shot examples and textual context. This is not another AutoML wrapper. The meta-agent orchestrates data generation, architecture selection, and training loops to produce task-specific forecasters in minutes. The result is a deployable model that runs inference without touching the foundation layer again. The Economic Gap Time-series forecasting in production faces a resource trap: Foundation models (TimeGPT, Chronos) deliver strong zero-shot performance but cost $0.002 to $0.02 per prediction at scale. Lightweight forecasters (PatchTST, DLinear, FEDformer) run for pennies but need thousands of training samples and hours of GPU time. Few-shot scenarios (new trading pairs, emerging markets, privacy-sensitive health data) don't have enough history to train from scratch. MetaCaster targets the intersection: resource-constrained environments where you need specialized models but can't afford foundation API calls or long training cycles. Meta-Harness Architecture The system has three layers: 1. Meta-Agent Orchestrator The top-level agent receives a few-shot time series (as few as 5-10 examples) and optional textual context (domain descriptions, seasonality hints). It decides: Which lightweight forecaster architecture to instantiate (PatchTST, DLinear, Autoformer, etc.) What synthetic data generation strategy to apply How to configure the training harness (learning rate, epochs, augmentation) The meta-agent uses a learned policy, not heuristics. It's pre-trained on a meta-dataset of diverse forecasting tasks so it generalizes to new domains. 2. Data Generation Agents These agents expand the f

Hi everyone, I’m researching how to design an AI agent for a medicine-reminder system. The agent has to decide, at each relevant time, whether to: send a reminder, wait (do nothing for now), or notify another person (e.g. caregiver), when it does not have complete information about the patient (has the dose already been taken? is the person nearby/attentive? are there adherence barriers? etc.). I’m trying to frame this properly before diving into implementation. Right now I’m looking at it as a sequential decision problem under partial observability (POMDP / belief-state RL territory), but I’m not sure how far that framing is actually useful in practice for this kind of system. I’d really appreciate any pointers on: Is a POMDP / belief-state approach overkill here, or is it the right formalization? What simpler alternatives (contextual bandits, MDP with engineered features, rule-based + uncertainty thresholds, etc.) have people used successfully for similar “remind vs wait vs escalate” decisions? Papers, open-source projects, or real systems that tackle medication adherence / context-aware reminders with uncertainty or incomplete observations. Common practical pitfalls (reward design, observation noise, alert fatigue, safety/escalation logic, evaluation metrics) that aren’t obvious from the theory. Any recommended starting points for someone new who wants to move from “I understand the concepts” to a small working prototype or simulation. I’m mainly in research/preparation mode right now, so even high-level advice, key papers, or “here’s what I’d do differently” comments would be very helpful. Thanks! submitted by /u/Senior_Disaster_7307 [link] [留言]

FP8 gave us a clean 1.5x on Qwen3-8B serving throughput on an RTX PRO 6000 Blackwell (1,725 to 2,597 tok/s at concurrency 32, vLLM). The uncomfortable question is always the same: did the model get dumber. This post is the exact check we ran before recommending the switch, with numbers, so you can run the same one. Why "run an eval suite" is usually the wrong first answer Standard benchmarks (MMLU and friends) are noisy instruments for quantization deltas at 8B scale. Score movement inside the error bars tells you nothing about whether YOUR prompts changed behavior. What you actually want to know is narrower: on the workload you serve, does the FP8 checkpoint produce materially different outputs than BF16, and are any of the differences wrong. That is answerable directly, cheaply, and per prompt. The method Both configurations run the same fixed workload: 20 prompts covering reasoning, code, summarization, translation, extraction, classification, math, and instruction following. Greedy decoding, temperature 0, 256-token cap, streamed. Greedy matters: it removes sampling noise, so any output difference is attributable to the numerics. Then a three-stage comparison: Byte equality. outputs_bf16[i] == outputs_fp8[i] . Anything identical is settled. Similarity triage. For non-identical pairs, difflib.SequenceMatcher.ratio() sorts near-identical wording drift from real divergence. Side-by-side review under a written rubric. Every non-identical pair gets read. The rubric asks one question: is there a factual or numerical claim that one precision gets right and the other gets wrong. Wording changes, reordering, and equally-defensible readings are recorded but not counted as regressions. The core loop is small: import difflib , json bf16 = json . load ( open ( " vllm_bf16_conc1.texts.json " )) fp8 = json . load ( open ( " vllm_fp8_conc1.texts.json " )) for i , ( a , b ) in enumerate ( zip ( bf16 , fp8 )): if a == b : print ( i , " identical " ) continue r = difflib . Sequenc

I am working on an evaluation design and would appreciate criticism before running it. Most coding-agent benchmarks collapse the model and its harness into one score. If a run fails, it is difficult to tell whether the cause was model capability, context assembly, task decomposition, tool design, retry policy, or the acceptance gate. A model can also look worse because the harness truncated its output, or look better because the gate only checked for plausible surface markers. The experiment I am considering crosses two independent variables: Workflow: one monolithic task versus decomposition into bounded slices with explicit contracts and acceptance criteria. Model policy: frontier-only versus cheapest-capable with escalation after a capability-graded failure. That produces four cells: frontier monolith, routed monolith, frontier decomposed, and routed decomposed. The frontier-decomposed cell seems especially important because it changes the task architecture while holding the model tier fixed. I would freeze the original tasks, source revisions, available tools, total retry budget, final acceptance criteria, validator versions, and the verifier. Every cell would be judged against the same final delivered outcome rather than against the persuasiveness of the agent's report. Proposed primary measures are cost per independently accepted change, false acceptance, false rejection, first-pass accepted yield, verification time, and reproducibility across three fresh runs. Token use, latency, escalation count, and context volume would be secondary measures. The confound I am least satisfied with is budget normalization. Decomposition changes the task distribution and may create more calls, which is part of the architectural treatment, but giving every slice the monolith's full context or retry budget would subsidize the decomposed condition. A shared system-level budget is cleaner, although it may hide which slices actually needed more capacity. There are no results yet,

I often remember the shot I want before I remember its filename. That gap is what binquery is for. It is a local Python CLI that indexes video clips and turns a sentence into a ranked shortlist for a human to review. It deliberately stops before editing: no timeline generation, no automatic cut, and no render. The smallest reproducible trial You can test the complete installed command path without supplying footage: python3 -m venv .venv .venv/bin/pip install binquery .venv/bin/binquery demo --out /tmp/binquery-demo The demo generates a synthetic 30-second video locally, then exercises splitting, indexing, validation, and querying. The first run may download OpenCLIP model weights. This is an end-to-end pipeline smoke test, not evidence of semantic search quality on real footage. Why keep the architecture small? The current design uses: ffmpeg to sample three frames from each clip OpenCLIP ViT-B-32 to build the local visual index plain JSON and NumPy files for metadata and vectors a JSON result containing clip paths, scores, and ranking signals There is no database, vector service, or daemon to operate. Querying an existing index does not resample the footage or rebuild the full index. The trade-off is straightforward: three frames keep indexing understandable and bounded, but they can miss important content in long or visually varied clips. I would rather expose that limitation than market a synthetic demo as a quality benchmark. Ranking signals are not explanations The output includes fields such as score , gate , and reasons . Here, reasons means ranking signals recorded by the pipeline. It should not be interpreted as a reliable semantic explanation of why a clip is correct. That distinction matters because a plausible-looking explanation can create more confidence than the underlying retrieval quality deserves. The shortlist is meant to reduce what a person must inspect, not replace editorial judgment. What binquery does not do It does not build a timeline or e

I wrote a technical breakdown of how search works on Papers with Code. The system combines keyword and semantic search, which produced better results than either approach alone. The stack includes: PostgreSQL with pgvector Qwen3-Embedding-0.6B for text embeddings Hugging Face Jobs with an NVIDIA L4 for batch embedding generation Hugging Face Buckets for storing artifacts A live embedding model served through Hugging Face Inference Endpoints The same infrastructure also powers the “related papers” recommendations shown on individual paper pages. Full write-up: How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code I’d be interested to hear how others are implementing hybrid search for research papers or similarly technical content. Disclosure: I work at Hugging Face and on Papers with Code. submitted by /u/NielsRogge [link] [留言]

Paper: https://huggingface.co/spaces/tri-fair-lab/publications/blob/main/Thomson_1_0_Technical_Report.pdf The development of frontier models is commonly perceived to be in the exclusive remit of a small number of heavily funded players, creating an information, economic and power asymmetry between developers and the diverse user base of modern AI. Recent public discourse acknowledges this concern, calling for SovereignAI (an organisation's capability to independently build, deploy and govern AI use), but often providing little concrete advice on how this can be achieved in the short term under a diversity of funding settings. In this report, we argue that frontier performance can be achieved by a wide range of institutions through Continual Learning on readily available open-weight models. As opposed to existing limited approaches such as small-scale fine-tuning, prompt engineering, or tool-augmentation with a frozen model, our Continual Learning approach takes advantage of the effectiveness of a modern mid- & post-training stack while introducing safeguards preserving both plasticity and stability at each training stage and seeking to make the minimal number of high-impact interventions on the parameters. This strategy results in model improvements comparable to the gains typically seen across multiple successive model generations. Crucially, such results are achievable with compute and personnel budgets substantially lower than commonly thought, making ownership of large parts of the SovereignAI stack (model, tool infrastructure, values & data privacy) viable for a wider range of actors. To demonstrate this, we introduce Thomson, a new general-purpose frontier model trained with an enhanced focus on high-stakes professional work: domains commonly predicted to undergo large productivity improvements through AI. Through a unique focus on Continual Learning, data-centricity, and efficiency, we demonstrate that Thomson performs competitively with recent frontier model

Hi I am a PhD student, My paper got accepted in EMNLP 2026, As this is my first paper I wanted some information. My professor has agreed to give the registration costs, but I am on my own for the travel and stay costs. I am currently in a Singapore university but south Asian. No funding from department. Queries: I searched and found this Call for EMNLP 2026 Diversity and Inclusion Subsidies - EMNLP 2026 and Call For EMNLP 2026 Volunteers - EMNLP 2026 , does anyone know some other kinds of grant/subsidies etc. available which can be used in general for AI conferences? How much does the D&I cover for? Will it cover the full costs or partial? Sorry If these are basic questions, but could not find answer to them in here. submitted by /u/Happy_Today_3288 [link] [留言]

I got my batch of four papers for AAAI 2027. All four papers make empirical claims, none include code, data, or anything I can actually check. Just the PDF and the checklist. AAAI-27's own rules say code/data should be provided at submission, and "we'll release it after acceptance" doesn't count as reproducibility. That said, I don't think missing code alone is an auto-reject. Saw an older thread here where someone claiming to have helped write the AAAI checklist argued reviewers rarely have time to audit code anyway, and plenty of authors have legit reasons (funding, IP) for not releasing it yet. If the paper's whole pitch is "look at these numbers" and I can't verify them, that tanks my confidence score even without a hard reject. I'm flagging it explicitly in the review and asking for anonymized code in the rebuttal. How's everyone else handling this round? Auto-ding for no code or does it depend on how much the paper leans on the empirical results? submitted by /u/SimpleObvious4048 [link] [留言]

By zipflow.xyz This is an independent technical analysis of DeepSeek's public research and documentation. It is not an official DeepSeek statement, and it does not claim that the current Vision-Exp API is available through our upstream channel. When DeepSeek released deepseek-v4-flash-vision-exp , the obvious story was that a text-focused model had finally gained native image input. The more useful story is longer: DeepSeek had already spent years exploring visual data, vision-language alignment, OCR, charts, documents, and unified visual understanding and generation. This article reconstructs that public research lineage and separates three things that are often mixed together: What DeepSeek's papers actually disclose What the current API documentation says What we still cannot verify about the newest model's training data 1. DeepSeek-VL: starting from real-world visual data DeepSeek-VL's 2024 paper, Towards Real-World Vision-Language Understanding , did not frame vision as only a captioning problem. It explicitly targeted practical inputs such as web screenshots, PDFs, OCR, charts, and knowledge-oriented visual content. The project also described a taxonomy derived from real user scenarios. That taxonomy was used to build instruction-tuning data for tasks including recognition, transcription, conversion, analysis, commonsense reasoning, logical reasoning, multi-image comparison, and safety-related prompts. The model family combined three major pieces: A hybrid vision encoder A vision-language adaptor A DeepSeek language model The hybrid encoder paired a lower-resolution semantic branch based on SigLIP-L with a higher-resolution branch derived from a SAM-B-style encoder. The design goal was practical: global semantic understanding is not enough for small text, dense documents, OCR, and visual grounding. The three-stage training recipe The paper described a staged approach: Adaptor warm-up: train the vision-language adaptor while the primary vision and language comp

Classic Machine Learning Through the Eyes of an SRE — Part 8 The most dangerous output in my whole Week-1 study set wasn't a bad prediction. It was a beautiful tree. Hierarchical clustering produces a dendrogram, that elegant diagram where every account, ticket, or incident nests inside ever-larger families. It looks like discovered truth. Stakeholders lean in. Someone screenshots it for the QBR deck. Nothing else in the set looks as convincing while being as capable of being completely wrong. A bad K-Means gives you blobs that feel arbitrary, and people push back. A dendrogram built with the wrong linkage on flat data still looks like a family tree of your business. Nobody pushes back on a tree. The bet and the build Hierarchical clustering completes the answer-finding taxonomy I've been using through this series. That's my own shorthand, not standard terminology: K-Means SEARCHES, DBSCAN DEFINES, PCA SOLVES, and hierarchical clustering BUILDS. Start with every point as its own cluster. Repeatedly merge the closest two clusters. Never undo. Greedy and irreversible, a little like growing a decision tree. Same skeleton, different family. There is also a top-down version, called divisive clustering, which starts with everything together and splits it. In practice, when people say hierarchical clustering, they're usually talking about the bottom-up, agglomerative version. Two things were genuinely new to me. You choose the cut after seeing the structure. Fitting doesn't require you to decide K upfront. The dendrogram gives you the hierarchy, and you choose where to cut it to get the number of clusters you want. That makes the output unusually flexible. For a delivery organization it also feels natural, because account family → sub-segment → individual account is already how a lot of governance gets organized. Linkage is a selectable worldview. "Closest clusters" needs a definition, and every definition makes a different assumption. Ward pushes toward compact, variance-

Tired of explaining the same things again and again to your AI Agent? Frustrated because the AI keeps forgetting minute things custom to your codebase which needs to be kept in mind in each change? This is the current scenario for most people using AI agents to build their software. You handoff a task to it, it gives back the solution but misses something. You explain that to it, it nods back and then does it again. I myself did it until i came to know about Skills. What are Skills? Remember the CONTRIBUTING.md file we find in almost every open source repository? The file which explained anyone coming to the repo what to check, understand and keep in mind when contributing to it so that you don’t break it. The Skills works like that for any AI Agent who is going to make changes in your codebase. Its a folder that your AI checks anytime it needs to perform a specific task, specialized jobs or multi-step workflows without requiring you to prompt every time. And the best thing is, it follows an open standard that works with almost every AI agent be it Claude Code, Cursor, Copilot and more. It follows a folder-based structure around a SKILL.md file containing YAML metadata about that skill and instructions for that in markdown. How to build a Skill? Skills can vary from simple instructions to multi-step workflows depending on your need and there are 3 ways (limited by my knowledge) to build a skill: Manually First you need to create a dedicated folder for your skill and place a SKILL.md file inside it. This file needs to have 2 things: YAML frontmatter for metadata( name & description ) Instructions in markdown. Below is a basic sample SKILL.md file for your reference: — - name: word-counter description: Counts the total number of words in a given text. — - Word Counter Instructions Take the user’s input text. Count the total number of words. Return only the final word count as a number. Using a generator/CLI It is a tooling interface (command-line or script) which can

Originally published at ai.bedvibe.studio . I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it. Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS. So I wrote the QA framework I wished I had, packaged it as ttsproof , and then ran it as a blind study against a production TTS service so the results would be more than an opinion. The two failures WER cannot see A TTS pipeline breaks in two different ways, and a single WER number blurs both. Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform. Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names. ttsproof splits them apart and handles each one honestly: Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else. Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error. ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS. The study: 390 samples, and a blind human check I evaluated the method

Originally published at ai.bedvibe.studio . There is an assumption almost every practitioner carries without examining it: if your dataset has bad samples in it, the loss will tell you. Corrupted rows spike. Broken files stick out. Sort by per-sample loss, look at the top of the list, and there is your garbage. I believed it too. Two separate failures in my own work say it is wrong, and they fail in the same direction — quietly. The reproducible one: a dataset that cannot be learned While validating trainproof I ran a controlled fault-injection study: one base setup, a Qwen2.5-3B QLoRA, run six ways, three seeds each, eighteen runs total. Every log ships in the repo so the verdicts can be checked rather than believed. One configuration shuffled the dataset's labels into pure noise. The labels no longer corresponded to the inputs at all. This is not a hard dataset or a noisy dataset. It is a dataset that cannot be learned , because there is no relationship left in it to learn. That run reduced its loss by 62%. On its own curve it was textbook-healthy — a clean downward slope, no spike, no plateau, nothing a human or a rule would flag. It was learning nothing useful. It was memorising the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve it is indistinguishable from a real one. That is where the assumption broke for me. Not "loss is a weak signal for this." Loss is not a signal for this at all, in isolation. The production one, and what I can and cannot prove about it The second failure came from real work rather than an experiment, and it is the one I think about more. Building a text-to-speech corpus of roughly 110,000 recordings, a small number of the files were pure loud white noise. Not corrupted in the file-format sense — they opened fine, played fine, had valid headers and valid duration. They simply contained no speech. Just noise, at volume. They did not surface as high-loss outliers. Being precise about