标签：#MachineLearning

Hi, How much does adding an honest limitations section hurt the paper (apart from making it better)? Does it bias the reviewers? Will they want you to fix the things in the limitations section? If the reviewers let AI read the paper, will the limitations section bias AI? Would it be better if the limitations section was hidden from the reviewers? And if the reviewers would have to author a limitations section? submitted by /u/strammerrammer [link] [留言]

When you build a system that detects key moments in a live sports feed, one architectural decision shapes everything downstream: where does the inference happen? At the edge, close to where the video is produced, or in the cloud, after the stream has been ingested? There is no universally right answer, but the trade-offs are sharp and worth laying out. The case for the edge Running detection near the source, at the venue or in an on-prem encoder, minimizes the round trip. The video does not have to travel to a data center and back before a moment is flagged, which can shave critical seconds off the time to clip. For genuinely live use cases, where a clip is worthless if it lands a minute late, that latency saving is the whole game. Edge inference also reduces egress: do the heavy frame analysis locally and ship only the clips that matter, instead of streaming everything to the cloud. The cost of the edge Edge hardware is constrained. You run on whatever GPU or accelerator fits in the rack at the venue, not on an elastic fleet. That bounds model size and concurrency. Updating models across many distributed edge nodes is an ops problem in itself, and a venue that hosts one event a week is idle hardware the rest of the time. Edge is fast but inflexible. The case for the cloud The cloud gives you elastic compute, easy model updates, and the ability to run larger or ensemble models you could never fit at the edge. If you process many concurrent streams, centralizing inference pools capacity instead of overprovisioning every venue. For workflows where a few seconds of extra latency is acceptable, near-live rather than instant, the cloud is simpler to operate and cheaper to scale. The cost of the cloud You pay for it in latency and bandwidth. Every frame you want to analyze has to be ingested first, and for high-bitrate broadcast feeds that adds up. The end-to-end path, capture, encode, transport, ingest, infer, clip, deliver, has more hops, each adding delay and a potenti

Hey! Can you tell me how you would go about building a recommendation engine for our question bank? The idea is that it understands a student’s strengths and weaknesses and recommends questions accordingly — more questions around the areas they’re weak in, but without making them so difficult that they feel demotivated. I also want it to occasionally bring back questions from older topics to check whether the student has forgotten something. Based on how they perform, it could then decide whether to recommend more questions from that topic or move on. Basically, the goal is for the recommendation engine to continuously understand where the student is struggling and use that to help them become better at problem-solving over time. I was learning some basics of AI/ML and this question came to my mind, so I was just curious — do you have any idea how something like this could be built? submitted by /u/whizzkidme [link] [留言]

I built a RAG assistant, then found out my architecture change made it worse, and I'm glad it happened I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, "Helix," designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60. But the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out. The setup The pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic ( answer , sources , confidence ), using gpt-4o-mini at temperature=0 , because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here. Chunking wasn't one-size-fits-all. Three formats needed three strategies: Markdown docs were split by header first, so a chunk never crosses a topic boundary, with a recursive splitter as a fallback for long sections. PDF runbooks (no header structure to exploit) got a straight recursive character split. HTML tickets were kept as one whole chunk per ticket whenever possible, because a resolution often only shows up in the final turn of the conversation, and splitting a ticket risks separating the question from its answer. 5 scanned PDFs with no extractable text layer were detected and skipped gracefully rather than OCR'd, a conscious call I'll come back to. Result: 95 of 100 document

Most classification metrics for oncology AI models (AUC, ICC, MAE) measure global agreement. They don't answer the question that actually matters at the point of care: how reliable is this model at the exact cutoff that decides whether a patient gets flagged, biopsied, or treated? I built oncothresh to evaluate models at a specific clinical threshold rather than in aggregate: sensitivity/specificity/PPV/NPV at the cutoff, bootstrap confidence intervals, threshold-sensitivity curves, boundary-weighted calibration, decision-curve net benefit, and number-needed-to-test. It's a small, dependency-light Python library (numpy/scipy/scikit-learn/pydantic) built for tasks like tumor cellularity, Ki-67, TMB, and PD-L1 scoring, where a continuous model output gets collapsed into a yes/no clinical decision at a fixed cutoff. Pathology-specific benchmarks like PathBench and PathBench-MIL evaluate foundation models globally but don't evaluate at predefined clinical thresholds with uncertainty quantification, which is the gap this fills. There's also a companion web dashboard ( oncothresh-web ) for people who want the same analysis without writing code: upload a CSV of predictions and labels, pick a threshold, get the full set of charts plus a downloadable PDF report. docker compose up and it's running locally, no cloud dependency. Library: github.com/omkaradhali/oncothresh Dashboard: github.com/omkaradhali/oncothresh-web Still v0.1, so I'd genuinely welcome feedback: use cases I haven't considered, edge cases in the DCA/calibration math, or places the API doesn't fit how people actually work with threshold-based models. submitted by /u/adom2989 [link] [留言]

This is the project my last two posts were building towards (this is the last of this silliness). I ported the Doom rendering algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in Hugging Face without trust_remote_code -- it's just a standard transformers checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame. The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself. One frame is a 3,614-token prompt plus 53,747 generated tokens -- just over 40 minutes on a B200. The original Doom could achieve 35 FPS on a 486. This achieves 35 FPD (frames per day) on a B200. Write-up: https://ood.dev/posts/doom/ Weights: https://huggingface.co/physicsrob/torchwright-doom-e1m1 Github for the source code which gets compiled: https://github.com/physicsrob/torchwright_doom/ submitted by /u/notforrob [link] [留言]

Implied vs Realized Volatility: Reading the Gap By Shakti Tiwari · Educational only · Not investment advice This article explains implied vs realized volatility: reading the gap from first principles. No live market numbers are quoted; the structure is what lasts. Why this matters Implied vs Realized Volatility: Reading the Gap is one of those subjects that sounds simple until you implement it, at which point the hidden complexity appears. The first version works on a laptop with a tiny file; the second version breaks at 3am when the WebSocket drops, the replay file is half-written, and you cannot tell which ticks you already stored. This article is a structural walkthrough: the concepts, the math where it helps, the code shape where it helps, and the failure modes that quietly cost money or correctness. No live market numbers are quoted because a number without a dated source is decoration, not education. The structure here does not expire, and unlike a specific price level, you can reuse it on the next dataset without re-deriving anything. If you only remember one sentence from this page, make it this: the boring parts are the product, and the interesting parts are a small fraction of what separates a demo from a system. Core concept At its heart, implied vs realized volatility: reading the gap is about being honest with your own assumptions. The trap is not that the idea is wrong; it is that a half-implemented version looks right in a demo and breaks in production. We separate the idea from the implementation so you can tell which one you actually have. A clean concept on paper can still produce a broken system if the boundary between 'what I meant' and 'what the code does' is never made explicit. Write the concept as a contract: given X observable at time t, the system produces Y, and any deviation is a bug, not a feature. A contract you can state in one sentence is also one you can test in one assertion, and that testability is the entire difference between an

Been working on this for the last few months. I've been working on PyTorch for the past few years and I always felt, many a times my work went into dump, because of some mistakes I made in the code. torch-preflight reads your PyTorch code and catches the bugs costing you GPU hours. Things like losses.append(loss), which holds the autograd graph from every step until CUDA dies on you or no zero_grad() in the loop or gradient accumulation without dividing the loss or DDP with no DistributedSampler, so every rank trains on the same batches. I've been able to get 13 rules so far. Your code never gets imported or executed, so you need no GPU and no torch install. There's another part to this that estimates VRAM. Point the tool at a training script and a GPU, and you learn whether the run fits before you pay for the instance. You also get the list of changes to make the run fit, with the GiB each one saves. pip install torch-preflight https://github.com/highwaterlabs/torch-preflight https://pypi.org/project/torch-preflight/ Please try this out, and I would like to get your feedback! It's still a work in progeress. Would like to know what breaks on your code. False positives kill a linter, and my only large test target so far has been the PyTorch source tree. Same for the memory numbers. Mine land within 4% of measured peaks, but from four models on one T4. PS: open to contributions, and issues are already open on the repo. Soon I'm going to add a few "Good first issues" as well. Feel free to ping me if you have any questions! submitted by /u/LeJanbandhu [link] [留言]

Hello, I was curious about the differences you can get from the human reviewers and the llms. Any insight is welcome, thank you! submitted by /u/obliviousphoenix2003 [link] [留言]

i wanna build a text diffusion model which interpret text and convert it into ascii images so like Text : build a cat Output : /\\\_/\\ ( o.o ) \> \^ < So , i have a decent background of ml algo ( completed cs229 , cs230 , Ml architecture and basic CNN and diffusion model ) ik making a project like this is tricky and making diffusion model like that from scratch is hard but i wanna try it because that's wot make me excited lol ... I am currently reading GANs research paper , can u guys help me in finding more papers which helps me in making this project or guide me through this good title for this Thx in adv submitted by /u/Udbhav96 [link] [留言]

A hands-on test of BGE-M3 + Qwen3 (RAG vs. direct-context answering) on a real research paper and a full-length book including a retrieval bug hiding in a footnote, and one surprisingly good model behavior. I wanted to answer a simple question: when you feed a document to an AI model, is it actually reading it or just pattern-matching to whatever text happens to look similar to your question? So I built a small open-source pipeline to test this directly. For any document and question, it generates two separate answers: RAG answer: BGE-M3 finds the most relevant chunks of the document, and Qwen3 answers using only those chunks. Direct answer: Qwen3 reads the raw document text directly, no retrieval involved. Both run on a free Google Colab GPU. I kept the retrieval side deliberately "vanilla" fixed-size chunking, plain cosine similarity, no reranking, no fancy tricks so I could see exactly where the basic version breaks before adding any fixes. Before running my first real test, I already knew one thing to guard against: reference lists. Early experimentation (not covered here) showed that a paper's bibliography, once chunked like any other text, can get retrieved as if it were real content a citation for a paper about "text embeddings" can look deceptively similar to a generic question about a document's topic. So going in, my pipeline already strips everything after a References/Bibliography heading before chunking. With that fix in place, I ran two real tests. Test 1: A research paper on Nepali legal machine translation First document: a SIGUL 2024 workshop paper on a bidirectional English-Nepali machine translation system for the legal domain. Question: "What is this paper about?" RAG answer: This paper presents the first transformer-based bidirectional machine translation system for the English-Nepali legal domain, using a custom-built parallel corpus of 125,000 sentences. It achieves encouraging BLEU scores and addresses the scarcity of domain-specific legal tr

I recently had a paper accepted to TMLR and was wondering how prestigious it is, in comparison to A* conferences (ie. NeurIPS, ICLR, ICML), but also vs journals like JMLR. submitted by /u/Awesome_Nerd10 [link] [留言]

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry

City2Graph is a Python library I built that turns geospatial data into analysis-ready graphs (for spatial analysis, network analysis, and Graph Neural Networks as GeoAI), and the paper describing it has just been published, so I wanted to share it here. Repository: https://github.com/c2g-dev/city2graph import city2graph as c2g # buildings + street segments -> heterogeneous morphological graph nodes, edges = c2g.morphological_graph(buildings, segments) # straight into PyTorch Geometric data = c2g.gdf_to_pyg(nodes, edges) What it covers: Morphology : graphs of buildings, streets, and tessellated urban fabric from OpenStreetMap and Overture Maps Transportation : GTFS and GBFS feeds loaded through DuckDB, with GTFS aggregated into stop-to-stop transit graphs Mobility : OD matrices and flow data (migration, bike-sharing, pedestrian counts) as weighted spatial graphs Proximity and contiguity : KNN, Delaunay, Gilbert, Waxman, plus queen/rook contiguity, under Euclidean, Manhattan, or network distances Heterogeneous graphs and metapaths : several node and edge types in one graph, with metapath-derived edges composing relations across them Conversion : round trips between GeoDataFrames, NetworkX, rustworkx, and PyTorch Geometric Data / HeteroData , with geometries and attributes kept intact It sets out why urban data is better treated as heterogeneous graphs than as flat feature tables, how the morphological, transport, mobility, and proximity constructions relate to each other, and how the library keeps geometry and graph structure consistent across conversions. If you use the library in research, that is the citation. Paper Sato, Y., Pietrostefani, E., Mahabir, R., & Arribas-Bel, D. (2026). City2Graph: A Python library for Heterogeneous Graph Neural Networks and spatial analysis in urban systems . Computers, Environment and Urban Systems , 130, 102492. Happy to answer questions about the design, and issues or PRs are very welcome. I am especially keen to hear which data so

Almost every graph starts life as relational tables. The conversion is mechanical once three decisions are made, and one of the three — id remapping — is a silent correctness bug rather than a matter of taste. Deciding what is a node Start with three tables: customers (customer_id, region, signup_date, tenure_days), products (product_id, category, price), and orders (order_id, customer_id, product_id, amount, ordered_at). The rule that resolves nearly every case: A table with a primary key that other tables point at is a node type. A table whose whole job is to link two keys is an edge type. So customers and products are nodes, and orders are edges — even though orders has its own primary key. The order id is not an entity you want to reason about; it is an identifier for a relationship. The harder case is a repeated categorical column such as region . It can stay a customer feature, or it can become a node type with a customer–region edge. The test is behavioural, not aesthetic: do you want information to flow between rows that share this value? As a feature, region is a tag on each customer and nothing more. As a node, it creates a two-hop path between every pair of customers in the same region, so their representations start blending. If a region contains 400,000 customers, that node is a hub through which everything mixes, which is usually a way of turning four hundred thousand distinct customers into one regional average. Keep high-cardinality-of-membership categoricals as features; promote a category to a node when its membership is small and meaningful. If you end up with more than one node type, the model has to change too — see heterogeneous graph neural networks . The id remapping nobody warns you about Graph libraries do not store your ids. They store a node feature matrix and an edge index of integer positions into it, because a message-passing layer is a gather over rows of a dense array. So node ids must be contiguous integers from 0 to n−1 , per node

You put “use British English spelling” in the system prompt. The first three paragraphs are fine. By paragraph nine there is a color , and by the end there is an organization . The instruction was not ignored; it was outvoted. The symptom The characteristic pattern is not uniform failure. It is a document that starts correct and degrades — and the degradation is usually inconsistent within the document, so you get colour in one paragraph and color two paragraphs later, sometimes in the same sentence as behaviour . Long outputs are worse than short ones, and a long conversation is worse than a single call. A second symptom is domain-specific: the spelling holds in ordinary prose and fails in technical contexts. Code comments, API field names, CSS properties and library names are American by convention ( color is a CSS property; serialize is what the method is called), and text near them pulls the surrounding prose across. Both patterns point at the same cause, and it is not that the model did not read the instruction. Why it drifts back Each token is sampled from a distribution conditioned on everything in the context. The system prompt is part of that context, but so are the two thousand tokens the model has generated since, and so is the enormous prior from training data in which American spelling outnumbers British by a wide margin in almost every technical domain. At the start of a response the instruction is close by and there is little else in the context, so it dominates. As the response grows, the local statistics of the text being generated carry more weight relative to a single instruction several thousand tokens back. And the drift is self-reinforcing in exactly the way described in mid-answer code-switching : once one American spelling is in the context, the conditional probability of the next one rises. The key insight for fixing it is that spelling is not a mode the model is in. There is no British-English state that gets set and then holds. Each word i

Brazil’s AI bill is described in a great deal of writing as though it were in force. It is not, and the distinction is not pedantic: the risk tiers, the prohibitions and the regulator that summaries attribute to Brazilian law exist only in a text that one chamber of Congress has approved. Where the bill stands PL 2338/2023 was introduced in the Federal Senate in May 2023 by the then-President of the Senate, building on the report of a commission of jurists that had been convened to draft a substitute for earlier and much thinner AI bills. After committee work through 2024, the Senate plenary approved the bill on 10 December 2024 and sent it to the Chamber of Deputies, where it has been examined by a special committee rather than passed straight to a floor vote. As at the date on this page, the bill has not been enacted. It has been approved by one chamber and remains before the other. This is a status page about a live legislative process and it is written to be checked, not relied on. It is not legal advice. Before making any decision that depends on whether Brazil has an AI statute, verify the current stage on the official tracking pages linked below—a page written at any date can be overtaken the following week. How a Brazilian bill becomes law The reason “approved by the Senate” is so frequently misreported as “passed” is that the remaining route is substantial and can change the text materially. A bill originating in the Senate goes to the Chamber of Deputies as the revising chamber. If the Chamber amends it, the amended text returns to the Senate, which decides between its own text and the Chamber’s. Only when both chambers have settled on one text does it go to the President, who may sanction it in whole, or veto provisions in part, with vetoes subject to being overridden by Congress. Each of those stages has changed the substance of comparable Brazilian technology legislation. The LGPD itself, Brazil’s data protection statute, was enacted in 2018 and then am

The instinct is to hand the whole reference list to a model and ask for an array of citation objects. On a list of eighty entries that produces seventy-three, with two merged and five hallucinated into tidiness. The fix is to make segmentation a separate, deterministic step. Two stages, and why the first one is harder Parsing one reference string into author, year, title and venue is a task current models do well. Deciding where one reference ends and the next begins is a task they do badly, because the boundary is typographic rather than semantic: a hanging indent, a numeric label, a line break that is either a wrap or a separator depending on the column width. Splitting the work also gives you a count to assert against. If the list is numbered 1 to 84 and you segmented 81 entries, you know the parse is wrong before you have looked at a single field. A single-call extraction gives you no such handle — a merged pair looks identical to a list that was three shorter. Step 1: segment the list Three reference-list styles cover almost everything, and each has a different boundary signal: Numbered (Vancouver, IEEE). Each entry begins with 1. or [1] . Boundary detection is a regex, the sequence is monotonic, and you get the assertion for free. Author-date (APA, Harvard, Chicago author-date). No labels. Entries are separated by a hanging indent — the first line starts at the margin and continuations are indented — which is invisible in a flat text stream and obvious in the layout. Note-bibliography (Chicago notes). Also unlabelled, also hanging-indented, and additionally uses a three-em dash for a repeated first author, which is the case discussed below. For the unlabelled styles, segment on the indent rather than on the text. If you have coordinates from the PDF, an entry starts at every line whose left edge is at the block minimum and continues through every line indented further. If you do not have coordinates, a reasonable proxy is a line that begins with a capital lett

If your Azure Pipelines eval gate runs on pushes to main but never on a pull request, the YAML is not the problem. Microsoft’s documentation is explicit: for an Azure Repos Git repository you cannot configure a PR trigger in the YAML file, and the functionality is implemented by a branch policy instead. Why your pr trigger does nothing The pr: key exists in the Azure Pipelines YAML schema, and it works — for GitHub and Bitbucket Cloud repositories. For Azure Repos Git it is inert. The Azure Repos Git documentation states that pull request triggers are implemented using branch policies, and that to enable PR validation you configure the Build validation policy on the target branch. A pr: block in the file is not an error and produces no warning; it simply never causes a run. Two related things surprise people once the policy exists. Draft pull requests do not trigger a pipeline even with a branch policy configured, so a gate that seems not to run may be running against a draft. And you must be a project administrator of the project to configure validation builds at all, which is why this is often the step that a developer cannot complete themselves. This is a product behaviour rather than a version detail, but it is the kind of thing that changes. Check the Azure Repos Git page in Microsoft’s Azure Pipelines documentation before assuming it still holds. The pipeline A single-stage pipeline is enough. The CI trigger below covers pushes; the pull request path comes from the policy in the next section, and no pr: key appears at all because on Azure Repos it would only be misleading to a reader. trigger : branches : include : - main paths : exclude : - docs/* pool : vmImage : ubuntu-latest variables : - group : llm-eval-keys - name : EVAL_MODEL value : gpt-4.1-mini-2025-04-14 steps : - task : UsePythonVersion@0 inputs : versionSpec : ' 3.12' - script : pip install -r evals/requirements.txt displayName : Install eval dependencies - script : | python -m evals.run \ --cases