Run and Compare AI Evaluations with a CLI for Developers and Coding Agents

TL;DR: This walkthrough shows how developers and coding agents can use Quantiles , an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow. Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it. Quickstart: Run an example benchmark The Quantiles CLI is called qt on the command line. A simple curl ... | bash command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI: curl -fsSL https://cli.quantiles.io/install.sh | bash If you don't want to run code directly sourced from the internet, see the install.sh source code first. Next, let’s run a built-in benchmark from start to finish using a single command. SimpleQA Verified is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI's SimpleQA benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in problem , its reference answer , topic and answer-type metadata, and supporting URLs. Use the following command to run simpleqa-verified using the built-in Quantiles demo model, which doesn't incur any usage charges: qt run simpleqa-verified Results from the demo model are intended only to demonstrate the evaluation workflow because its output