How do you catch it when a model update changes your agent's tool calls?

Your agent calls get_weather(city="London") . The provider ships a new model version. Now it calls get_weather(location="London, UK") , your downstream parser breaks, and nothing in CI told you. I built a small library for exactly this failure: pip install toolcontract GitHub: https://github.com/Divyansh2202/toolcontract PyPI: https://pypi.org/project/toolcontract/ You pin a golden set of expected tool calls as a contract, re-run them against the live model, and get pass / fail / inconclusive with a diff showing what changed. It is not an eval framework. promptfoo, DeepEval and the rest score whether an output is good — semantic quality, usually judged by another model. toolcontract asks a narrower, cheaper question: is the tool call structurally the same as the one I pinned? Same tool, same argument shape, same trajectory. That is a regression test, not an eval, and it is the question that matters when a provider bumps a version under you. Details: pass / fail / INCONCLUSIVE — anything the structural comparators cannot resolve is never silently turned into a pass or a fail trajectory matching: strict, unordered, subset, superset optional argument support, so you can assert a field must stay absent works with OpenAI, Anthropic, anything OpenAI-compatible, or via LiteLLM thin pytest plugin, and a CLI that produces the same verdicts without pytest MIT Happy to hear where this breaks. It's v0.1.