OpenEval: Why LLM Evaluation Needs a Standard Format

Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem (checking whether a model's output is correct) but none of them can read each other's eval datasets. That means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over. OpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it. What's in the repo: A versioned JSON Schema spec for test cases, grader configs, and result records. TypeScript and Python SDKs for reading and writing OpenEval-formatted datasets. A CLI with validate, convert, init, and summarize commands. Converters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite. The project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered. Repo: https://github.com/adhabnr-ux/openeval Would love feedback from anyone who has hit this same portability problem while switching between eval tools.