今日已更新 310 条资讯 | 累计 24969 条内容
关于我们

标签：#opensource

找到 1635 篇相关文章

开源项目

🔥 0x4m4 / hexstrike-ai - HexStrike AI MCP Agents is an advanced MCP server that lets

GitHub热门项目 | HexStrike AI MCP Agents is an advanced MCP server that lets AI agents (Claude, GPT, Copilot, etc.) autonomously run 150+ cybersecurity tools for automated pentesting, vulnerability discovery, bug bounty automation, and security research. Seamlessly bridge LLMs with real-world offensive security capabilities. | Stars: 9,216 | 38 stars today | 语言: Python

2026-06-04 原文 →
AI 资讯

What is an LLM evaluation harness? A deep dive into lm-eval-harness

What is an LLM evaluation harness? A deep dive into lm-eval-harness You fine-tuned a 7B model. It aced your smoke tests, your colleague ran a few prompts and shrugged approvingly, and the README is now full of cherry-picked outputs that look great in a screenshot. Then someone asks: how good is it, really? — and you realize you have no number to point at. No MMLU score. No HellaSwag. Nothing reproducible, nothing you can defend in a PR review, nothing you can compare to last week's checkpoint. That's the gap an evaluation harness fills. It turns "vibes-based evaluation" into something with a score, a stderr, and a config file you can re-run next Tuesday. Why evaluate LLMs at all? Two reasons that actually matter: Comparability. If you can't put a number on a model, you can't compare it to anything else — not the previous checkpoint, not the open-source baseline, not the commercial API you're trying to replace. Leaderboards are noisy and gaming-prone, but a local leaderboard with the tasks you care about is one of the most useful artifacts a team can build. Regression detection. Most model regressions are silent. A 0.3-point drop on MMLU won't show up in a chat session, but it will show up in CI. People who ship models for a living treat evals the way backend engineers treat unit tests: mandatory, run on every PR, and blocking on regressions. You don't need a hundred benchmarks. You need the three to five tasks that map to your actual use case , plus one or two general capability anchors (MMLU, HellaSwag) so you can sanity-check that you didn't accidentally destroy basic reasoning while you were tuning for your domain. What is an "evaluation harness"? An evaluation harness is the software that sits between a model and a benchmark. It handles the boring-but-critical parts: loading the model weights, tokenizing prompts in the way the benchmark expects, running inference, extracting the answer from a longer generation, scoring it against a ground-truth key, aggregating

2026-06-03 原文 →
← 上一页 下一页 →