Copilot's Organization-Level Custom Agents Need a Diff Review Before Rollout

Visual Studio 2026's July update added organization-level custom agents for GitHub Copilot. An organization owner can now define a custom agent that every repository in the org automatically detects and offers in the agent selector. This is powerful and introduces a rollout risk: a single agent definition affects every developer in the organization simultaneously. The rollout problem When an org-level agent is published, every developer working in any repo in that org sees it in their agent selector. If the agent definition contains an incorrect instruction, a broken tool reference, or an overly permissive scope, it affects all developers at once. There is no canary by default. The agent definition review checklist Before publishing an org-level custom agent, review its definition against this checklist. 1. Instruction review # agent-definition.yml (example structure) name : org-code-reviewer description : Reviews PRs for security and style instructions : | Review each file change. Flag: - SQL injection in string concatenation - Missing input validation on public endpoints - Hardcoded credentials Do not modify files. Only comment. tools : - read_file - search_code - post_comment scope : organization Check: [ ] Instructions are specific enough to be testable ("flag SQL injection in string concatenation" not "review for security") [ ] Instructions do not conflict with existing repository-level AGENTS.md files [ ] The agent does not claim capabilities it does not have (e.g., "run tests" when no tools entry supports it) 2. Tool surface audit # Extract all tool references from the agent definition rg -n 'tools:' agent-definition.yml -A 20 | grep '^\s*-' For each tool: What resource does it access? (filesystem, network, repository API) Is it read-only or read-write? Does it require elevated permissions beyond the developer's own role? A post_comment tool can create noise across every PR. A write_file tool can modify code in every repo. Audit accordingly. 3. Canary test Be