GitHub found that upgrading Copilot code review to use better-maintained shared tools initially made performance worse—higher review costs and fewer issues caught. The problem was not the tools themselves, but instructions that guided the agent to broadly explore the repository like a coding assistant rather than focus narrowly on the changed code like a reviewer. After rewriting the instructions to prioritize targeted searches and minimal context, GitHub achieved roughly 20% lower review costs while maintaining quality.
Summaries like this, in your inbox every morning.
Sign up free →What happened
GitHub migrated Copilot code review to use shared code exploration tools (grep, glob, view) from the Copilot CLI harness, expecting better performance. Instead, benchmarks showed higher costs and fewer issues caught. After rewriting the tool instructions to match how reviewers actually read pull requests—starting from the diff and using narrowly targeted searches—the regression flipped into a win: roughly 20% lower average review cost while maintaining the same review quality.
Why it matters
The fix reveals that agent performance depends less on tool quality than on how those tools are instructed to behave. The shared tools were sound, but their original instructions guided the agent toward broad repository exploration instead of focused diff-based review, wasting context tokens. For teams relying on Copilot for code review, this means the product should now catch issues more efficiently without adding cost.
What to watch
GitHub's reframing of tool instructions for code review—from generic coding-assistant guidance to review-specific workflows—suggests a broader principle for agent design: the same tools can fail or succeed depending on the instructions shaping their use. The company framed this as a lesson in agent-framework design, indicating it may apply these insights across other Copilot products that share the CLI harness.
GitHub's experience highlights a common pitfall in agent design: inheriting tools from a parent framework without adapting the instructions that control how those tools are used. The company's custom code review tools were designed when earlier AI models made fewer tool calls and benefited from having surrounding context automatically included. As models improved, those assumptions no longer held, yet the tools persisted because they worked well enough. When GitHub attempted to migrate to shared infrastructure—a sensible engineering goal to reduce duplication and enable cross-product improvements—the team expected a straightforward upgrade. Instead, benchmarks revealed a performance regression caused not by tool quality but by instruction mismatch.
The fix required reimagining the agent's workflow to mirror how human reviewers actually work. Rather than exploring broadly and pulling in context speculatively, the rewritten instructions prioritize narrow investigation: form specific questions from the diff, batch cheap searches before reading files, and use file-reading only once the agent knows exactly what it needs. This approach treats tool results not as disposable outputs but as context tokens that persist in the agent's working memory—a crucial insight for constrained inference budgets. The result was a win on both metrics: lower cost and maintained quality.
The broader implication is that agent performance tuning may require instruction engineering as much as tool engineering. GitHub's framing of this as a design lesson suggests the company views it as a scalable principle applicable across multiple Copilot products that share the CLI harness, potentially improving efficiency wherever agents inherit general-purpose tools for specialized tasks.
No comments yet. Be the first to share your thoughts!
Log in to join the discussion
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
