GitHub improved Copilot code review by rewriting tool instructions, not swapping tools

GitHub's experience highlights a common pitfall in agent design: inheriting tools from a parent framework without adapting the instructions that control how those tools are used. The company's custom code review tools were designed when earlier AI models made fewer tool calls and benefited from having surrounding context automatically included. As models improved, those assumptions no longer held, yet the tools persisted because they worked well enough. When GitHub attempted to migrate to shared infrastructure—a sensible engineering goal to reduce duplication and enable cross-product improvements—the team expected a straightforward upgrade. Instead, benchmarks revealed a performance regression caused not by tool quality but by instruction mismatch.

The fix required reimagining the agent's workflow to mirror how human reviewers actually work. Rather than exploring broadly and pulling in context speculatively, the rewritten instructions prioritize narrow investigation: form specific questions from the diff, batch cheap searches before reading files, and use file-reading only once the agent knows exactly what it needs. This approach treats tool results not as disposable outputs but as context tokens that persist in the agent's working memory—a crucial insight for constrained inference budgets. The result was a win on both metrics: lower cost and maintained quality.

The broader implication is that agent performance tuning may require instruction engineering as much as tool engineering. GitHub's framing of this as a design lesson suggests the company views it as a scalable principle applicable across multiple Copilot products that share the CLI harness, potentially improving efficiency wherever agents inherit general-purpose tools for specialized tasks.