Code mode helps smaller AI models call tools reliably

Language models can interact with external tools—search APIs, bash commands, databases—in two ways. The traditional approach is to have the model generate a "tool call" signature, a structured command like search_web("query") or bash("ls -l") that directly invokes a function. This method works well for powerful models trained extensively on tool-calling syntax, but smaller language models frequently make mistakes generating these signatures because the training required to teach them proper formatting is tedious, and many models have been released without tool-calling ability altogether.

Code mode flips the problem. Instead of asking a model to output a formatted tool signature, developers ask it to write Python or TypeScript code that calls the same tools. Since smaller models are already well-trained on code from their ingestion of code repositories, they succeed far more often—particularly when a task requires multiple tool calls chained together. The model generates executable code, which is then run in an ephemeral (temporary) Linux sandbox environment to avoid security risks. Execution overhead is minimal; most sandbox providers start up in under 200ms, adding no noticeable delay to inference.

Anthropic has adopted code mode to reduce token spending significantly, a benefit that compounds the primary advantage of improved tool-calling reliability. Microsoft's recent recommendation to use specially trained models for specific tasks creates an additional use case: code mode allows developers to deploy smaller, cheaper models without having to retrain them on tool-calling syntax. The tradeoff is the small risk of executing malformed code, but sandbox isolation contains that risk. The approach thus offers a double benefit for cost-conscious deployments: it works better for smaller models out of the box, and it reduces the total tokens consumed in the process.

Code mode addresses a practical gap in smaller language models' abilities. While large language models have been trained extensively to format tool calls correctly, smaller models—which offer cost and latency advantages—often fail at this task because the training required to teach them the syntax is resource-intensive. The insight behind code mode is that these same smaller models are already strong at code generation, a skill they picked up naturally during their training on code repositories. By reframing the problem—asking the model to write executable code that calls tools rather than generate a tool-call signature—developers sidestep the need for retraining while improving reliability, especially in complex scenarios with multiple tool calls.

Anthropic's adoption of code mode signals a shift toward pragmatic architecture choices that balance capability with cost. The approach delivers a second benefit: reducing token consumption, which directly cuts inference expenses. For organizations considering smaller, task-specific models (as Microsoft has recently advocated), code mode removes one of the traditional barriers—the assumption that model specialization requires custom training. Instead, it leverages existing model strengths in a way that aligns with how developers already write and think about code.

No comments yet. Be the first to share your thoughts!