Open-Ultra routes coding-agent requests to cheaper models while maintaining quality

Open-Ultra is an open-source proxy that sits between a coding agent and backend AI models, transparently forwarding requests to a frontier model while running them in parallel through cheaper alternatives in a shadow lane. The tool intercepts requests byte-for-byte (preserving auth headers and latency), stores primary request/response pairs, and simultaneously replays each request to shadow models configured in pool.json. An agentic grader then scores the shadow output against the frontier's actual response—either comparing prose directly or, in worktree-exec mode, running the shadow model agentically in an isolated git worktree and comparing the resulting code diff. These graded comparisons are logged as pairs.jsonl, which a training step uses to fit per-model capability heads on a conversation-grouped time split. Once a router passes an evaluation precision gate, routing turns on, and the proxy begins sending traffic to the cheapest eligible model that is currently available and has scored at least as well as the frontier on that task type; any failed routed call falls back to the frontier transparently. The tool is installed via uv (uv tool install open-ultra) and attached to coding agents via three methods: per-session launch wrappers (openultra claude), persistent config edits (openultra attach claude), or manual proxy configuration. Supported agents on day 1 are Claude Code, Codex, and OpenCode. The proxy speaks Anthropic Messages and OpenAI Chat Completions / Responses inbound and routes to any OpenAI-compatible endpoint outbound, covering OpenRouter, vLLM, Ollama, SGLang, Together, DeepSeek, and similar stacks. Setup is a one-time init command (openultra init) that collects an API key, picks a pool preset (free, budget, or balanced), and runs a connectivity check; configuration is stored in ~/.openultra/ as env, config.json, pool.json, traffic.jsonl, pairs.jsonl, and router/ weights. The shadow lane doubles request volume but on models costing 1–3% of frontier prices, with a configurable spend cap that pauses the lane when hit. The tool is designed to be local-first (traffic goes only to configured endpoints, logs stay on the machine), invisible (shadow requests never touch your session or files, and worktrees are throwaway and isolated), and safe (routing only turns on after passing the gate, and defaults to frontier on any failure). The roadmap includes stronger features, epsilon exploration so cheap models continue earning re-verification after routing is live, versioned routers with hot-swap and rollback, systemd/launchd service units, and an offline end-to-end test suite is already included.

Open-Ultra addresses a concrete friction point in agentic workflows: frontier models like Claude or GPT-4 are expensive to run continuously, yet many coding tasks do not require frontier-level reasoning. The tool's core insight is to route dynamically based on graded evidence from your own live traffic rather than relying on synthetic benchmarks or static performance claims. By running a shadow lane invisibly in parallel—re-running requests on cheaper models and comparing their output to what the frontier actually produced—Open-Ultra lets cheap models earn routing privileges incrementally. The design defaults to safety: routing stays off until a model passes the precision gate, and any routed request that fails falls back to the frontier transparently. This means the worst case is the status quo (paying frontier prices for everything), and the upside is significant cost reduction for tasks where cheaper models prove themselves on your workload. For teams running coding agents at scale, this calibration to live work rather than abstract benchmarks is a departure from how model selection typically works.

No comments yet. Be the first to share your thoughts!