What happened
A developer built a Go CLI tool that benchmarks large language models across OpenRouter's API on four test suites—accuracy (15 tasks), instruction following (12 tasks), tool calling (12 tasks), and JSON output (5 tasks)—each run three times for reliability. The tool compares models side by side, tracking cost per correct answer and outputting results in JSON, CSV, and Markdown formats.
Why it matters
Testing reveals that smaller models can outperform much larger ones on real-world tasks. Ling 3.0 Flash achieved 11 of 15 correct answers on accuracy tasks and nailed every instruction-following test, while Nemotron 3 Ultra—a 550 billion parameter model—scored only 9 on accuracy and struggled with basic constraints like "don't use commas." This suggests that for production work, raw parameter count matters less than a model's ability to follow instructions, call tools correctly, and output valid JSON at a reasonable cost.
What to watch
The tool is open source and freely available at github.com/cheikh2shift/go-snippets/tree/main/llm-bench. Users can run benchmarks on their own models by installing the CLI with `go install` and passing OpenRouter API credentials and model names as flags; no external dependencies are required beyond Go's standard library.
Summaries like this, in your inbox every morning.
The tool addresses a gap in how language models are typically evaluated. Academic benchmarks like MMLU and HumanEval measure expertise and coding ability on idealized tasks, but production systems care about different qualities: whether a model follows formatting constraints, calls the right functions, and returns valid structured data. By testing against OpenRouter's unified API, the benchmark avoids the friction of managing multiple SDKs and allows direct cost-per-correct-answer comparisons across models from different providers.
The results challenge the intuition that bigger models are better. Nemotron 3 Ultra, with 550 billion parameters, failed consistently on basic math and factual recall—tasks that Ling 3.0 Flash, designed for speed and instruction following, handled reliably. The instruction-following gap is particularly stark: Nemotron's inability to follow constraints like "no commas" or "exact word count" suggests that scale alone does not guarantee usability in constrained production scenarios. For teams choosing a model, the benchmark implies that effectiveness at specific real-world tasks and cost efficiency matter more than parameter count.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Cammarata reinterprets InceptionV1's cat-car neuron as a V-shape detector, not two unrelated concepts
Google Deepmind's Gemini 4 Argon scored 97.5 percent on ARC-AGI-3 at a cost of $4,800, second only to GPT-6 As…
Microsoft's Surface Laptop Ultra, using Nvidia's RTX Spark platform, starts at $2,599 (up to $5,899)
Meta's new commercial terms, effective October 30th, appear to let it crawl and train AI on content from websi…
Vercel data published Thursday shows TypeSafe AI's Jev, a lightweight decision model for choosing among predet…
Reflection AI introduced Beam, an open-weight model for agentic coding and long-horizon tasks, with 501 billio…
