Artificial Analysis launches Optima for custom AI model benchmarks

Artificial Analysis, which operates well-known LLM evaluations including GDPval-AA and AA-Briefcase, has released a new platform called Optima designed to close a gap in AI model comparison. While public benchmarks test models on predefined tasks, they often don't reveal which model actually performs best for a particular organization's needs. Optima allows users to build their own benchmarks using their specific data and workflows, then run tests across leading current models and compare results on quality, cost per task, and time per task.

The platform offers three paths to building a custom benchmark. Users can upload existing evaluation datasets from their own files or from Hugging Face, provide AI agent traces from platforms like Arize, Braintrust, or Langfuse, or install a skill that gathers information from their coding environment and past sessions. For those without ready-made data, Optima accepts a description of the intended use case along with sample inputs and outputs, then generates suggested test inputs, evaluation criteria, and example tasks for users to review and refine. Two scoring approaches are available: rubric-based evaluation against objective criteria, or a pairwise comparison method that matches the approach Artificial Analysis uses for benchmarks like GDPval-AA and AA-Briefcase, where users evaluate a sample of response pairs and indicate which answer they prefer before Optima derives the full ranking across the test dataset.

Pricing is straightforward: Optima charges only the actual token costs of models used, with no markup. Rubric-based evaluations cost $0.125 per criterion per model, and pairwise evaluations cost $0.375 per comparison. At the start of benchmark creation, each benchmark run, and each evaluation round, the platform holds a balance based on a cost estimate, then bills based on actual usage and evaluation costs incurred.

The need for custom benchmarking reflects known shortcomings in general-purpose AI evaluations. Research by Epoch AI demonstrated that benchmark results depend heavily on implementation details rarely disclosed—different prompt wording and temperature settings caused the same model to score noticeably differently, and for agentic benchmarks like SWE-bench, swapping the agent's control software and tool environment accounted for up to 15 percentage points of difference. A broader study examining 445 benchmark papers from leading AI conferences found that nearly all had methodological weaknesses in at least one area, including unclear definitions, unrepresentative samples, and missing statistical validation. Only about 10 percent used complete real-world tasks reflecting actual application scenarios. Early testers of Optima have already found value: finance and accounting teams built benchmarks to find which model could cut costs by a factor of ten without major quality loss, and others tested which model best matched the writing style of lawyers or most accurately identified elements in a proprietary image dataset.