gpt-5.6-luna wins on cost per outcome, not per token

The starting point for this work is a common habit among organizations building generative AI applications: comparing models on dollars per million tokens because it is the number on every pricing page. The article argues that production workloads actually buy outcomes, and between the pricing page and the outcome sit multipliers the sticker price ignores — how often the model is right, how many tokens it needs to get there, and, for agentic workloads, how many turns it takes. Each extra turn re-sends the growing conversation, so a model that finishes in five turns instead of eight can save more than the reduction in turns alone suggests.

The benchmark harness, called openai-on-aws/benchmarks-openai, runs one identical code path (the OpenAI Responses API) against both Amazon Bedrock and the OpenAI API, switching the backend and model ID while holding evaluation logic constant. The article is explicit that the results reflect differences in the models, provider infrastructure, and model-specific configuration, and that the Amazon Bedrock models ran with reasoning disabled as a deliberate cost floor. Grading combines deterministic checks with an LLM judge using frozen prompts whose hashes are recorded in every result file.

The findings across single-call accuracy, agent trajectories, and professional deliverables consistently point to token and turn efficiency as decisive pricing variables that are invisible on the pricing page. The article also notes that pricing pages move — the July 2026 GPT-5.6 reductions being a case in point — and that the durable part of the work is the methodology, not any single ranking. Whether luna remains the lowest-cost option for a given team hinges on whether its observed efficiency holds on that team's own tasks and configuration, which is why the authors recommend re-running the evaluation rather than assuming the sample results transfer.