Kimi K3 inference costs ~$1.37/M tokens—but providers charge $15

The article constructs a framework for understanding LLM inference profitability by first establishing how tokens are priced and produced. Tokens are billed by the million, with input tokens (the prompt) and output tokens (model predictions) priced separately; cached input tokens—those stored in the GPU's key-value cache after the first forward pass—are cheapest because they require no compute, only a cache lookup. The foundational formula is simple: cost per million tokens equals the hourly GPU rental rate divided by tokens produced per GPU per hour, multiplied by one million. The critical question then becomes: what determines tokens_per_gpu_hour? The answer lies in two adjustable parameters: batch size (how many requests the model processes simultaneously) and number of GPUs (how many chips run the model in parallel). These two variables exist in what the author calls a "situationship"—larger batches and more GPUs increase aggregate compute and memory bandwidth, allowing bigger batches and more KV cache; but more GPUs also introduce network overhead, and larger batches require more data to traverse that network, both raising latency. Token latency (the time for one forward pass) determines both user experience (lower latency means faster responses) and throughput (how many total tokens the system produces per unit time). Providers operate on a Pareto frontier where no single operating point can improve both cost and speed simultaneously; they must choose a point on that frontier that matches their target market. Kimi K3 is a ~2.8 trillion parameter model that, when quantized to MXFP4, occupies about 1.4TB—too large for any single GPU. Eight B300 GPUs, each with 288GB of memory, provide 2.3TB total, enough for weights plus KV cache headroom. At the Pareto-optimal operating point, such a setup achieves roughly 30 tok/s for a single user while delivering about 1,500 tok/s of aggregate throughput per GPU (the difference illustrates batching: no single user sees 1,500 tok/s, but the GPU collectively serves that many tokens per second across all users). With B300s renting for about $7.39 per hour, the marginal cost per million tokens comes to approximately $1.37. Yet OpenRouter advertises Kimi K3 at $15 per million input tokens (or $3 for input and $15 for output on their platform), plus a $22 per million option from Fireworks for faster delivery. The gap between $1.37 and $15 appears enormous—but two major factors narrow it. First, utilization is never 100%. Demand is bursty: some hours see heavy traffic, others are quiet, and weekends and holidays see less activity. Providers must provision infrastructure for peak demand, not average demand. Assume 50% average utilization, and the cost immediately doubles. Second, the $1.37 figure is marginal cost—the cost of producing one additional token. It excludes fixed costs: salaries, power, networking, data center overhead, and most importantly, the training and fine-tuning of the model itself. A model is a depreciating asset; someone paid the cost to train and refine Kimi K3, and that expense must be recovered across all inference revenue. The article notes that inference optimization continues to advance: speculative decoding offers a ~3× speedup according to the vLLM blog; KV cache offloading, prefill/decode disaggregation, and other techniques are actively being researched. Any inference provider may possess proprietary optimizations that improve token throughput without proportionally raising cost, shifting their operating point further along the cost/speed frontier and materially improving profitability. The difference between Fireworks' standard ($15/M) and fast ($22/M) offerings underscores this tradeoff: the fast tier likely uses smaller batches or more GPUs per batch, sacrificing throughput for speed—showing that providers can adjust their position on the frontier to serve different customer needs at different price points.

The economics of LLM inference hinge on two key variables: batch size and the number of GPUs used. Together, these determine token latency—both the speed each individual user experiences (1/token_latency in tokens per second) and the aggregate throughput per GPU. Providers face a fundamental tradeoff: larger batches and more GPUs increase total throughput but also increase latency and network overhead. The vLLM cost/speed frontier illustrates this Pareto boundary, where every point represents the optimal price for a given speed. Kimi K3's case exemplifies the gap between theoretical marginal cost and retail pricing. At the Pareto-optimal operating point (roughly 30 tok/s per user, 1,500 tok/s aggregate throughput per GPU), the hardware-only cost comes to $1.37/M tokens. Yet OpenRouter's advertised $15/M price is more than 10× higher. The mathematics do not imply mispricing or excessive margins; rather, they highlight the structural costs hidden in any inference operation. Utilization is the first brake: because demand fluctuates (coding models see less activity nights, weekends, and holidays), providers must provision for peaks. Even a large provider with demand smoothed across many users likely operates at 50% utilization, immediately doubling the per-token cost from hardware alone. Beyond that lies the depreciation of the model itself—training and fine-tuning Kimi K3 is a sunk cost that someone must amortize across inference revenue. Salaries, power, and other operational overhead add further layers. The analysis also notes that inference optimization is an active frontier: speculative decoding, KV cache offloading, and prefill/decode disaggregation could each improve token throughput without proportionally increasing cost, potentially moving providers further along the frontier toward better economics—or allowing them to absorb lower prices while maintaining margins.

No comments yet. Be the first to share your thoughts!