Thinking Machines Lab: Qwen3-235B gives 80 unique completions at temp 0

The finding matters because it undercuts a common assumption: that setting temperature to 0 guarantees the same answer every time. In practice, the sampler is deterministic, but the numbers it samples from differ with the batch around the request. When two candidate tokens are nearly tied, such as "Queens" and "New" at 12.3417 and 12.3415, a tiny rounding shift can flip the greedy choice and send the whole completion down a different path.

The effect traces to how GPU kernels pick a parallelization strategy. A small batch does not give all 132 SMs on an H100 SXM enough work, so long sums are split across SMs and added back together in a different order than a large batch would use. The same three fp16 numbers can produce 1024 or 1025 depending on which pair is added first.

The practical response is a batch-invariant kernel, which reduces each row in the same fixed order no matter how many rows share the batch. This restores identical logits across batch sizes but costs throughput because the kernel can no longer pick the best split per batch size. With this on, Thinking Machines got all 1,000 Feynman completions identical.