Liquid AI's open d1-3B trails d1 API by 10.8 points on JNLI

Liquid AI published d1 as an API-only decision model, but the October 5 weight release of d1-3B on Hugging Face made it runnable on a single GPU such as an RTX 4080. To test whether the open version matches the paid one, the author ran both models on the same Japanese benchmarks: JNLI, JCommonsenseQA, and Livedoor news classification, feeding one item per request in the same order. The d1 API numbers came from an October 1 measurement, while d1-3B was measured on October 8, so the two runs are a week apart. d1-3B also required answer choices as a dictionary rather than a list, which was swapped in without explanation.

The paired comparison showed the API ahead on JNLI and JCommonsenseQA by margins that exceed chance, while the Livedoor result showed no meaningful difference. This pattern is consistent with the API running a stronger model than d1-3B, since the published weights only cover d1-3B and the smaller d1-omni-600M, and Liquid AI has not disclosed the size of the API model. The article's own limits qual_ify that comparison: it covers only three Japanese benchmarks, the API results were reused from an earlier date, and the open model received no explanation alongside its answer choices.