OpenAI's custom chip beats Nvidia in inference benchmarks

OpenAI's Jalapeño chip, developed with Broadcom, took only nine months from first design to manufacturing blueprint, a speed that SemiAnalysis says signals the end of Nvidia's CUDA moat. The chip delivers up to 104x token throughput per kilowatt at matched decoding speed, and it does so without relying on optimizations like multi-token prediction, leaving room for further gains. However, caveats remain: competitors like Nvidia and AMD have tested larger models on their chips, and Jalapeño is still an engineering sample, not yet in production. OpenAI's CFO Sarah Friar frames the chip as part of an integrated compute strategy, complementing existing partnerships with Nvidia, AMD, and others, rather than replacing them. This dual dynamic of cooperation and competition underscores the broader industry trend where major players are developing custom silicon, even as they continue to buy from each other.