Bonsai 27B: First 27B AI Model Runs on iPhone

PrismML, a team of Caltech researchers backed by Khosla Ventures, Cerberus, Google, and Samsung, announced Bonsai 27B, a compressed large language model based on Qwen3.6 27B. The model comes in two variants, each optimized for a different deployment target. The ternary version uses ternary weights {−1, 0, +1} with FP16 group-wise scaling, delivering 1.71 effective bits per weight and occupying 5.9 GB; it is designed for laptops and retains full reasoning, tool-calling, and agentic capability. The 1-bit version uses binary weights {−1, +1}, also with group-wise scaling, delivering 1.125 effective bits per weight and occupying 3.9 GB; it fits within the memory budget of an iPhone 17 Pro, marking the first time a 27B-class model has been deployed on a phone.

Both variants are multimodal, with a compact 4-bit vision tower enabling on-device understanding of screenshots, documents, and camera input. The models carry a full 262K-token context window and support speculative decoding, an acceleration technique that pairs draft and verification stages to compound inference speed. Critically, the compression—1-bit and ternary quantization—runs end to end across the entire architecture (language network, embeddings, attention, MLPs, and language modeling head) with no higher-precision escape hatches, meaning every computation uses the ultra-low-bit representation.

On benchmarks, the ternary variant retains 95% of full-precision baseline performance across a 15-benchmark suite that spans knowledge, reasoning, math, coding, instruction following, tool calling, and vision (all evaluated in thinking mode, where the model's full reasoning is exercised). The 1-bit variant retains 90% of baseline. The body notes that math and coding are nearly untouched by quantization, and tool calling stays within a few points of full precision—exactly the capabilities agentic workflows depend on. For comparison, the most aggressive conventional low-bit build of the same base model scores significantly lower than 1-bit Bonsai 27B while occupying 2.5× more memory. By the metric of intelligence density (capability per GB), 1-bit Bonsai 27B delivers 0.53 per GB, more than 10× the full-precision baseline and roughly 2.7× the best low-bit alternative available.

Performance on standard GPUs is substantial: the 1-bit variant reaches up to 163 tok/s on an NVIDIA GeForce RTX 5090 and 87 tok/s on an M5 Max; the ternary variant reaches 134 tok/s and 58 tok/s on those same devices respectively. The body emphasizes that fitting a phone is a stricter constraint than raw storage, because a phone never exposes its full memory to an app—a 12 GB iPhone offers about 6 GB for the model to use on-device, shared with KV cache and activations—and no conventional 27B build comes close to clearing that threshold.

The announcement frames this as a paradigm shift: valuable AI workloads are shifting from single responses to sustained, multi-step work—agents that operate real tools, workflows that run unattended, research that synthesizes many documents. Cloud APIs will remain appropriate for many products, but agentic execution on cloud-only infrastructure imposes structural constraints: every step is a remote request, per-token costs accumulate across iterations, and every plan, tool call, intermediate result, and user data (files, screens) crosses the network. Local execution eliminates those constraints, enabling on-device agents that can reason over private data by construction, work offline, and operate at zero marginal cost. The body also describes a hybrid architecture: routing non-frontier and privacy-sensitive tasks to the local model and reserving frontier cloud models for the hardest steps, collapsing the cost-per-task of agentic systems overall.

Bonsai 27B is available today under the Apache 2.0 License and runs natively on Apple devices (Mac, iPhone, iPad) via MLX and on NVIDIA GPUs via CUDA. PrismML is offering a limited-time free developer preview API so developers can try the model. The full technical details of compression, evaluation, and benchmarking are available in a whitepaper.

The shift from cloud-only AI to on-device execution has been blocked by a simple bottleneck: models capable of sustained multi-step reasoning and tool use—the kinds of tasks modern agents require—have been too large to fit on consumer hardware. A 27B-parameter model in 16-bit precision occupies roughly 54 GB; even aggressive 4-bit quantization brings it only to 18 GB, still too large for a phone or typical laptop. Bonsai 27B breaks through that barrier by using ultra-low-bit weights (1-bit and ternary) across the entire model stack—embeddings, attention, language modeling layers, and output heads—with no higher-precision fallback. The result is a 3.9 GB (1-bit) or 5.9 GB (ternary) model that the body demonstrates retains 90–95% of full-precision performance on a broad evaluation suite.

This matters because agentic workloads have fundamentally different economics than single-turn inference. Where a chatbot makes one model call, an agent makes hundreds—each one carrying context, producing structured output, and feeding the next iteration. In cloud-only architectures, each step is a network round-trip, accumulating per-token costs and forcing user data (files, screens, reasoning steps) to cross the network repeatedly. Local execution inverts the equation: once the model is on the device, the marginal cost of a hundred-step loop is zero, data never leaves the machine, and the model can operate offline. The body explicitly frames this as enabling "persistent on-device agents, assistants that work offline, assistants that reason over private local data by construction" and opening a new hybrid architecture where privacy-sensitive or cheap tasks run locally while only frontier tasks go to the cloud. For businesses deploying AI assistants, this represents a shift from per-token cloud dependency to locally amortized cost.

No comments yet. Be the first to share your thoughts!