DeepSeek V4 Flash hits 32 tok/s on AMD Ryzen AI MAX+ 395

In July 2026, the Lucebox team demonstrated that a full 284-billion-parameter DeepSeek V4 Flash mixture-of-experts model can run locally on a single AMD Ryzen AI MAX+ 395 APU (a processor with integrated Radeon 8060S GPU), using 128 GB of unified LPDDR5X memory. The target model was quantized to 102.3 GB, paired with an 11.3 GB DSpark draft model, leaving 14.4 GB headroom before OS and runtime allocations. No discrete GPU, remote inference service, or second machine was involved; prompts and model execution stayed entirely on the local system.

The core innovation is the ROCmFPX quantization family, a set of block formats built around AMD's ROCm/HIP compute path. Each block holds 32 weights as packed low-bit codes plus one or two small scales. ROCmFP2 stores a block in 10 bytes (2.50 bits per weight), ROCmFP3 uses 3.50 bits per weight, and ROCmFP4 uses 4.25 bits. For DeepSeek V4 Flash, the team added the missing 2-bit format and its HIP kernels, then constructed a mixed-precision recipe: the enormous routed-expert gate and up matrices use ROCmFP2, expert down projections use ROCmFP3, and dense or more sensitive projections retain ROCmFP4 or higher. The final quantized model uses roughly 2.88 bits per parameter on average. Both target and draft fit in the unified-memory pool because the CPU and GPU share the same 128 GB LPDDR5X address space.

Decoding achieved up to 32.0 tok/s with speculative verification. At batch one, every generated token streams active weights across all 43 layers, making decode memory-traffic-bound. The ROCmFPX kernels read packed blocks directly and expand codebooks using AMD byte-permute instructions in registers, feeding integer dot products without a separate gather step. The team added a DeepSeek-specific HIP decode path for hyper-connections, attention, routing, and expert work. With no draft, the target ran at 25.31 tok/s autoregressive. DSpark then added the next layer: a small three-layer draft proposes up to three new tokens, and the 284B target verifies four positions (including the current seed) in one fused pass. With a q=4 batch and adaptive width disabled, the result reached 32.0 tok/s, a 26.4% gain over autoregressive. Work with Geometric optimized the routing path with dedicated ROCm TOP_K and ARGSORT kernels, improving end-to-end decode by an additional 0.44%, though that measurement was separate from the published main-branch run. A refactoring of q=4 weight reuse (decoding each packed dense weight once and applying it to all four verification columns instead of unpacking it four times) added 2.1–2.3%.

Prefill throughput reached roughly 250 tok/s using indexed sparse prefill. The public LocalMaxxing request reported 245 tok/s with 2,048 prompt tokens; separate validation on 7,960-token input achieved 251.79 tok/s, with the 8,000-token test cases ranging from 246.8 to 255.9 tok/s. Sparse prefill uses DeepSeek V4's learned indexer to limit compressed-history attention and batches work layer by layer, which changes floating-point reduction order. Output is not byte-identical to tokenwise exact prefill (which measured 22.5–23 tok/s on short prompts), so sparse mode remains opt-in. Quality checks showed 10/10 on a small GSM8K set and 3/3 on HumanEval smoke tests; no broad evaluation has been run yet. The decode speed uses --ds4-expert-top-k 4 instead of the model default of six experts, trading quality margin for speed.

When submitted to LocalMaxxing on July 18, 2026, the 32.0 tok/s result placed it 68.5% ahead of HipFire (18.99 tok/s on Radeon 8060S) and 2.05× DwarfStar (15.6 tok/s on Ryzen AI Max 395). The article notes these are community submissions with different engines, quantizations, context sizes, and output lengths, not controlled A/B tests. The full implementation, including ROCmFPX kernels, fused decode path, DSpark verifier, sparse prefill, and q=4 weight reuse, is available on Lucebox main with no private patches. The team provided complete reproduction instructions, cmake build flags, and URLs to download the quantized models from Hugging Face. Setup requires ROCm 7.2.4, the platform performance profile, and Radeon high clock setting (2.9 GHz observed); automatic clocks can cost several tokens per second.

DeepSeek V4 Flash has become a benchmark for efficient local inference, and this Lucebox result demonstrates that running a large 284B mixture-of-experts model on consumer APU hardware is now practical. The key enabler is the ROCmFPX family of block quantization formats, introduced here with the missing 2-bit variant (ROCmFP2) and corresponding HIP kernels. The mixed-precision recipe—using ROCmFP2 for the routed-expert gate and up matrices, ROCmFP3 for expert down projections, and ROCmFP4 or higher for dense and sensitive layers—compresses the model to roughly 2.88 bits per parameter (102.3 GB) while preserving quality enough to pass regression checks on GSM8K and HumanEval smoke tests.

The speed gain comes from two complementary paths. Decode throughput (32.0 tok/s) is lifted by 26.4% above autoregressive baseline (25.31 tok/s) through speculative verification: a compact three-layer DSpark draft proposes up to three tokens, and the 284B target verifies four positions in one fused HIP pass with a q=4 cap. Prefill throughput (roughly 250 tok/s with indexed sparse mode) trades exact floating-point matching for speed by leveraging DeepSeek V4's learned indexer to reduce attention computation. The article also credits Geometric's optimized TOP_K and ARGSORT kernels for a 0.44% end-to-end decode improvement and 2.1–2.3% gains from q=4 weight-reuse refactoring on the main branch.

Compared to prior public entries on the same hardware—HipFire's 18.99 tok/s and DwarfStar's 15.6 tok/s—this represents a substantial step forward, though the article notes these are community submissions with different engines, quantizations, context sizes, and output lengths, not controlled A/B tests. Availability of the code and quantized models on Lucebox and Hugging Face lowers the barrier for reproduction and exploration.

No comments yet. Be the first to share your thoughts!