A hobbyist running large language models on a four-GPU home server found that memory bandwidth, not compute, is the real bottleneck in AI inference. The older PCIe 3.0 motherboard with x8 lanes (instead of the faster x16 standard) makes it too slow to move data between GPUs for tensor parallelism, a technique that splits each layer across multiple GPUs to multiply bandwidth.
Instead, the author must use layer parallelism—putting different layers on different GPUs—which delivers only the memory bandwidth of a single GPU.
This reveals why AI infrastructure at scale requires specialized high-bandwidth inter-GPU connections like Nvidia's NVLink or AMD's Infinity Fabric, not just piling GPUs together.
What happened
A hobbyist built a home server using four AMD Radeon Pro V620 GPUs to run large language models, but discovered that the older PCIe 3.0 motherboard running at x8 lanes instead of x16 severely limits inter-GPU communication, making tensor parallelism impractical and forcing a layer-parallel architecture.
Why it matters
Token generation in language models is constrained by memory bandwidth (reading model weights from VRAM), not raw compute speed. This fundamental bottleneck means that even a four-GPU setup with ~512 GB/s per card cannot achieve the theoretical 2 TB/s bandwidth gain from tensor parallelism because the overhead of shuffling data between GPUs over slow PCIe outweighs the benefit. For anyone running AI models on consumer or older enterprise hardware, this illustrates why scaling horizontally on commodity GPUs is far harder than it appears.
What to watch
Layer parallelism—splitting different model layers across GPUs rather than splitting each layer across GPUs—becomes the only practical approach on this hardware, though the author notes it still incurs a performance cost versus a single GPU with enough VRAM. The author plans further optimization work on llama.cpp settings to maximize throughput within these constraints.
Ask the AI about this article →
The post is part of a series documenting the author's attempt to run state-of-the-art language models on salvaged consumer/enterprise hardware in a garage setup. The first chapter assembled the hardware; this chapter confronts the hard reality of scaling AI inference beyond a single GPU. The core insight is that modern language models are memory-bound workloads: every token requires reading the entire model from VRAM, and that I/O dominates the runtime. This is why the data center GPU market has shifted to prioritize high-bandwidth memory, as the author notes. The two parallelism strategies—layer parallel and tensor parallel—represent a trade-off. Layer parallel keeps communication simple (data moves once per GPU per token) but offers no bandwidth multiplier. Tensor parallel theoretically multiplies bandwidth by splitting each layer across four GPUs, but requires constant synchronization and data movement that, on older interconnects, destroys any gain. The author's setup, with PCIe 3.0 at x8 lanes, lands firmly in the "simple interconnect" camp, making layer parallel the only viable path forward. This illustrates why commercial AI providers invest heavily in custom silicon (NVLink, Infinity Fabric) and why off-the-shelf multi-GPU hobby servers remain impractical for demanding workloads.
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytime
Ask AI anything about this article. Q&As are published on this page for other readers too.
