Bachelor's project: YOLO26n inference built from ARM64 assembly, no frameworks

For his Bachelor's final project, a student undertook the ambitious task of implementing YOLO26n inference—a modern object detection neural network—entirely from scratch using ARM64 Assembly Language and C, without relying on existing inference frameworks like TensorFlow or PyTorch. The goal was twofold: to understand how modern neural network inference engines operate at a low level and to explore optimization techniques that could enable faster, more efficient edge AI execution on Raspberry Pi 4, a common embedded platform.

The implementation is comprehensive. At its core lies an ARM64 Assembly Language and C inference engine, coupled with ARM NEON SIMD (Single Instruction Multiple Data) optimization—a feature set that allows modern ARM processors to execute multiple operations in parallel. The project incorporates advanced convolution techniques, including Winograd convolution, which reduces the number of multiplications required by rearranging computation in the frequency domain. Custom GEMM (General Matrix Multiply) kernels provide hand-tuned linear algebra, while cache-aware tiling ensures data is organized in memory to minimize cache misses during computation. The student also implemented operator fusion—combining multiple neural network operations into single, more efficient steps—along with custom ARM64 micro-kernels and an attention mechanism. All YOLO26 architecture components were built manually: convolution layers, C3K2 blocks, SPPF (Spatial Pyramid Pooling Fusion), C2PSA, PSA (Permute, Scale, Attention), bottleneck layers, and the detection head.

A critical detail: the student extracted the YOLO26n model's trained parameters and redesigned the memory layout into a custom binary format optimized specifically for the inference pipeline, rather than relying on standard serialization formats. This level of control reflects both the hands-on nature of the project and the reality that inference performance depends on how data flows between CPU caches, registers, and main memory.

Despite the effort and technical depth, the implementation revealed a humbling lesson. The inference engine successfully produces correct object detection results—it works—but the actual performance improvement fell short of the author's initial expectations. This outcome underscores why production inference frameworks invest heavily in vendor-optimized code paths: achieving real-world speedups requires not only understanding the techniques but also balancing trade-offs between code maintainability, portability, and marginal gains. The author is actively seeking feedback, suggesting openness to refining the approach or identifying bottlenecks.

This Bachelor's final project tackles a rarely-seen challenge: rebuilding inference from first principles rather than using libraries like TensorFlow or PyTorch. The student's approach—implementing YOLO26n in ARM64 assembly and C—is intentionally low-level, designed to expose the internal mechanics of how neural networks execute on processors, especially on constrained edge devices like Raspberry Pi 4. The optimizations listed (NEON SIMD, Winograd convolution, cache-aware tiling, operator fusion, custom micro-kernels) represent standard techniques in production inference engines but are usually abstracted away. By reconstructing them manually, the author gains direct visibility into the trade-offs between code complexity, memory layout, and clock-cycle efficiency. The custom binary format for model parameters reflects a deliberate redesign to match the inference pipeline's memory access patterns, a detail typically handled automatically by frameworks.

The stated shortfall—"performance improvement was lower than initially expected"—is instructive: it signals that hand-optimization at the assembly level, while revealing in educational terms, does not automatically outpace highly-tuned, vendor-optimized inference engines. This mirrors real-world constraints: frameworks like TensorFlow Lite and ONNX Runtime employ teams of hardware engineers optimizing for specific chips (ARM, x86, GPU). The project's value lies not in beating those engines but in understanding the mechanics that make them work.

No comments yet. Be the first to share your thoughts!