Apple researchers explore how small language models can improve accuracy by strategically accessing external knowledge sources beyond just increasing parameters.

Small Language Models (SLMs) are inherently limited by their parameter size, which restricts the amount of world knowledge they can encode during pretraining

SLMs often generate factually incorrect outputs due to their limited capacity compared to larger models

The research proposes mitigating accuracy issues by giving SLMs access to external sources like larger models, documents, or databases

The study examines fundamental questions about what knowledge SLMs should learn versus what they should retrieve from external sources

Research was accepted at the Workshop on Memory for LLM-Based Agentic Systems at ICLR