Model servers like Ollama, vLLM, and TGI experience memory creep caused by glibc's heap allocator fragmenting and not returning pages to the OS
Setting MALLOC_MMAP_THRESHOLD_=65536 and MALLOC_TRIM_THRESHOLD_=65536 before process startup resolves the issue without code changes
Testing on 13 diffusion models showed dramatic improvement: before the fix, OOM occurred at 52GB after 17 hours; after the fix, memory remained stable at ~1.2GB indefinitely
Full benchmark data and testing script available on GitHub at brjen/pytorch-memory-fix repository
Ask the AI about this article →
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Chinese large-model developer Z.ai says it can now support large-scale inference using roughly 100,000 domesti…
Recent controversies include Ajinomoto's official X account posting an AI-edited image and a restaurant menu s…
Broadcom announced VMware AI Factory, a software-defined foundation for VMware Private AI Cloud, at VMware Exp…
AI coding has moved through three stages—prompt engineering, context engineering, and now harness engineering—…
Workday is integrating its AI agents with Google Cloud's Gemini Enterprise, letting employees start tasks like…
Over the last two years, writing code syntax has become much easier, with tools like Cursor, Claude Code, and…
