AIToday

Two environment variables prevent memory leaks in Ollama, vLLM, and TGI by fixing glibc heap fragmentation issues

r/LocalLLaMAMar 24, 20261 min read
Two environment variables prevent memory leaks in Ollama, vLLM, and TGI by fixing glibc heap fragmentation issues

Summaries like this, in your inbox every morning.

Sign up free →

3 Key Points

  1. Model servers like Ollama, vLLM, and TGI experience memory creep caused by glibc's heap allocator fragmenting and not returning pages to the OS

  2. Setting MALLOC_MMAP_THRESHOLD_=65536 and MALLOC_TRIM_THRESHOLD_=65536 before process startup resolves the issue without code changes

  3. Testing on 13 diffusion models showed dramatic improvement: before the fix, OOM occurred at 52GB after 17 hours; after the fix, memory remained stable at ~1.2GB indefinitely

  4. Full benchmark data and testing script available on GitHub at brjen/pytorch-memory-fix repository

Discussion

No comments yet. Be the first to share your thoughts!

Log in to join the discussion

Related Articles

Stay ahead with AI news

Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.

Get Started Free

Free · takes 30 seconds · unsubscribe anytime

1 minute a day. The AI essentials.

200+ sources · Email / LINE / Slack

Get it free →