Developer struggles to optimize RAG-based Android app performance on low-end devices, weighing offline LLM models against cloud API solutions.

Developer currently uses Qwen 1.5 2.5B model with Q4 quantization and GPU offloading for offline RAG processing on Android

Low-end devices face memory constraints causing app crashes or extremely slow text generation without GPU acceleration

SmolLM 135M model attempted as alternative but shows poor instruction-following capabilities compared to larger models

Developer considering switching to OpenAI API for low-end devices while maintaining local processing for better-equipped phones

Using in-house vector storage solution (snkv) and seeking best practices from community on running LLMs on resource-constrained Android devices