New evaluation framework reveals that only 35% of LLM responses on sexual and reproductive health in Nepali meet quality standards, highlighting gaps in low-resource language support for sensitive health topics.

Researchers introduced LEAF (LLM Evaluation Framework) to assess AI responses on sexual and reproductive health (SRH) queries in Nepali, a low-resource language

The framework evaluates responses across four dimensions: accuracy, language quality, usability gaps (relevance, adequacy, cultural appropriateness), and safety gaps (safety, sensitivity, confidentiality)

Study analyzed 14,000 SRH queries from over 9,000 Nepali users with manual annotations by SRH experts, finding only 35.1% of LLM responses met quality standards

Current LLM evaluation methods focus primarily on accuracy for objective queries in high-resource languages, overlooking usability and safety needs for culturally sensitive health topics