GitHub released a multilingual dataset of over 40 million repositories to help researchers and developers build AI tools that work better across languages, addressing a gap where many European languages remain underrepresented in AI training data.

What happened: GitHub published the GitHub Multilingual Repositories Dataset, a metadata collection covering over 80 million classification rows across more than 40 million public repositories. The dataset identifies languages used in README files, issues, and pull requests, and is now available under CC0-1.0 license. Portuguese tops the non-English README list with more than 3 million repositories, while Korean is the most common non-English language in issue text.

Why it matters: Many European languages remain underrepresented in the online text used to build and evaluate AI systems, which creates a risk that AI tools work well for some developers and communities while leaving others behind. Developer content like READMEs, issues, and pull requests contains the language of software collaboration—installation instructions, bug reports, feature requests—which is different from general web text and can help build AI systems that better understand how developers actually work. The dataset gives researchers and model builders a tool to study language representation in software development and identify gaps.

What to watch: The dataset deliberately exposes classifications from three different language-identification tools (fastText, gcld3, and lingua-py) with confidence scores, rather than collapsing them into a single label, so users can choose precision and recall tradeoffs for their own research. GitHub will discuss the dataset and multilingual AI at the Open Innovation Dialogue Hub in Strasbourg on June 16.