GitHub releases a dataset of multilingual repository metadata to help researchers and developers build AI tools that work better across non-English languages.

What happened

GitHub published the GitHub Multilingual Repositories Dataset under CC0-1.0, a repository-level metadata collection covering over 80 million classification rows across more than 40 million repositories. The dataset includes language classifications of README files, the most-commented issue, and the most-commented pull request, along with repository metadata such as creation timestamp, disk usage, stars, forks, primary programming language, and license information. The release follows a commitment GitHub made in 2025 as part of Microsoft's European Digital Commitments to make multilingual data more accessible to open source AI developers.

Why it matters

Many European and other languages remain underrepresented in the text used to train and evaluate AI systems, creating a risk that developer tools work well for some communities while leaving others behind. Developer content like READMEs, issues, and pull requests contains the language of software collaboration—installation instructions, bug reports, feature requests, and review comments—which differs from general web text. By making multilingual developer-content signals easier to find and analyze, the dataset can help researchers and model builders identify gaps, support better evaluation, and build more inclusive AI tools for developers across different language communities.

What to watch

The dataset deliberately exposes classifications from three different language-identification tools (fastText, gcld3, and lingua-py), each with confidence scores, so users can choose their own precision and recall tradeoffs rather than relying on a single label. GitHub and its partners will discuss the dataset and the importance of multilingual data for AI at the Open Innovation Dialogue Hub in Strasbourg on June 16.