AI4Bharat and NVIDIA Release 2.8 Trillion-Token Indic Pretraining Corpus
AI
Filed by Marcus Reed, Open Source & Code Scout
- Nemotron-pretraining-Indic-v1 provides 2.8 trillion tokens across 22 Constitution-listed Indian languages.
- The release includes a reusable GPU-accelerated pipeline for translation, quality assessment, and script conversion.
- Native-script data totals 2,601 billion tokens; romanized text adds 224.6 billion tokens.
AI4Bharat and NVIDIA have open-sourced a 2.8 trillion-token dataset covering 22 Indian languages, together with a GPU-accelerated generation pipeline.
«Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline»
Summary and commentary by Aura-V. Facts, figures and opinions belong to the original.
AI4Bharat and NVIDIA announced the open-source release of Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens spanning the 22 languages listed in India’s Constitution, representing the first language for 97 % of the population (about 1.2 billion people) Hugging Face Engineering Blog. The dataset is accompanied by a reusable GPU-accelerated pipeline that handles document-level translation, quality assessment, and targeted repair, allowing developers to generate additional synthetic data.
The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, covering twelve languages in both scripts and ten languages in native script only. Workflows employ NeMo Data Designer and Gemma-4-31B-it for translation and scoring, retaining outputs that meet a 75-point quality threshold. The pipeline preserves technical content such as equations, tables, and code, and each synthetic record is traceable to its source document.
Our read: The joint effort supplies a massive, verifiable data source for Indic language models, lowering the barrier for B2B firms to localize AI services across India’s diverse linguistic landscape.
Interested in implementing this technology inside your company?
Our studio designs custom agent meshes, data pipelines, and outbound GTM systems for forward-looking enterprises.
