Santosh Koshy Joy | AI Specialist |
The global Artificial Intelligence industry has long struggled with what experts call the “English-language bias.” Large Language Models (LLMs), the powerful engines behind today’s AI revolution—are trained predominantly on English data, making them masters of the “English road.” But when these systems are deployed in India, they don’t just change destinations—they enter an entirely different scenario: a breathtaking, complex mosaic of languages, dialects, and cultures that English-trained AI was never fully built to navigate.
This is where IndQA or IndicQA by OpenAI, which has Microsoft as the most significant external investor and strategic partner, emerged as the first initiative as a specialized essential tool. Conceptualized as a “driving test” for AI models in the Indian context, its primary purpose is to audit how culturally aware and linguistically capable an LLM is beyond simple English translation. IndQA focuses on Context-Grounded Question Answering (QA) across several major Indian languages, testing the model’s knowledge in specific Indian cultural domains, including Law and Ethics, History, Literature and Everyday Life. For AI researchers and developers, it is a vital diagnostic tool to measure multilingual QA performance and diagnose language biases, especially revealing the low performance in resource-scarce languages like Assamese or Odia compared to high-resource languages like Hindi. For AI deployers, it acts as a critical selection and risk assessment tool, establishing minimum performance thresholds before an AI is trusted to serve Indian users.
The benchmark’s high-stakes cultural questions aim to ensure the AI is not just translating, but understanding the context — a fundamental necessity for reliable deployment. However, despite its crucial role, IndQA is an infant testing ground that only scratches the surface of India’s linguistic richness. Its current design and the technical realities of LLMs mean it is a far cry from taming the full spectrum of Indian vernacular languages.
IndQA is limited by its very scope, which attempts to simplify an almost chaotic linguistic reality. The benchmark focuses on 11 major scheduled Indian languages. While significant, this excludes the reality that much of India’s vibrant communication happens in forms that AI models cannot access or process effectively. It conveniently forgets the fact that within a single official language like Marathi or Hindi, countless local dialects exist (e.g., Bhojpuri, Awadhi, Vidarbha Marathi) that possess distinct vocabularies, pronunciations, and grammars. AI models, primarily trained on standardized digitized content are linguistically deaf to these vernacular nuances, leading to failure in real-world voice applications and informal text.
Hundreds of regional and tribal languages in India maintain a strong oral tradition, with minimal or zero digital presence. Their knowledge exists in folk songs, local rituals, and storytelling—data that does not exist online to train an LLM. This creates a massive data scarcity problem, rendering large populations completely excluded from the benefits of modern AI.
Real-life communication in India is dominated by code-mixing where speakers seamlessly blend two languages in a single sentence. Importantly, IndQA has captured Hinglish, the combination of English and Hindi, as one of the 12 languages but many such blended language forms and dialects are still unaddressed. Code-mixing phenomenon forces the AI to instantly process and switch between the linguistic rules (grammar and syntax) of two completely different language families (e.g., Indo-Aryan and English). Benchmarks often struggle to fully capture the vast, unpredictable scope of this continuous switching, simplifying a major component of real-world Indian communication. The insufficiency of IndQA is most evident in the underlying technical limitations of how LLMs process Indian languages, specifically concerning tokenization and the resulting lexical gap.
Another challenge is tokenization, which is the critical first step where an LLM breaks a sentence into manageable units called tokens. This process, which is optimized for low-morphology languages like English, fails dramatically for Indian languages. Languages like Tamil, Malayalam, Telugu, and Kannada are highly agglutinative. They build massive, complex words by “gluing” multiple affixes called morphemes onto a single root word, where one word can carry the meaning of an entire English phrase. Standard tokenizers cannot recognize the internal, grammatical structure of these “super-words.” Instead, they often treat the entire word as one rare token, wasting the model’s vocabulary, or they break it down into tiny, meaningless character fragments. This forces the model to process 2x to 4x the number of tokens compared to English for the same amount of information, driving up computational cost and latency.
In languages like Hindi and Bengali which are fusional languages, affixes fuse or blend with the root, conveying multiple meanings simultaneously. The verb ending in Hindi fuses information about the present tense and the gender or the number of the subject. The tokenizer’s failure to accurately decode this fused information leads to poor accuracy and the inability to generate grammatically correct, contextually appropriate sentences. The inefficiency in tokenization directly leads to a massive Lexical Gap, the chasm between the vocabulary an AI knows and the real vocabulary used by Indian speakers is real and insurmountable with baby steps like the IndQA. Because the model’s core vocabulary is English-heavy, any term specific to a local dialect or vernacular like a local farming term or a regional cooking ingredient is treated as an out-of-vocabulary (OOV) word. The model is then forced to guess its meaning, degrading the reliability of its answer, even in the specialized cultural context of IndQA.
The most crucial evidence comes from the original IndQA research itself. It found that for many low-resource languages, the best performance was achieved using the Translate-Test Paradigm—translating the Indic question to English, having the powerful English LLM answer it, and then translating the answer back. This confirms that the model’s true linguistic competence in those languages is weak. It is merely a proficient translator running on a strong English core, unable to process the low-resource Indic language directly and efficiently at the token level. A high score in this scenario is a measure of the model’s translatability not its native linguistic fluency.
IndQA serves an essential function by holding AI models accountable for cultural and linguistic awareness in a defined, major-language subset. However, for AI to truly serve the billion-plus, linguistically diverse population of India, the focus must move beyond the limitations of the current benchmark. Developing AI-specific tokenizers that can efficiently decode the agglutinative and fusional complexity of Indic languages moving past English-centric architectures is the need of the hour. Aggressive, collaborative efforts with local communities to digitize and annotate oral content, providing the training data necessary for currently “invisible” languages is a necessary step forward. New benchmarks must be developed that rigorously test an AI’s ability to handle complex Hinglish conversation and identify toxicity or hate speech in vernacular social media, areas where IndQA provides no safety net.
Adding to the challenge is the fact there is a lack of a mature, indigenous and sovereign AI stack, which is India’s own foundational large language models, computing infrastructure and comprehensive regulatory framework which poses significant challenges across India’s economy, society, and national security. While India is a global hub for AI talent and IT services, its reliance on models primarily developed in the US and China creates multiple vulnerabilities. While IndQA opened the door to recognizing the Indian linguistic challenge, it stands as a clear indicator that the journey to creating truly proficient, vernacular AI in India has just begun. The pursuit of an AI that speaks, understands, and thinks in the true voice of India remains a monumental, unfinished task.
[Santosh Koshy Joy is a doctoral scholar on AI and its implication on learning pedagogies at University of Delhi. The views expressed are those of the author solely. ]
Get real time update about this post category directly on your device, subscribe now.
