Archive · 5 Nov 2025
OpenAI Introduces IndQA Benchmark for Indian Language AI Models
Imagine you have a really smart computer program that understands language, called an AI model.
India has many languages and unique ways of life that are different from other countries.
For a long time, the tests used to see how smart these AI models are didn't really understand India.
So, a company called OpenAI, with help from many Indian experts, created a new test called IndQA.
This test has many questions in different Indian languages and even includes things like Indian food, history, and art.
It helps check if AI models can understand not just words, but also the culture behind them.
This is important because many people in India speak languages other than English, and they want AI that understands them.
This new test is not meant to say which AI is the best overall, but to help developers see if their Indian language AI models are getting smarter and understanding India better over time.
OpenAI has launched IndQA, a new benchmark designed to evaluate AI models on their understanding of Indian languages and cultural context.
The benchmark includes 2,278 questions across 12 languages (including English and Hinglish) and 10 cultural domains, developed with 261 Indian experts.
IndQA aims to assess AI models' grasp of nuances in areas like architecture, arts, food, history, law, and literature specific to India.
The benchmark is intended to measure a model's improvement over time, rather than serving as a leaderboard to rank different AI models.
The creation of IndQA addresses a critical need for local benchmarks to support the development and fair evaluation of Large Language Models (LLMs) for Indic languages.
- Who
- OpenAI, in partnership with 261 experts from India
- What
- Introduced IndQA, a new benchmark evaluation for AI models focusing on linguistic ability and Indian cultural context across 12 languages.
- When
- Announced on Monday, November 3, with the company hosting its DevDay Exchange developer conference in Bengaluru on November 4.
- Where
- India
- Why
- To address the lack of local benchmarks for Indic LLMs, capture cultural nuance, and support the development of AI models relevant to the Indian context, where India is ChatGPT's second-largest market.
OpenAI's Perspective
Concerns from Indian AI Startups
Benchmark Design and Purpose
OpenAI's Perspective
IndQA is designed to test linguistic ability and cultural context in Indian languages, aiming to measure improvement over time within a model family, not to rank models against each other. It was developed with Indian experts and includes culturally grounded prompts.
Concerns from Indian AI Startups
Existing benchmarks, often Western-led, are criticized for lacking local context and potentially showing bias, hindering the progress and fair ranking of Indic LLMs. The absence of standardized local benchmarks is a major barrier for Indian startups.
Potential Impact on Indic LLMs
OpenAI's Perspective
The benchmark aims to create similar evaluations for other languages and regions, starting with India due to its large non-English speaking population and market size for AI. OpenAI's models may be at a disadvantage as the questions were designed based on where they struggled.
Concerns from Indian AI Startups
The lack of high-quality datasets and local benchmarks has been a significant challenge for Indic LLMs. IndQA, by focusing on cultural nuance, could potentially help bridge the gap if it's accessible and fairly applied.
- Benchmark Name
- IndQA
- Number of Questions
- 2,278
- Number of Languages
- 12 (including English, Hinglish, and 10 Indian languages)
- Number of Cultural Domains
- 10
- Number of Experts Involved
- 261
- Key Indian Languages Covered
- Bengali, Hindi, Hinglish, Kannada, Marathi, Odia, Telugu, Gujarati, Malayalam, Punjabi, Tamil