Why models can’t answer if the question is about Africa in an African language

A team of researchers at MBZUAI has created the first benchmark to test whether AI models can understand African cultures in the languages Africans actually speak. The early verdict: most of today’s models can’t.

Afri-MCQA covers 16 languages across 13 countries, with roughly 8,000 question-and-answer pairs built around culturally specific images, including examples such as a Setswana beaded garment, a Ugandan snack brand or monuments in Kenya. Every question comes in both the local language and English, and in two forms: written text and spoken audio recorded by native speakers. That makes it far broader than the handful of African-language benchmarks that exist, most of which cover a few languages and only cover text modality.

People who live where the languages are spoken built the dataset by hand. The team recruited native speakers in-country, ran a screening pilot, then had native-speaker coordinators check the roughly 500 items per language one at a time. Across all 16, those languages cover approximately 392 million speakers.

One question shows a photo of a traditional dish and asks, in Oromo and in English, what it’s called; another shows a rack of snacks and asks which Ugandan company makes them. Each comes with three plausible wrong answers, so a model can’t coast on elimination; it has to recognize what it’s looking at. The questions also lean on local knowledge a search engine wouldn’t surface.

Africa is home to more than a third of the world’s languages and a population heading past 2.5 billion by 2050, and many of those languages are mainly spoken. Literacy, where it exists, often lands in a language such as English or French instead. A chatbot that only reads is missing how most people would actually use it. That is why the benchmark insists on audio. Earlier cultural tests queried models through text alone, and one paper the authors cite labels the result AI with “WEIRD” (Western, Educated, Industrialized, Rich and Developed) coverage, tilted toward Western, English-speaking data.

Lead author Atnafu Tonja said: “Many African languages are primarily spoken, curating speech question answering for African languages to understand the gap of current models in audio modality”.    ”

The authors tested seven models from small open-weight models such as Qwen 2.5-Omni and Google’s Gemma up to Gemini 2.5 Pro, a proprietary model from Google.

If you give the model four options to choose from, it looks competent. But take the options away and ask it to produce the answer itself, and it falls apart. Gemini, the strongest, scored 78% on the multiple-choice benchmark in English but only 38% when it had to generate the answer. The smaller open models fell from the 50s to as low as single digits. The authors conclude that multiple choice questions flatter a model, because it can pattern-match its way to a lucky guess; open-ended answering shows whether the knowledge is really there.

English questions beat native-language ones for every model: by 2 percentage points for Gemini, and by as much as 19 for the Qwen models. Making the open models bigger barely helped: larger versions scored about the same as their smaller siblings on native-language questions, which points to the training data rather than the size of the model.

Speech breaks the models entirely. When tested on native audio questions , the open models scored close to zero on open-ended answer generation.

The authors also explored two common failures: not knowing the culture, and not understanding the language. They ran control tests of plain language ability and took the audio pipeline apart step by step.

For audio experiments, the authors explored two controlled experiments to understand why models fail on audio question answering: language identification (the authors evaluated the model to identify the spoken language) and generating a transcription for a given audio. For language identification, open-weight models fail to recognize African languages. Gemini managed 96% accuracy, while the Qwen models scored 2% and 4% accuracy in identifying the spoken language, barely above chance. On transcription, the open-weight models showed word error rates of 85% to over 100%, meaning the output was essentially noise. A model that can’t tell what language it’s hearing, let alone write down the words, has no chance of answering the question behind them. 

According to MBZUAI professor Thamar Solorio, “our language identification results underscore that speakers of many African languages cannot yet rely on LLMs to deliver consistently acceptable performance. They also highlight the need for greater public and government investment in training open models for these languages using culturally relevant data. We hope thatAfriCVQA will inspire the research community, policy makers and all relevant stakeholders to engage more deeply with building and adopting data from these regions and for these communities.”

Gemini was the consistent exception. It led in every task, maintained the smallest gap between English and native-language questions, and was the only model to achieve near-perfect language identification. However, the authors couldn’t help noticing the divide between proprietary model and the open-weight models used by most researchers today which failed hardest on the languages that need them most.

On text question answering, the drop from English to native languages was larger on the language benchmarks than on the cultural questions themselves which the authors mentioned as a sign that weak language understanding is the bigger bottleneck for low-resource languages. But the models also stumbled over African cultural questions asked in English, so the gap is cultural as much as it is linguistic.

Sixteen languages are a lot for this kind of work, and still a sliver of the thousands spoken across the continent. Culture is slippery and subjective, and any fixed set of questions can never fully encompass it; the annotators’ own judgments about what counts as “culturally relevant” shape the data. And because collecting it manually is slow, Afri-MCQA is meant as a benchmark for evaluating models rather than for training language models.

The authors of the paper were rewarded at ACL 2026 with the Social Impact Paper Award and Senior Area Chair Highlight, a recognition of their work to make multimodal AI more inclusive of African languages, cultures, and communities.

In the end, one of the main takeaways from this paper is that a model can carry a fact about a culture and still fail to retrieve it the moment someone asks in their own language which is, for a great many people, the only way they would ever think to ask.