AI can spot the right answer at an Emirati wedding. Saying it in Emirati is harder.

At a wedding in the UAE, a man carries a mabkhara (a little incense burner that fills a room with oud smoke) through the guests. If a friend visiting from outside the country asks what he was doing, the man can explain that he was perfuming people to honor them, an ordinary gesture of welcome. However, if that friend were to ask an AI model the same question, he’d probably get the wrong answers: the man was disinfecting the hall or staging a folk performance. 

The scene above is one of 6,942 in ArabCulture-Dialogue, a benchmark built by a team at MBZUAI to test whether AI understands Arab culture the way that people actually live it. Their paper, presented at ACL 2026, shows that every model tested does worse once the conversation shifts from standard Arabic into a local dialect. The strong models can still read the culture, but getting them to answer back in the right dialect is where it comes apart.

Almost every Arabic culture benchmark before this one ran in Modern Standard Arabic, the language used in newspapers, textbooks, and TV, and asked single, isolated multiple-choice questions. However, most people don’t actually use MSA in everyday life. Around 400 million people speak Arabic, and they speak it in dialects that drift far from the standard and from each other, carrying the idioms and manners that culture rides on. Grade a model only in MSA, the authors argue, and you’re testing on the easy register.

The team started from ArabCulture, an older set of single-turn MSA questions, and turned each one into a short back-and-forth. OpenAI’s models drafted first passes; everything after that was done by 26 native speakers, two per country, who rewrote the dialogues, translated them into their home dialect (told to write how people talk, not word-for-word), then cross-checked each other’s work. The annotators were forbidden from using any AI of their own.

In early drafts, the correct answer kept giving itself away (a little longer than the others, or opening with a telltale phrase), so a model could pick it without understanding a thing. The annotators rewrote all three options in every question, in both MSA and dialect, until they matched in length and tone, and only the culture told them apart. 

On the multiple-choice test, the big proprietary models cruised. GPT-5 and Gemini 2.5 Pro scored around 94 to 95% in both registers. Among smaller language models, the Arabic-specialized Hala-9B and SILMA-9B reached the high 70s and low 80s. Several others sat near 33%. ALLaM-7B managed 0.418 in MSA and 0.398 in dialect. Qwen3-8B got 0.354.

The dialect penalty turned up almost everywhere, though on the multiple-choice task it stayed modest. Questions about customs particular to one country were harder than questions about pan-Arab practices, for every model. North Africa was the worst region: the strongest open model dropped to 0.663 on country-specific dialect dialogues. Telling a model which country and region it was looking at nudged the scores up, a sign that the knowledge is in there but doesn’t always come out on its own.

Asked to translate dialogue, models handled dialect-to-MSA well: GPT-5 hit a BLEU score (a word-overlap measure) of 0.434 at that task, but only 0.276 going the other way. Asked to continue a dialogue in a named dialect, GPT-5 produced text that a judge model rated 0.956 for quality, yet an automatic dialect identifier matched it to the right country just 45% of the time. For the UAE, Libya, Sudan, and Yemen, that strict score was 0.000, even for GPT-5. Two Arabic models, Hala-9B and Qwen3-8B, barely produced dialect at all when asked to, scoring 0.02 and 0.04 on a scale where zero means pure MSA.

Fine-tuning the multilingual models on dialect data helped unevenly. It made them easier to steer and more fluent, and it sharpened dialect identity when a dialect had distinctive cues to grab. Moroccan, with its own heavy vocabulary, gained the most, jumping from 0.182 to 0.563 on the strict identifier for one model. Syrian output stayed fluent but kept sliding toward a generic Levantine or back toward MSA. More data, the authors report, didn’t reliably teach a model the line between one country’s speech and the next.

One challenge that is still unsolved is that the benchmark covers 13 of the 22 Arab countries and treats each as if it spoke one dialect, which it doesn’t; accent and custom move within borders too. 

A model can study an Emirati wedding, recognize cues such as the use of incense, and try to generate an appropriate response. Yet when asked to say it the way an Emirati would, it may still produce something that sounds Saudi. These systems are gradually learning to interpret the social context. Capturing the linguistic and cultural authenticity that makes them sound like insiders is still an open challenge.