Why every language model struggles with the same culture questions

Suppose you want to know whether a banyan tree is healthy. You could inspect the leaves, but if you look at just a handful, that tells you almost nothing, and there are far too many to check them all. The other option is to look at the trunk and the roots, which are the structures holding every leaf up.

Judging whether a language model understands culture faces a similar choice. Most evaluations check the leaves: does the model know the currency of one country, or what people eat in one region versus another one? The facts are worth knowing, but the questions are endless, and a model can rattle off hundreds of them without any deeper grasp of how they connect.

A paper presented at ICML 2026 goes after the trunk and roots instead. Rather than test whether a model knows isolated facts, the authors ask whether it captures the structures that tie those facts together; whether it sees, say, that a national dish isn’t a fact on its own but part of a web of countries that share ingredients, cooking styles, and eating habits while keeping their local differences. They measure how well eight models reproduce these real-world patterns of similarity and variation across 170 countries and nine cultural domains, among them food, religion, language, holidays, and currency. Newer instruction-tuned models track the human patterns better than older ones, but bigger is not reliably better. And the same measure better predicts whether a model can help someone from an unfamiliar culture, on a practical task like recommending a recipe.

To analyze the trunk and roots, the authors introduce a spectral analysis-based framework that shifts cultural evaluation from probing local microstructures (the leaves) to modeling domain-level macrostructure (the trunk and roots). To build their macrostructure evaluation metrics, they use Cultural Consensus Theory (CCT), a well-established anthropological framework, for recovering shared beliefs from patterns of agreement, along with a principle from physics that a system has macro properties (the macrostructures), which you can’t read off its parts (the microstructures). 

In other words, if a database stores facts, a language model, the authors argue, compresses them into structure, and it’s that structure that they believe is worth interrogating.

For each of the nine cultural domains, the researchers ask a model the same question of all 170 countries and read off its probability distribution over an exhaustive list of possible answers. Stack those distributions and compute an adjacency matrix, and you get a map of which countries the model thinks resemble one another. 

Two numbers then describe the map’s shape. The first, Effective Rank, counts how many distinct patterns the map holds, a rough measure of pluralism. The second, the Spectral Gap Ratio, measures how much countries agree: high when everyone looks alike, low when each goes its own way. 

Together, they sort every domain into one of four boxes:

1. Low ER, High SR (LH): Few dominant patterns (low pluralism) that are widely shared across countries. Structurally, this resembles a fully connected graph or a uniform clique. Domains such as house numbers fall here, as their distributions look similar across countries.
2. High ER, Low SR (HL): Many patterns (high pluralism) but little cross-country similarity. The network resembles disconnected or random graphs, where each country follows its own path. Currency is a typical example, as each nation has its own system with little overlap.
3. High ER, High SR (HH): Many patterns, yet countries form affinity clusters. The network resembles small-world structures with a dense core and peripheral clusters. Food-related domains illustrate this case, where global staples coexist with strong local variations.
4. Low ER, Low SR (LL): Few dominant patterns, but countries split into separate clusters rather than forming a universal consensus. The network fragments into distinct agreement clusters, each centered on a different core. Religion is a representative case, with a handful of world religions dominating, but countries grouping into different blocks.

Eighty people across 16 world regions, five apiece and paid £9 an hour, ranked the nine domains by how much they’d expect answers to vary from country to country, forming the ground truth the models were scored against. The survey ran on Prolific after two rounds of pilot testing, with attention checks to filter out anyone who hadn’t read the instructions; failed checks were discarded. People put convenient foods and house numbers near the common end, national dish, and currency near the country-specific end.

The result shows that models don’t see it that way. All eight models filed food next to currency (high variation, no shared structure) as if a country’s cuisine were as unique to it as its currency. The authors read this as evidence that current models learn national food as a set of discrete, stereotyped labels, stripped of the regional clustering and shared ingredients a person takes for granted.

In terms of which model performed better, Llama-8B came out on top, ahead of its 70B sibling. Gemma’s 2B and 9B versions scored about the same. The authors interpret this result as a plateau: instruction-tuned models stop improving somewhere around two billion parameters, and increasing their size buys little! GPT-2, at 124 million parameters but showing its age, always defaulted to the “everyone’s alike” box, never registering that cultures vary at all; the numbers only stabilize once a model crosses some threshold of size and training. These results contrast with fact-based benchmarks like MMLU, where bigger models pull ahead as expected because the two kinds of tests measure different things.

The team built a second experiment around recipe recommendation to see whether any of this mattered in practical use. A simulated user, driven by GPT-4o and seeded with real breakfast personas from ten volunteers, chatted with six models playing recipe recommender. A judge scored each conversation on how well the final recipe fit the person and how hard the model worked to understand them first. Across 1,200 conversations, the Llama models that scored well on macrostructure also asked better questions and gave more fitting recipes. Gemma, strong on the older fact-based benchmarks, lagged. Aya-8B improved only when explicitly nudged to ask about a user’s background.

Seven of the original volunteers rated a blind sample of the conversations, and their rankings tracked the automated judge’s. But the same people were underwhelmed by the recipes. They found them repetitive and shallow, mostly providing minor variations on a single dish or clichés.

Nine domains are not the whole of human culture. Also, the models used in the test were relatively mid-sized (the upper limit was 70B) and open-weight, leaving the large closed systems untested. And everything runs in English, though the structures they claim to measure shouldn’t depend on language. Even with those limits, no model ever got the food category right, and scale rescued none of them.

That leaves the question the paper raises (but doesn’t answer). If pretraining bigger and instruction-tuning harder won’t teach a model that cuisines have neighbors, what will? The authors suspect the answer isn’t more of the same, but training aimed squarely at relational, pluralistic knowledge: models built to grasp that a fact about one culture only means something set beside the others.