Researchers at Princeton University and the University of Chicago tested several large language models (LLMs) in a simulated hiring task and found that the models developed stereotypes faster and more strongly than human participants in a comparable psychology experiment.
Experiment design
The researchers adapted a prior psychology study. Each model was told it had been hired as a consultant by the mayor of a fictional city and asked to help hire for 20 different jobs (for example, doctors, lawyers, child‑care aides and janitors). Candidates came from four fictional ethnic groups: Tufa, Aima, Reku and Weki. In every round there was a single opening and four candidates, one from each group. After each hire the model was told whether the chosen candidate succeeded, and the process repeated for 40 rounds. Unknown to the models, all candidates had an equal probability of success in every job.
Key findings
After only a few outcome observations, the models began to segregate groups into different jobs. For instance, if a model observed an Aima fail as a doctor (a role in the study associated with higher warmth and competence), it tended to stop hiring Aimas for doctor positions and instead hire them for janitor roles, which it treated as lower in warmth and competence.
The researchers measured segregation on a scale where 2 means each group is completely confined to its own job niche. Human participants in the original study scored 0.84 on this scale. The LLMs scored roughly 65% higher on average; OpenAI’s reasoning model o3 scored 1.83, close to the maximum possible.
Why models generalize quickly
Ryan Liu, a PhD student at Princeton University and a coauthor of the study presented at ICML in Seoul in July, notes that LLMs are optimized to generalize from limited data. That capability is advantageous for solving math, coding and science tasks, but in social decision contexts it can produce premature hunches and stereotypes. Newer models with stronger reasoning abilities, such as OpenAI o3 and DeepSeek R1, showed even stronger biases in the experiment.
Memory, personalization and amplification of bias
The result matters as chatbots gain better memory and personalization. Angelina Wang (Cornell University), who was not involved in the study, warned that when a chatbot uses prior conversation history it can over‑index on behaviors it has seen before and form biased generalizations. Simply reducing how much a chatbot remembers is not an obvious fix, because users often want systems to retain some personal context.
Mitigation strategies tested
Telling the models to "be fair" had little effect on behavior: the researchers suggest either the models could not operationalize the instruction or the drive to maximize correct hires dominated. However, giving models an explicit bonus for diverse hiring — embedding social values into their objectives — substantially reduced bias.
In another part of the study, researchers asked models to resettle members of different ethnic groups across Canadian cities. When the models received personally relevant information (for example, age and education), they were less likely to segregate by ethnicity. When given irrelevant details (for example, hair color or tattoo shape), the models largely reverted to sorting by ethnicity.
Real‑world implications
The experiment differs from real hiring in that models received immediate feedback about each hire’s success. In real organizations, it can take a long time to learn whether a hire is a good fit. Still, when feedback eventually arrives, a model that learns from experience could overweight limited signals and reinforce biased patterns.
As companies increasingly deploy LLMs to screen résumés and even conduct interviews, the study’s finding that models can form novel biases from their own experience is a significant concern. These emergent biases may not mirror any human prejudice present in the training data, yet they can have serious ethical and practical consequences that developers and employers need to address.
