404 Media has published details about an internal OpenAI review workflow called Project Lily. According to the report, human workers — described as "prompt reviewers" — inspect anonymized ChatGPT conversations and rate the quality of the model’s responses.
What the reviewers do
Reviewers assess whether a response actually answers the user’s question and whether it contains certain undesirable elements, such as AI jargon, a condescending tone, excessive emoji use or overt flattery. The guidelines also forbid the model from presenting itself with human attributes (anthropomorphizing) or claiming personal experiences — for example, it should not say "as a chef I..." or "I know how that feels."
The version of the conversation delivered to reviewers reportedly includes a "user summary" that encapsulates the user’s questions and interests, provides context, and in some cases may include the user’s location.
Limits of anonymization and privacy implications
Although OpenAI says chats are anonymized before human review, the report notes that certain information can still be revealed to reviewers, particularly in shorter conversations. A source told the outlet that most users likely do not realize strangers may read their exchanges, which raises concerns given that many people treat ChatGPT as a casual confidant or therapist and share intimate details.
Pay and working conditions
A worker quoted in the article described the task as visually intensive but said it pays relatively well: more than $50 per hour, which the report converts roughly to over 15,000 forints. That rate is notable given the apparently simple nature of the task.
What Project Lily does not do
404 Media emphasizes that Project Lily does not evaluate the factual accuracy of conversations. That suggests at least one other team or process may be responsible for assessing factual correctness and other dimensions of model performance.
OpenAI’s responses and transparency
The report says OpenAI initially did not provide a substantive reply to questions about whether users were explicitly informed that humans might read their chats. The company later shared a link to an FAQ page that addresses human review in model development; OpenAI indicated the page had been updated 29 days earlier, while Tom's Hardware reported that the FAQ has existed for at least two years or longer.
Broader context
These disclosures come amid ongoing scrutiny of how AI companies use copyrighted material and user data. For example, The New York Times and the Daily News sued OpenAI and Microsoft at the end of 2023, alleging that their articles had been used without permission to train language models; reporting around that litigation also noted that potentially relevant evidence was at one point accidentally deleted.
The details of internal review processes and the limits of anonymization remain central to debates about transparency and user privacy in the development of large language models.
