Two US‑based economists, Robert Novy‑Marx and Mihail Velikov, describe in a recent study how large language models (LLMs) can be used to generate complete scientific papers with minimal human intervention. The authors implemented and tested an automated research pipeline within the field of financial economics.
Method and key numbers
- The researchers began by assembling roughly 30,000 accounting variables.
- Using statistical screening, they narrowed this set down to 96 variables deemed potentially useful for financial research.
- For each of the 96 variables, they used an LLM‑based system to automatically generate a full paper.
Each generated paper included distinct hypotheses, forecasts, and literature citations, and the documents met typical formal and substantive expectations for academic papers.
Findings: strengths and weaknesses
The authors evaluated the quality of the AI‑produced documents and found that:
- Several papers revealed interesting and relevant economic relationships;
- In some cases, the texts contained fabricated (invented) citations;
- The approach enables processing of datasets at a scale that is difficult to handle efficiently with traditional methods.
The study also highlights a methodological concern familiar from human research: the tendency to form hypotheses after seeing results (post‑hoc hypothesizing), which the AI‑driven process can replicate.
Implications for academia
The case study shows that LLMs can reshape scientific workflows by speeding up hypothesis generation and manuscript production while introducing new risks. Fabricated citations are a particular worry because they can undermine the reliability of scholarly communication.
According to Defacto’s commentary, it is unlikely that AI will fully replace researchers in the near term, but the polished, professional appearance of the automatically generated papers is notable.
Conclusion
Applying LLMs in scientific research offers clear benefits in terms of speed and scalability, yet raises substantial ethical and methodological challenges. Research communities and publishers should strengthen verification mechanisms to reap the advantages of automation while limiting the risk of false or misleading scientific output.
