Chester Ismay and Michelle Smith describe an ongoing project that hands common NFL arguments to an automated AI agent and verifies them against play-by-play data from nflverse. The series, called The Monday Morning Agent, uses roughly a quarter-million plays: five complete seasons of public nflverse data plus plays from the current season as it unfolds.
The series will run eight episodes; each episode gives the agent an additional capability. By the finale the agent is intended to operate the desk unattended, and the authors will review what it investigated and what it properly refused to answer.
Define the claim: what does “establish the run” mean?
The first episode tackles the familiar claim “You have to establish the run.” The authors note this phrase can mean different things — for example, that early runs pay off later, that runs wear down a defense, or that a running game sets up play-action passing. Each interpretation requires different data. For this episode the focus is whether teams that run more often early in a game (the first half) win more often.
Before calling the tool, the authors asked the agent to write one sentence stating what would have to be true in the data for the claim to hold, and to specify when in the game the claim applies. They emphasize that “establish” here is meant to be an early-game claim, so whole-game averages are not an adequate test. They advise giving your agent the same assignment and checking that it answers the question you intended.
Give the agent a job you can check, and document limits
Before the agent ran any queries, the authors ran the analysis code themselves: the tool did the counting, and the agent interpreted the results. They then wrote the agent’s policy in a Markdown file that spelled out the agent’s job, the read-only tool it may use, and what the answer needed to include. The agent must explain what the plays cannot tell us and is not allowed to claim that running causes winning — the analysis measures association, not causation.
When asked questions outside that job — for example, which team to bet on — the agent refused without calling the tool because betting advice was beyond its scope. Other viewer questions tangled with the “establish the run” instructions, so the authors stress: as you give an agent more responsibility, check both its results and whether it understood the task.
Filters change the story; show them in the answer
At first glance the data looked decisive: over the last five seasons, teams with the highest run share (the percentage of their plays that were runs) won 77.7% of their games, while those with the lowest run share won 19.2%. That gap initially seemed to settle the question.
However, much of that gap reflects teams that were already leading and ran to drain the clock. When the agent restricted the analysis to first-half running in games that were within three points at halftime, the result reversed: teams with the highest first-half run share won 46.3% of the time; those with the lowest won 55.3%. The current season through Week 5 points the same way, but each group contained only 11 team games, so a single game could swing the result. This does not mean passing causes winning — the analysis measures association, and the filters change which games are being compared.
Because a single filter altered the entire story, the authors require that filters be visible in the agent’s output. They wanted to see how many rows remained and which commands produced the results so they could verify and reproduce the work. Make those details part of an agent’s answer, especially when the sample is small.
A few numbers from last week’s games
In Week 5, the team that ran more in the first half won 6 of the 15 games, lost 8, and tied 1 (Giants at Washington). Over the past five seasons that team won 54.3% of the time, so last week was below that average. That’s not unusual: in 27 of the 95 regular-season weeks since 2021, the team that ran more early finished the week with a losing record. One week of games doesn’t say much by itself, which is why the agent reports five seasons alongside the current one.
One concrete example: the Monday night game, Buffalo at the Los Angeles Rams, was the first game the agent had not seen. Buffalo ran on 60% of its first-half plays compared with the Rams’ 35%, led by six at halftime, and won by three in overtime. Those 199 plays are now in the dataset, and adding that game did not change the close-at-halftime numbers used in the episode’s calculations.
Next up: “Ban the Tush Push” (October 26)
The next episode will tackle “Ban the Tush Push” on October 26. The authors note that in the 260,927 play descriptions from 2021 through the October 11 games there is no instance of the phrase “tush push.” After adding Monday night’s game the database contains 261,126 descriptions, and the word “sneak” appears in only two descriptions, both from 2021. The data alone will not hand them a definition.
Before the agent can judge a play, it must define it. The authors will require the agent to write that definition into its first permanent document — a shared data dictionary — and to cite it in every subsequent ruling. They invite readers to bring their own definitions because the verdict depends on which definition is chosen.
Closing thoughts
The project’s aim is to build AI agents that not only run queries but also work from clear, checkable assignments with documented filters and explicit statements of what the data cannot show. The first episode demonstrates that conclusions are sensitive to the details of the analysis: filters and sample sizes change outcomes, so an agent’s answer must include those details and its own limitations.
