One of this morning's AIE keynotes covered dspy, which reminded me to test whether it could help improve the system prompts used by Datasette Agent. To explore this I launched an asynchronous research task in Claude Code for web, using Claude Fable 5.
The experiment's goal was practical: after pip installing the latest Datasette alpha, datasette-agent and dspy packages, determine how to use DSPy to evaluate and improve the main system prompts that Datasette Agent uses when it executes read‑only SQL queries to answer user questions.
Claude Fable 5 proposed several directions and chose to test with GPT‑4.1 mini and GPT‑4.1 nano. The automated testing turned up multiple promising areas for improvement, and one concrete issue stood out:
- The schema listing in the current prompt provides only table names and omits column names. Coupled with guidance in the prompt that says “don't call describe_table if you already have the information”, models ended up guessing column names (examples seen in baseline traces include page_count, o.order_id, first_name) and entering error‑retry loops.
Based on these findings, two straightforward mitigation options emerged: include column names in the prompt's schema listing, or relax the advice discouraging calls to describe_table. Either approach would help the model generate more accurate queries and avoid guesswork and repeated error recovery.
Why this matters
Datasette Agent generates SQL queries automatically to answer user questions, so the content and structure of its system prompts directly affect query correctness and user experience. When prompts omit schema details, models may resort to guessing, which increases the incidence of incorrect queries and costly retry loops.
This experiment demonstrates how DSPy and automated evaluation can identify specific prompt weaknesses that are addressable with small changes, reducing errors in LLM‑driven data access.
Next steps
From the results, the practical next steps to improve Datasette Agent prompts include:
- Listing column names alongside table names in schema outputs.
- If listing column names is infeasible, softening the instruction that discourages using describe_table so the model can safely request schema details when needed.
- Running further DSPy‑based evaluations with different models and real traces to quantify the impact of prompt changes on error rates and query accuracy.
The quick test highlights how evaluation tooling like DSPy can aid prompt engineering for LLM integrations with databases and help reduce common failures such as unnecessary guessing and retry loops.
