Agile Deliberation is a two‑stage framework that treats building classifiers for subjective visual categories as an iterative human–AI conversation. Instead of asking humans to supply complete, static definitions up front, the system helps users scope a concept and then iteratively refine it by surfacing semantically borderline images. The authors have released the implementation on GitHub and provide an interactive Colab/Jupyter demo.
Why this approach matters
Standard machine learning workflows assume annotators have clear, stable category definitions before labeling. That assumption holds for objective categories (for example, a stop sign or a golden retriever) but breaks down for subjective domains such as "Healthy Food", "inspiring artwork", or nuanced content moderation tasks. People rarely possess fully specified mental models for such concepts; they reveal exceptions and edge cases only when confronted with real examples. Traditional processes force rigid labels onto models and do not support this natural evolution of human intent.
The two stages of Agile Deliberation
Agile Deliberation decomposes the task into concept scoping and concept iteration, turning a fuzzy idea into a precise instructional blueprint for a model.
Stage 1 — Concept scoping
The system guides users to decompose a broad topic into a hierarchy of positive subcategories and explicit exclusions. For example, a user might split "Healthy Food" into subcategories like "Salads," "Grain Bowls," and "Lean Proteins," while listing exclusions such as "Processed Foods" or "Raw, Unprepared Ingredients." This structured baseline anchors the model's initial understanding.
Stage 2 — Concept iteration
With an initial structure in place, Agile Deliberation moves to active testing and collaborative refinement. Drawing on active learning ideas, the system uses vision‑language models (VLMs) to mine and present semantically borderline images that challenge the current text definitions.
For example, a fruit smoothie image might force the user to decide whether it belongs to "healthy" because it contains raw fruit or should be excluded due to added sugars and processing. By confronting ambiguous edge cases, users surface hidden assumptions, clarify intent, and update policy rules. User feedback is routed into an automatic prompt optimization loop so the VLM's classification boundaries evolve in step with the user's refined definitions.
Live user study
Because subjective classifiers lack static ground truth, the authors evaluated the approach with a functional prototype in 18 live user sessions, each 1.5 hours long. Participants interacted with Agile Deliberation and three baseline paradigms in randomized order to mitigate learning effects.
Domains tested:
- Healthy Food — a domain influenced by personal health philosophies, dietary restrictions, and cultural background.
- Pay‑to‑Play — a content moderation problem requiring expert nuance to identify clickbait or misleading images promising financial or digital rewards.
Baseline methods compared:
- Zero‑shot classifiers: a single initial prompt without visibility into edge cases or adaptation.
- Modeling collaborator: an automated top‑down agent that decomposes concepts without keeping the human in the loop.
- Manual deliberation: the manual, spreadsheet‑and‑prompt‑editing workflow.
Results
The live sessions showed that Agile Deliberation improves classifier performance and the user experience.
-
Consistency and performance: Participants who used the structured deliberation process produced significantly more consistent labels over time. Resolving boundary ambiguities early led to classifiers that outperformed all baselines, especially on the hardest, most subjective images.
-
Exposing blind spots: Expert users reported that the system effectively diagnosed gaps in their initial definitions by surfacing confusing borderline images they had not anticipated.
-
Reducing cognitive fatigue: Manual prompt engineering imposed a marked cognitive burden; participants reported fatigue and anxiety that changing a prompt to fix one issue could harm other parts of the classifier. Agile Deliberation mitigates this by presenting curated edge‑case galleries and accepting natural feedback, which the system then absorbs and uses to update the model automatically. Every participant preferred Agile Deliberation over manual methods and reported less stress and mental exhaustion.
Conclusion
Agile Deliberation reframes human‑in‑the‑loop machine learning from a one‑way transfer of human labels to a two‑way collaboration where humans and AI jointly refine subjective concepts. This approach matters for deploying ML systems on complex, socially relevant tasks — for example, supporting trust and safety teams in moderation, helping curators shape digital collections, or aiding scientists in categorizing nuanced visual data.
The paper describes the active learning techniques and the scoping and clustering algorithms in detail; the implementation is open‑source on GitHub, and an interactive Colab/Jupyter demo walks through the "Healthy Food" example so practitioners can adapt the approach to their own domains.
Acknowledgements
Contributors: Wei Qiao, Thomas Denby, Krishnamurthy Viswanathan, Enming Luo, Chun‑Ta Lu, Tushar Dogra, Ranjay Krishna, and Ariel Fuxman. The research was conducted through a collaboration between Google Research and Google Ads.
