This article walks through a hands‑on tutorial for building a multimodal, agentic Retrieval‑Augmented Generation (RAG) app that places text, URLs, PDFs, images, audio, and video into a single 768‑dimension embedding space. The two main components are Gemini Embedding 2 — which embeds all modalities into the same vector space — and the Google Agent Development Kit (ADK), which runs a small coordinator agent to convert retrieved evidence into grounded, citation‑friendly answers.
Why this matters
When your sources are heterogeneous (product PDFs, UI screenshots, recorded calls, demo videos, support notes), the simple “retrieve the right chunk” approach fragments into isolated embedding silos. Typical fixes require multiple pipelines and vector stores plus brittle glue. The approach shown here relies on one embedding space and a single retrieval per question, avoiding that complexity and improving citation consistency.
What we build
A working multimodal agentic RAG demo where you can drop in files or URLs and ask questions across the full index. The same retrieval packet feeds both the answer generation and the citation panel, preventing divergence between the UI and the agent. Key features:
- Truly multimodal index: text, URLs, PDFs, images, audio, and video live in one cosine‑similarity space with 768 dimensions.
- Gemini Embedding 2 with task prefixes: separate prefixes for documents and queries to improve retrieval quality.
- Google ADK agent: coordinates inspect_embedding_space and retrieve_relevant_context tools, then synthesizes a grounded answer.
- Single retrieval, two consumers: /ask performs one retrieval and passes the same packet to the agent and the UI.
- 3D PCA embedding view: each source is a single point; after asking a question the query and cited sources light up in the same projection.
- SSRF‑safe URL ingestion: private and loopback IPs blocked unless explicitly allowed.
End‑to‑end flow
-
Add sources. Text and URLs are chunked; files (PDF/image/audio/video) are uploaded. Each chunk/file receives a Gemini Embedding 2 vector with the "task: retrieval document" prefix. Files get a media vector blended with an annotation/text vector so titles and notes help retrieval.
-
Ask a question. The /ask endpoint embeds the query with "task: question answering | query", scores chunks by cosine similarity, keeps the best chunk per source, takes the top_k results, and projects vectors and the query into 3D via PCA.
-
Agent runs. run_adk_agent builds a per‑request agent whose retrieve_relevant_context tool is a closure returning the already‑computed retrieval packet. The agent inspects the embedding space, invokes the retrieval tool (which returns the cached packet), and writes a grounded answer without inline citation IDs.
-
UI renders. The frontend displays the answer text, the citation panel (built from the same matches), the agent trace, and the updated 3D view with the query point and highlighted sources.
The architecture’s key insight is the single‑retrieval contract: one query embedding, one ranked list of matches, and two consumers (agent and UI). That contract keeps citations honest and auditable.
Implementation details
- Constants used: EMBED_MODEL = "gemini-embedding-2", DEFAULT_DIMENSIONS = 768, CHUNK_WORDS ≈ 170, CHUNK_OVERLAP ≈ 35, INLINE_MEDIA_LIMIT_BYTES = 18 * 1024 * 1024.
- Task prefixes in embedding calls improve retrieval by signaling whether the content is a document or a query.
- Files: small images and PDFs are embedded inline; large files and all audio/video use the Gemini File API. The File API path uploads the file, polls until ACTIVE/SUCCEEDED, embeds via Part.from_uri, and deletes the uploaded file in a finally block to avoid storage leaks.
- Media + annotation blending: media vectors are blended with a text annotation vector (title + notes) at an example ratio (68% media / 32% text) so file labels remain searchable.
- Search: every chunk is scored, but only the best chunk per source is kept; source vectors and the query are projected together so the frontend’s 3D view uses the same basis; the backend returns a fully formed space snapshot so the frontend doesn’t need to requery.
- PCA projection: implemented in pure Python using power iteration to find the top three principal components, avoiding heavy numeric dependencies.
- Retrieval payload: the agent receives a clean payload (provider and a matches array with citation, source, modality, similarity, evidence). The same packet is returned to the frontend so UI and agent remain consistent.
ADK agent and closure injection
The ADK agent exposes two tools (retrieve_relevant_context and inspect_embedding_space) and a focused instruction that requires grounding answers in retrieved evidence and avoiding inline citation IDs. The server injects a closure for retrieve_relevant_context that returns the exact retrieval packet computed for the request; the agent believes it called a real retrieval tool while skipping redundant embedding work.
FastAPI endpoints
The service surface includes: /health, /space, /sources/text, /sources/url, /sources/file, DELETE /sources/{id}, and POST /ask. The /ask handler retrieves once, builds the retrieval payload, runs the ADK agent with a closure retrieval tool, collects an agent trace, and returns the final answer along with matches, query_point, trace, and the embedding space snapshot.
Safety and operational notes
- SSRF protection: URL ingestion validates schemes and resolves hostnames; if any resolved IP is private, loopback, link‑local, or reserved, ingestion is rejected unless ALLOW_PRIVATE_URLS is enabled.
- Blocking tasks are offloaded to run_in_threadpool (text chunking, file reads, search, PCA) so the FastAPI event loop stays responsive.
- CORS is configurable via ALLOWED_ORIGINS, defaulting to the Vite dev server during development.
Frontend (brief)
The frontend is a React + Vite app with three panels: a source manager, a Q&A panel that calls /ask and renders answers plus citations, and a Three.js 3D embedding view that uses the backend’s PCA coordinates. Each source is a colored point and after a question the query point and cited sources are highlighted.
Running the demo
Start the backend (python server.py) — it listens on http://localhost:8897. Start the frontend (cd frontend; npm install; npm run dev -- --port 5177). Add some sources (a paragraph, a public URL, a PDF, an image), watch points appear in the embedding view, ask a question, and inspect the answer, cited sources, and agent trace.
Conclusion and next steps
This implementation demonstrates a compact way to support multimodal retrieval and agentic answer synthesis without a separate vector database. Natural next steps include persisting to a managed vector DB (pgvector, Qdrant, Vertex AI Vector Search), adding re‑ranking between cosine retrieval and the agent, background ingestion for large media, creating eval sets to measure citation precision and faithfulness, adding auth and multi‑tenancy, and logging retrieval packets for observability and audits.
