BigSet is a new open-source tool from TinyFish that turns a single-sentence natural-language description of a desired dataset into a structured table. The system gathers information from live web sources, verifies and deduplicates results, and attaches citations to the returned records.
How it works, briefly
- The user describes the dataset in one sentence — for example: "YC companies currently hiring engineers, with their funding stage, location, and number of open roles."
- BigSet infers the schema automatically: it determines column names, data types, and primary keys, so no manual schema design is required.
- Multiple AI agents operate in parallel across the web to research and verify data against real sources.
- The tool deduplicates entries, returns a clean table with citations, and allows export to CSV or XLSX.
Refresh schedules
A key feature is automatic refresh scheduling: users can choose a cadence from every 30 minutes up to once per week so datasets stay up to date without manual intervention.
Deployment and license
BigSet is released as open-source software under the AGPL-3.0 license. It can be self-hosted and the project notes that it can be started with a single Docker command.
Technology stack
The project ships as a full stack application: a Next.js 16 frontend, a Fastify backend, Mastra workflows for agent orchestration, and it relies on TinyFish's Search and Fetch APIs for web research under the hood.
Why this matters
BigSet lowers the barrier to producing structured, refreshable datasets when sources are distributed across the web and change frequently. Automatic schema inference, parallel agent research, and scheduled updates can reduce manual data collection effort while keeping provenance via citations.
Caveats
The description outlines the tool's capabilities but does not provide independent verification of result accuracy or performance at scale. Because BigSet relies on external web sources and AI agents, users should evaluate source reliability and legal or ethical considerations for their specific use cases.
