Build a dataset

Follow the build from finding a source to publishing and refreshing the finished dataset.

Check the sources, write a recipe, inspect its capabilities with recipe explain, and forecast the work. Test on a scratch table when you need to inspect a bounded run. Then run the real table in full and verify its rows and coverage. The first successful run makes a new table live; a refresh schedule works only when its recipe supports it. How a build works explains the objects and states involved.

Stage	Guide	What it covers
1	Probe a source	The catalogue, saved connections, reading a source before you commit to it, when to stop.
2	Write a recipe	The document top to bottom, the rules agents get wrong, registration and the identifiers it returns.
3	Run and inspect	Safe testing, one full build, status and events, checks, preview, bounded queries, receipts and download.
4	Publish and keep it fresh	What going live means, cadence, reschedule, pin and unpin, demote, rename and retire.
	Make builds efficient	Forecast, reuse, source pacing and observed performance.
	Source credentials	Storing a credential by name and naming it from a recipe.
	Live streams	Recording a wss:// venue and building a table that grows from it.
	Many-page sources	Bounded collections of pages, and the crawler behaviour operators see.
	Documents and files	Document projection and the scientific formats.
	Fix a failed run	The failures an author sees, what each one means, what to change.

The CLI reference specifies every command these guides use, the recipe document reference every field, and the recipe examples every shape.