Build a dataset

Follow the build from finding a source to publishing and refreshing the finished dataset.

A build follows one loop. Check the sources, write a recipe and run a small sample. Inspect the result and revise the recipe before running it in full. The first successful run makes the table live, and a refresh schedule keeps it current. How a build works explains the objects and states involved.

Stage	Guide	What it covers
1	Probe a source	The catalogue, saved connections, reading a source before you commit to it, when to stop.
2	Write a recipe	The document top to bottom, the rules agents get wrong, registration and the identifiers it returns.
3	Run and inspect	The sample run, ceilings, status and events, checks, preview, bounded queries, receipts, the full run, download.
4	Publish and keep it fresh	What going live means, cadence, reschedule, pin and unpin, demote, rename and retire.
	Source credentials	Storing a credential by name and naming it from a recipe.
	Live streams	Recording a wss:// venue and building a table that grows from it.
	Many-page sources	Bounded collections of pages, and the crawler behaviour operators see.
	Documents and files	Document projection and the scientific formats.
	Fix a failed run	The failures an author sees, what each one means, what to change.

The CLI reference specifies every command these guides use, the recipe document reference every field, and the recipe examples every shape.