Recipe examples
Thirteen worked recipe documents, one per source shape, each complete enough to copy, edit and register.
Each page carries one whole recipe document in a single fence, the commands that register and run it, and the mistakes that shape makes easy. Each document validates against the v4 recipe schema as printed.
Read Write a recipe first if you have not written one before, and Recipe document for the field-by-field reference.
What each page covers
Section titled “What each page covers”| Source shape | Page | Connector and Reader | What it shows |
|---|---|---|---|
| One published CSV response | One CSV file | public.https, direct CSV |
the smallest complete document: one source, one step, three checks |
| A daily series from two stations | Daily city temperatures | public.https, direct CSV |
a union, an aggregation, and a request window that refreshes a few days |
| A JSON API | JSON API | public.https + json.tabular@1.1.0 |
a records pointer, nested column pointers, and two ways to paginate |
| Many HTML pages behind one index | Many HTML pages | public.https.collection@2.0.0 + html.web_extract@1.1.0 |
page discovery, fetch budgets, and the seven provenance columns |
| A table inside a PDF | Documents and PDFs | document.projection@1.0.0 |
the document lane, and why mr-data cannot reach it yet |
| A weather model file | Weather GRIB2 | public.https + weather.grib2@3.0.0 |
exact message selection, named places, and a closed source |
A wss:// venue |
WebSocket stream | stream.wss@1.0.0 |
recording before building, and the fixed eight-column frame relation |
| A changing stream roster from a live table | Table-driven stream discovery | table_members series + stream.wss@1.0.0 |
member selection, typed context, freshness, limits and segment replay |
| Two feeds joined | Two sources joined | public.https + json.tabular@1.1.0, twice |
a left join, and the check that catches a join that fanned out |
| Benchmark scores and release dates | Benchmark scores and release dates | public.https + json.tabular@1.1.0 and delimited_text@2.0.0 |
a key the recipe builds from two names, and the rights basis each publisher needs |
| A forecast and what happened | Forecast against observation | public.https + weather.grib2@3.0.0 |
a model file joined to the station it predicted, and the error columns |
| A listing that keeps no history | Snapshot windows | public.https + json.tabular@1.0.0 |
window.snapshot, and why an explicit full resync can replace live material |
| Twenty stations of one kind | Twenty stations, one hourly table | public.https, twenty times |
a twenty-way union, per-source byte ceilings, and a learned cadence |
| A recorded stream reduced | A stream aggregated into bars | stream.wss@1.0.0 + json.tabular@1.1.0 |
one-minute bars, and the two headers a recorder seals |
What every page assumes
Section titled “What every page assumes”The dataset page exists before the recipe does. mr-data dataset create --name "…" --json prints a
dataset_id for the document’s dataset.id. Without it, registration resolves a container from the
name and creates a second page that is not connected to your existing dataset. These examples print dataset.name and leave
dataset.id out, so add your own before you register.
When you bind dataset.id, omit dataset.description. Write the page description with mr-data dataset set DATASET_ID --description-file description.md --name "Title" --topics "topic" instead.
You do not compute a digest by hand. Submit the document as written. The backend
canonicalizes the bytes, takes the digest of that, and returns recipe_id, recipe_digest,
dataset_id, table_id and source_ids. Registering identical bytes twice registers one recipe.
Every run needs both identifiers: mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST. The
digest is the bare lowercase hex the registration printed, with no sha256: prefix. The CLI refuses
mr-data run RECIPE_ID.
Never sample the real table as a preflight: its first successful sample can go live with truncated data. Use recipe test to inspect a bounded test report, or register a separate scratch recipe under a different table name. Run and inspect explains both paths. --max-rows limits rows per source; it does not bound collection fetches.
The first succeeded run goes live. There is no promote click. mr-data promote records cadence for
an eligible incremental table; it does not make the table readable. A snapshot or other unwindowed
source refuses ordinary refresh before acquisition and needs an explicit full resync instead.
Reading the documents
Section titled “Reading the documents”No fractional number appears anywhere. Every number is a count, a byte ceiling, a parts-per-million
bound or a number of seconds. A check bound that is not a whole number travels as a decimal string:
"min_value": "-2".
Every acquired source reaches the SQL as text. The engine binds each source as all-text CSV, so
every declared type becomes an explicit cast in the statement. A column declared decimal must come
back as DECIMAL(p,s), and a column declared timestamp as TIMESTAMP WITH TIME ZONE.
A Reader pin lives on connector.parameters, under three reserved names: reader.family_id,
reader.family_version and reader.decode_options. Write all three or none. The options value is
the canonical JSON text of what the family itself admits, with every defaulted setting written out.
A stated subset is refused.
The real datasets behind these shapes
Section titled “The real datasets behind these shapes”Datasets of every shape here run on this platform:
- LLM benchmark scores joined to model release dates
- NOAA GEFS forecasts
- a twenty-city weather dataset
- corpora built from HTML collections
- a Y Combinator company snapshot
Browse them at Datasets. Copy these documents and edit them. They do not reproduce those datasets byte for byte.