# Recipe document

Source: https://mostlyright.md/docs/reference/recipe/

One JSON file names the dataset, the question, the sources, the SQL that builds the table, and the checks the result has to pass.

Register it with one call:

```sh
mr-data recipe RECIPE.json --json
```

The server puts the document into canonical form, takes its SHA-256, and registers it. One transaction covers the dataset, the question, every source, every connector configuration and the table plan.

## What comes back

```json
{
  "schema_version": "4.0.0",
  "recipe_id": "0a4f…",
  "recipe_digest": "9c1f…",
  "dataset_id": "4d01d86e-…",
  "table_id": "b7e2…",
  "source_ids": ["…", "…"]
}
```

| Field | Type | Meaning |
| --- | --- | --- |
| `schema_version` | `"4.0.0"` | the v4 envelope constant |
| `recipe_id` | UUID | the recipe resource |
| `recipe_digest` | 64 lowercase hex, unprefixed | the digest of the canonical document |
| `dataset_id` | UUID | the dataset the table belongs to |
| `table_id` | UUID | the table this recipe builds, stable across revisions |
| `source_ids` | array of UUID, 1 to 256 | one id per declared source, in declaration order |

`mr-data run` takes `recipe_id` and `recipe_digest` together, so the run states which recipe it means and which bytes of it.

`mr-data recipe show RECIPE_ID` reads the recipe back and adds `workspace_id`, `created_at` and the stored `document`.

## Two rules that cost a round trip

No fractional number appears anywhere in the document. The digest covers harness canonical bytes, which refuse floats, so every quantity is an integer or a decimal string. A value range bound is `"min_value": "-50.5"`, not `-50.5`. `mr-data recipe` refuses a fractional number by position before it sends anything, with `THIN_RECIPE_DOCUMENT_INVALID`.

The client does not compute the digest, and the document cannot state one. The submit body carries `document` and at most `workspace_id` and `require_category`. It has no digest property and refuses unknown ones. Registration is idempotent by the digest the server derived, so identical bytes sent twice register one recipe and return the same ids. A revised document produces a new `recipe_digest` against the same `dataset_id` and `table_id`.

## The document

Eight members, all required. Registration refuses anything else.

| Member | Type | Required | Meaning |
| --- | --- | --- | --- |
| `dataset` | object | yes | which dataset this table belongs to |
| `question` | object | yes | the question the table answers |
| `table` | object | yes | the table's name, grain and columns |
| `sources` | array, 1 to 256 | yes | where the rows come from |
| `transform` | object | yes | how the sources become the table |
| `checks` | array, 0 to 128 | yes | what has to hold about the result |
| `units` | array, 0 to 512 | yes | what the numbers are measured in |
| `timezone` | IANA name or `UTC` | yes | must be `UTC`, see [Time zone](#time-zone) |

An empty `checks` or `units` array is valid. The member itself is not optional.

### Identifier grammar

A table name, a column name, a source name, a step id and a check id are each an `identifier`: `^[a-z][a-z0-9_]{0,62}$`. Lowercase, starts with a letter, at most 63 characters.

## `dataset`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `dataset.name` | string | yes | 1 to 200 chars | the dataset's name |
| `dataset.id` | UUID | no | must belong to the registering workspace | bind to a dataset that already exists |
| `dataset.description` | string | no | 1 to 4000 chars | the initial page documentation, as Markdown; omit beside `dataset.id` and use `dataset set --description-file` for an existing page |

`dataset.id` is what `mr-data dataset create` printed. With it, registration attaches to a page that already exists. Without it, the server resolves the dataset from `(workspace_id, name)` and creates it when the name is free. A stated `name` that disagrees with a bound dataset's name does not rename it. A dataset in another workspace answers `409 DATASET_MISMATCH`.

`mr-data recipe` asks the server to enforce a saved primary category. In practice `dataset.id` is required and the dataset must already carry a category. Without the id the answer is `422 DATASET_ID_REQUIRED`, and without a saved category it is `422 DATASET_CATEGORY_REQUIRED`. The categories are in [Units and vocabulary](/docs/reference/units/#dataset-categories).

`dataset.id` must be a lowercase canonical UUID. `mr-data recipe` refuses anything else before sending rather than rewriting it.

Leave `description` out of a document that carries `id`. Registration writes it, so a later revision overwrites a description somebody edited by hand. Use `mr-data dataset set ID --description-file F` instead, and state `description` only where the registration creates the dataset.

## `question`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `question.text` | string | yes | 1 to 4000 chars | the question the table answers, written out |

## `table`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `table.name` | identifier | yes | `^[a-z][a-z0-9_]{0,62}$` | the table's name |
| `table.grain` | array of identifier | yes | 1 to 32, unique | the columns whose tuple is one row |
| `table.columns` | array of object | yes | 1 to 512 | the columns, in order |
| `table.description` | string | no | 1 to 2000 chars | a sentence about the table |

Table identity is `(workspace_id, dataset_id, name)` and holds across revisions. A revised document builds the same `table_id`, with no `_v2` renaming.

Every name in `grain` must be a column `columns` declares, or registration answers `422 RECIPE_DOCUMENT_INCOHERENT` with the pointer `/table/grain/N`. The grain is also the natural key a `key_uniqueness` check falls back to when it names no columns of its own.

### `table.columns[]`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `name` | identifier | yes | unique across the table | the column name |
| `type` | enum | yes | one of eight names | the declared output type |
| `nullable` | boolean | no | absent means `true` | whether it may be missing |
| `description` | string | no | 1 to 2000 chars | what the column means |
| `presentation` | object | no | none | what a reader should see |

`type` is one of:

```
string   integer   decimal   float   boolean   date   timestamp   json
```

`decimal` is exact and refuses a floating result. `float` takes the row `decimal` refuses, so declare it where the measurement is approximate. [Transforms](/docs/reference/transforms/#declared-types) says which engine result type satisfies each declared type.

`description` is optional to the schema and counted by the client. `mr-data recipe` reports `columns_without_description` on the receipt it prints, and names up to twelve of them as a warning. Registration goes through either way.

### `table.columns[].presentation`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `chart` | enum | yes | `histogram`, `timeline`, `top_values`, `share`, `none` | what to draw |
| `story` | string | no | 1 to 240 chars | the sentence a reader should be able to say |
| `bins` | integer | no | 4 to 12 on `histogram` and `timeline` only | how many marks the chart gets |
| `bucket` | enum | no | `hour`, `day`, `week`, `month`, `quarter`, `year`, on `timeline` only | the grain one bucket covers |

A chart describes only some column types, and registration refuses the pairs it cannot draw:

| `chart` | what it draws | allowed `type` |
| --- | --- | --- |
| `histogram` | a measurement, binned | `integer`, `decimal`, `float` |
| `timeline` | an event time, in contiguous buckets | `date`, `timestamp` |
| `top_values` | the commonest values, counted | anything but `json` |
| `share` | one split bar, the true part against the false part | `boolean` |
| `none` | no marks, and the distinct count in words | any type |

Registration refuses `bins` on a chart with no bins, and `bucket` on a chart with no grain, with the pointer `/table/columns/N/presentation/bins`.

A column that states no `presentation` still gets a chart, picked from the declared type. The dataset page records that the builder made the choice.

## `sources`

One entry per source, at least one and at most 256. A many-page collection is one entry however many pages it covers.

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `name` | identifier | yes | unique across the document | the relation name the SQL reads |
| `source_class` | enum | yes | one of eight names | what kind of source it is |
| `data_classification` | enum | yes | `public`, `internal`, `confidential`, `restricted` | how sensitive the bytes are |
| `locator` | object | yes | none | where it is |
| `rights_claim` | object | yes | none | the stated basis for using it |
| `connector` | object | yes | may arrive as [`connection`](#submitted-form-versus-registered-form) instead | how it is fetched |
| `limits` | object | yes | none | how much of it to fetch |
| `description` | string | no | 1 to 240 chars | one sentence saying what this source is |
| `credential` | object | no | required exactly when `credential_mode` is `opaque_reference` | a workspace secret or a saved connection |
| `binding` | enum | no | `text` (default) or `typed` | how its columns reach the SQL |
| `closed` | boolean | no | absent means `false` | this source's range is over |
| `window` | object | no | none | which address parameters carry time |
| `collection` | object | no | required on `public.https.collection@2.0.0`, refused elsewhere | how the pages of one corpus are listed |

`source_class` is one of:

```
mostlyright_sdk   external_adapter   user_file   user_url
user_api          database_extract   webhook      stream
```

The dataset page draws `description` beside the endpoint. Write it about the data, like `Daily climatological reports, the official highs and lows.`, not about the run.

`binding` decides what the transform sees. `text` is the default, and it binds every relation column `VARCHAR`. `typed` keeps the types the bytes carry: Parquet and JSON/NDJSON keep theirs, and the Reader infers CSV columns over the whole file.

### `sources[].locator`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `kind` | enum | yes | `sdk_connector`, `https_url`, `artifact`, `document_capture`, `webhook`, `stream`, `database` | what sort of address this is |
| `display_locator` | string | yes | 1 to 2048 chars | the address a person would recognise |
| `artifact` | object | required when `kind` is `artifact` | none | where the stored snapshot is |
| `connector_contract_version` | string | no | `^[1-9][0-9]*\.[0-9]+\.[0-9]+$` | the connector contract version |

`locator.artifact` carries `snapshot_id` (UUID), `content_digest` (`sha256:…`), `size_bytes` and `media_type`.

Where a source states both an `origin` and an `https_url` locator, the two must name the same host. The worker fetches it from that one host, even when a sibling source declares another.

### `sources[].rights_claim`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `claimed_basis` | enum | yes | one of six names | the stated basis for using the source |
| `claim_evidence_digest` | string | yes | `^sha256:[0-9a-f]{64}$` | the digest of the evidence for that claim |
| `claim_note` | string | no | 1 to 2000 chars | the claim in words |

```
unknown   prohibited   permission_asserted
public_domain_asserted   contractual_license_asserted   terms_of_service_asserted
```

The document records the claim its author makes, and the run carries it. Studio derives the rights outcome itself. Registration refuses `claimed_basis: "prohibited"`.

### `sources[].connector`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `adapter_id` | enum | yes | one of ten names | the adapter the worker acquires through |
| `credential_mode` | enum | yes | `none` or `opaque_reference` | whether a credential is presented |
| `origin` | string | conditional | canonical `https://host`, no port, path, query or fragment, ≤255 chars | the one host this source reaches |
| `parameters` | array | no | ≤64 entries | connector parameters |
| `generic_api` | object | required on the two generic adapters, refused elsewhere | none | the frozen request, auth, reader and pagination |
| `stream_connector_document_digest` | string | no | `sha256:…` | the connector document a capture ran under |

`adapter_id` is one of:

```
external.openligadb
public.https
public.https.collection@2.0.0
snapshot.artifact@1.0.0
authenticated.https.api_key@1.0.0
authenticated.https.api_key@2.0.0
authenticated.https.connector@3.0.0
stream.wss@1.0.0
stream.wss@2.0.0
document.projection@1.0.0
```

What each adapter requires:

| `adapter_id` | `credential_mode` | also required | forbidden |
| --- | --- | --- | --- |
| `authenticated.https.api_key@1.0.0` | `opaque_reference` | `origin`, exactly one `credential_scheme` parameter valued `bearer` or `x-api-key` | nothing |
| `authenticated.https.api_key@2.0.0` | `opaque_reference` | `origin`, `generic_api` | a `credential_scheme` parameter, `generic_api.limits`, `generic_api.job` |
| `authenticated.https.connector@3.0.0` | `opaque_reference` | `origin`, `generic_api`, `generic_api.limits` | a `credential_scheme` parameter |
| `public.https.collection@2.0.0` | `none` | `collection`, `limits.max_requests` | `window`, `closed` |
| every other adapter | any | nothing | `generic_api`, `collection` |

The adapters themselves are in [Source kinds and connectors](/docs/reference/sources/).

### `sources[].connector.parameters[]`

Each entry is a `name` and a text `value`.

| Path | Type | Required | Constraints |
| --- | --- | --- | --- |
| `name` | string | yes | `^[A-Za-z][A-Za-z0-9._-]*$`, 1 to 128 chars |
| `value` | string | yes | ≤2048 chars, except `reader.decode_options` at ≤262144 |

The grammar accepts any name. These names are reserved and mean something to Studio:

| Parameter | Used by | Meaning |
| --- | --- | --- |
| `reader.family_id` | any adapter that fetches | the Reader family the bytes are decoded with |
| `reader.family_version` | any adapter that fetches | that family's version |
| `reader.decode_options` | any adapter that fetches | the family's settings, as canonical JSON |
| `stream.members` | `stream.wss@*` | the exact members this source reads, comma separated |
| `stream.series` | `stream.wss@*` | the series whose members it follows |
| `document.receipt_digest` and seven siblings | `document.projection@1.0.0` | the capture's exact address |

The three `reader.*` parameters are all or none, and registration refuses one or two. The settings must be the canonical form of what the family accepts, with every default written out. Registration refuses a subset and quotes the spelling that works. It refuses any other `reader.`-prefixed name rather than ignoring it. See [Readers](/docs/reference/readers/).

A stream source names `stream.members` or `stream.series` or neither, never both. Registration refuses the pair.

`document.projection@1.0.0` requires all eight reserved `document.` parameters: `document.receipt_digest`, `document.raw_document_digest`, `document.projection_manifest_digest`, `document.projection_content_digest`, `document.candidate_manifest_digest`, `document.projection_policy_digest`, `document.media_type` and `document.source_version`.

### `sources[].limits`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `max_source_bytes` | integer | yes | ≥1 | where the fetch stops for this source |
| `max_rows` | integer | no | ≥1 | the row ceiling for this source |
| `max_requests` | integer | no | ≥1 | the per-run request ceiling |

The schema requires `max_source_bytes`. Exceeding this clamp fails every mode, including sample.

It says where the fetch stops, not how big the source is, so a large declaration registers. How far a source may fetch depends on how it will be read. A Reader-pinned source and an unpinned `csv` source reach 256 MiB. An unpinned `ndjson` source reaches 37,745,664 bytes. An unpinned `json` or `parquet` source reaches 16 MiB, and pinning a Reader is the way past that. [Limits](/docs/reference/limits/) carries the table. `mr-data recipe` warns on the receipt when a declaration is larger than a fetch can reach. A lower ceiling turns a `full` run into `CLAMP_EXCEEDED` and truncates a `sample`, so lowering it is not the fix for a source that really is larger. Split the range across several sources and union them in the first transform step.

### `sources[].credential`

Exactly one of two shapes, present exactly when `credential_mode` is `opaque_reference`:

```json
{"secret_name": "example_api_key"}
```

```json
{"connection_id": "…-uuid", "connection_version_digest": "sha256:…"}
```

| Path | Type | Constraints |
| --- | --- | --- |
| `secret_name` | string | `^[a-z][a-z0-9]*(?:[-_.][a-z0-9]+)*$`, ≤128 chars |
| `connection_id` | UUID | none |
| `connection_version_digest` | string | `^sha256:[a-f0-9]{64}$` |

The value never appears in the recipe, a receipt, an event or a log. The worker redeems a one-time handle at acquisition. A `secret_name` that is not enrolled in the workspace answers `422 SECRET_UNKNOWN` at registration. See [Source credentials](/docs/build/credentials/).

Registration refuses a credential that is present without `opaque_reference`, or absent with it, at `/sources/N/credential`.

### `sources[].closed`

`true` says the bytes behind this address cannot change any more. A full build or explicit resync
fetches it. With a predecessor, `closed: true` is the one condition that authorizes
`reuse_predecessor` in a strict refresh plan, and a closed source rides beside a request window in
a plan the current materializer executes. A plan of closed sources alone has no bounded partition
to replace, so Studio returns `RESYNC_REQUIRED` before acquisition. Omitting it means `false`.

Write it only where it holds whenever anybody asks. It states something about the source, not about a date. A query naming a range that has ended is no promise, because publishers correct historical data and republish it under the same address. A normal refresh is predecessor-bound; use a full build or explicit resync when a whole-source acquisition is intended.

The schema refuses `closed` on the collection adapter, and refuses `closed: true` beside a `window.snapshot`. A closed source skips the fetch that would stamp today's partition. Beside a request `window`, registration refuses `closed: true` with `422 RECIPE_DOCUMENT_INCOHERENT` at `/sources/N/closed`. Declare one or the other.

### `sources[].window`

Which parameters of this source's own address carry time, so a refresh asks for a bounded range instead of the whole corpus.

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `start_at` | UTC timestamp | yes | none | the first moment this corpus begins, a UTC midnight |
| `granularity` | const | yes | `"day"` | the unit every window is aligned to |
| `timezone` | const | yes | `"UTC"` | the zone the rendered parameters are read in |
| `lookback_seconds` | integer | yes | 0 to 2592000 | how far before the committed boundary a refresh re-requests |
| `max_span_seconds` | integer | yes | 3600 to 315360000 | the widest single request |
| `request` | object | exactly one of `request` or `snapshot` | none | which parameters carry the range |
| `snapshot` | object | exactly one of `request` or `snapshot` | none | partition by the acquisition day instead |
| `merge` | object | yes | none | how the slice becomes the stored relation |
| `physical_scopes` | object | no | refused beside `snapshot` | per-scope completeness bindings |
| `bootstrap` | object | no | refused beside `snapshot` | the seed scopes an empty table's first refresh requests |

`request.start` and `request.end` are each one endpoint:

| Path | Type | Required | Constraints |
| --- | --- | --- | --- |
| `encoding` | enum | yes | `date_parts`, `iso_date`, `epoch_seconds` |
| `bound` | enum | yes | `inclusive` or `exclusive`, what the publisher's parameter means |
| `parameters` | object | with `date_parts` only | `year`, `month`, `day`, each `^[A-Za-z0-9_.~-]{1,64}$` |
| `parameter` | string | with `iso_date` or `epoch_seconds` only | same grammar |
| `pad` | enum | `date_parts` only | `zero` (default) or `none` |
| `location` | enum | no | `query` (default) or `path` |

The engine's window is always half-open `[start, end)` in UTC, and the renderer converts. Start/inclusive renders the start, and start/exclusive renders the day before it. End/exclusive renders the end, and end/inclusive renders the day before it. `iso_date` renders `YYYY-MM-DD`. `epoch_seconds` renders the unpadded decimal seconds of that UTC midnight.

`merge` is how the slice becomes the relation:

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `materialization` | const | yes | `"partition_replace"` | the only merge there is |
| `partition.column` | identifier | yes | a column of the acquired relation | the cell that carries a row's time |
| `partition.key` | enum | yes | `iso_date_prefix`, `iso_date_value`, `compact_utc_hour` | how one day is read out of that cell |
| `row_identity` | array of identifier | yes | 1 to 8, unique | what the merged relation must be unique on |

`partition.column` is a column of the acquired relation, and one more rule applies where a refresh
plan names more than one source. Every window in such a plan has to name a `partition.column` the
table also declares, and all of them have to agree on the materialization and the partition key.
The multi-source lane writes one table manifest and maps no renamed field, so a partition column
that never reaches the table is refused with `RESYNC_REQUIRED` before acquisition. A single-source
windowed recipe has its own lane and takes a renamed column.

`window.snapshot` records a publisher's current state by acquisition day. It takes `column` (an identifier) and `max_partitions` (1 to 3660). Every decoded row carries the acquisition day. With it, `lookback_seconds` is `0` and `max_span_seconds` is `86400`. `merge.partition.column` must be that column, `merge.partition.key` must be `iso_date_value`, and `merge.row_identity` must carry the column. The source is never `closed`.

A snapshot address has no normal-refresh action. A normal or scheduled refresh returns
`RESYNC_REQUIRED` before acquisition. Use an explicit resync when you deliberately need another whole
snapshot; do not describe a snapshot's daily partition as a diff from its publisher.

The worker merges the slice into the previous run's relation before the transform runs. The transform, the checks and a replay all see the whole relation. A window that gained nothing ends the run `unchanged`.

### `sources[].collection`

Required on `public.https.collection@2.0.0` and refused on every other adapter. It has two members, `discovery` and `pages`. That adapter also requires `limits.max_requests` and `credential_mode: "none"`, and refuses `window` and `closed`. The full grammar is in [Many-page sources](/docs/build/collections/).

## `transform`

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `transform.engine` | enum | yes | `duckdb_sql` or `none` | the engine |
| `transform.steps` | array | yes | ≤64 entries | the steps, in array order |
| `steps[].step_id` | identifier | yes | unique, and never equal to a source `name` | the relation this step defines |
| `steps[].sql` | string | yes | 1 to 65536 chars | exactly one `SELECT` |
| `steps[].description` | string | no | 1 to 2000 chars | what the step does |

The schema accepts `engine: "none"` and an empty `steps`. A run refuses both with `TRANSFORM_PLAN_INVALID`. State `duckdb_sql` and at least one step.

Steps run in array order, and the last step's result is the table. [Transforms](/docs/reference/transforms/) has the dialect, the relation names, what the statement must return and what it may not read.

## `checks`

Deterministic checks the worker runs inline over the finished table. At most 128.

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `check_id` | identifier | yes | unique across the document | the name the result is reported under |
| `kind` | enum | yes | `key_uniqueness`, `null_ceiling`, `value_range`, `continuity`, `row_expectation` | what kind of check it is |
| `enforcement` | enum | yes | `required` or `advisory` | whether failing it stops the run |
| `columns` | array of identifier | no | ≤64, unique, each a declared column | which columns it is about |
| `max_null_ppm` | integer | no | 0 to 1000000 | null ceiling in parts per million |
| `min_value`, `max_value` | decimal string | no | `^-?(?:0\|[1-9][0-9]{0,38})(?:\.[0-9]{1,18})?$` | the inclusive range |
| `min_rows` | integer | no | ≥0 | the row floor |
| `max_rows` | integer | no | ≥1 | the row ceiling |
| `max_gap_seconds` | integer | no | ≥1 | the widest gap allowed |
| `description` | string | no | 1 to 2000 chars | what the check is for |

The schema puts every optional bound on every kind. [Checks](/docs/reference/checks/) says which bounds a kind actually reads, and what happens when a document states one it does not.

## `units`

One entry per column that has a unit.

| Path | Type | Required | Constraints | Meaning |
| --- | --- | --- | --- | --- |
| `units[].column` | identifier | yes | must be a declared column | the column |
| `units[].unit` | string | yes | 1 to 64 chars | the code its numbers are in |

Declaration order is part of the digested document, and no column appears twice under two spellings of one key. Registration checks that the column exists and that the code resolves in the accepted subset. It returns `422 RECIPE_DOCUMENT_INCOHERENT` at `/units/N/column` or `/units/N/unit` when one does not. The accepted vocabulary is in [Units and vocabulary](/docs/reference/units/).

## Time zone

`timezone` accepts any IANA zone name or `UTC` at registration, and the worker builds every table in `UTC`. A document naming another zone registers and then refuses on its first run:

```
RUN_RECIPE_INVALID: the member recipe.timezone states America/New_York, and every table
this worker builds is built in UTC; state UTC and put the conversion in the steps, where
the result of it is sealed
```

State `UTC` and write the conversion into the SQL.

## Refresh cadence

The recipe document carries no cadence member. Refresh control is per source: a request window narrows
what a refresh fetches, `closed` can describe predecessor reuse, and `window.merge` says how the slice
becomes the relation. A source with no normal-refresh action, or an action shape with no current bounded
materializer, makes Studio return `RESYNC_REQUIRED` before acquisition rather than re-read a whole
source. `mr-data promote` records the schedule and `mr-data reschedule` changes it. See
[Publish and keep it fresh](/docs/build/publish-and-refresh/).

## Recorded streams

`stream.wss@1.0.0` (an open venue) and `stream.wss@2.0.0` (a keyed one) name a source built from a capture rather than from an address. The source states a `stream` locator whose `display_locator` is the `wss://` endpoint, `credential_mode: "none"`, and `connector.stream_connector_document_digest`, the digest of the connector document the capture ran under. A build never reaches the venue.

The relation is fixed, and the document does not declare it. The connector picks one of two shapes: eight text columns, or nine when it declares a member field, with `member` fourth, directly after `event_time`. A connector that says nothing about attribution writes the eight columns it has always written.

| Column | Content |
| --- | --- |
| `epoch` | the connection this frame arrived on, as text |
| `event_id` | the identity the Recorder assigned the frame |
| `event_time` | the declared event time in the connector's kind, or the RFC 3339 arrival time when that frame was timed by receipt |
| `member` | which member of the venue the frame is about, present only when the connector declares a member field. A table can partition by it without parsing the message |
| `message` | the provider frame, as canonical JSON text |
| `message_class` | `snapshot`, `delta`, or whatever else the connector classifies |
| `message_type` | the venue's own message type |
| `received_at` | RFC 3339 UTC receipt time |
| `sequence` | the frame's position within its connection, as text |

`event_time` can hold two kinds of value row by row, so a blind cast returns nulls for half the column with no error. See [Live streams](/docs/build/live-streams/).

## The digest

`recipe_digest` is `sha256(canonical_json_bytes(document))`, unprefixed lowercase hex. Canonical form sorts object keys, uses the tightest separators and emits UTF-8 without ASCII escaping. It refuses floats, `NaN`, duplicate keys, byte-order marks and integers outside the 64-bit signed range.

The server hashes the document after it resolves the [`connection` shorthand](#submitted-form-versus-registered-form) and before anything else. It hashes every other member as written. Any edit, a reordered array, a changed description, one more space inside a SQL string, produces a different digest and a different recipe.

The digest is also the idempotency key, so it takes the unprefixed 64-character form. Every other digest on the v4 wire carries the `sha256:` prefix.

## Submitted form versus registered form

One member exists on the way in and not in the stored document. A source may arrive with `connection` instead of `connector` and `credential`:

| Path | Type | Required | Meaning |
| --- | --- | --- | --- |
| `connection.connection_id` | UUID | yes | the granted saved connection |
| `connection.version_digest` | `sha256:`-prefixed digest | yes | the connection version being used |
| `connection.parameter_values` | object, at most 32 members, each a string, integer or boolean | yes | what the connection's own parameters are set to for this source |
| `connection.response` | object | yes | how the response is read, in the same shape a `connector` source states it |
| `connection.limits` | object | no | the per-source limits, when this source narrows them |

The server replaces it with the resolved connector and a credential reference before it takes the digest. The stored document and the digest describe the resolved connector. The dataset must already hold an explicit grant for that connection.

## A complete document

This validates against `recipe.schema.json`. Two sources, one windowed and one closed, joined in two steps, with every check kind and two declared units.

```json title="recipe.json"
{
  "dataset": {
    "id": "4d01d86e-1f2a-4c3b-9e7d-8f1a2b3c4d5e",
    "name": "New York Central Park weather history, 2026"
  },
  "question": {
    "text": "What was the hourly air temperature and relative humidity at Central Park in 2026?"
  },
  "table": {
    "name": "station_hourly_readings",
    "description": "One row per station and observation time, from 2026-01-01 onward.",
    "grain": ["station", "valid"],
    "columns": [
      {
        "name": "station",
        "type": "string",
        "nullable": false,
        "description": "The reporting ASOS station identifier.",
        "presentation": { "chart": "top_values" }
      },
      {
        "name": "station_name",
        "type": "string",
        "nullable": false,
        "description": "The station's published name.",
        "presentation": { "chart": "none" }
      },
      {
        "name": "valid",
        "type": "timestamp",
        "nullable": false,
        "description": "The instant the observation was taken. The acquired relation names this column valid, and the table declares it under that name so a refresh can partition by it.",
        "presentation": {
          "chart": "timeline",
          "bucket": "day",
          "story": "Readings arrive every hour with no long outages."
        }
      },
      {
        "name": "air_temp_c",
        "type": "decimal",
        "nullable": true,
        "description": "Air temperature at the station, converted from the published Fahrenheit reading.",
        "presentation": { "chart": "histogram", "bins": 8 }
      },
      {
        "name": "relative_humidity_pct",
        "type": "decimal",
        "nullable": true,
        "description": "Relative humidity at the station.",
        "presentation": { "chart": "histogram", "bins": 8 }
      }
    ]
  },
  "sources": [
    {
      "name": "asos_observations",
      "description": "Hourly airport surface observations, the official temperature and humidity readings.",
      "source_class": "user_url",
      "data_classification": "public",
      "locator": {
        "kind": "https_url",
        "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=NYC&data=tmpf&data=relh&year1=2026&month1=1&day1=1&year2=2026&month2=9&day2=12&format=onlycomma&tz=Etc/UTC&report_type=3&latlon=no&missing=empty&trace=empty"
      },
      "rights_claim": {
        "claimed_basis": "public_domain_asserted",
        "claim_evidence_digest": "sha256:fc6ec632f676ccb16126dc24d86f07ffc4bb1051fc7c299d8b91762628050510",
        "claim_note": "Federal observations redistributed by the Iowa Environmental Mesonet, whose archive page states the data are in the public domain."
      },
      "connector": {
        "adapter_id": "public.https",
        "credential_mode": "none",
        "origin": "https://mesonet.agron.iastate.edu",
        "parameters": [{ "name": "format", "value": "onlycomma" }]
      },
      "limits": {
        "max_source_bytes": 3145728,
        "max_rows": 65536,
        "max_requests": 4
      },
      "window": {
        "start_at": "2026-01-01T00:00:00Z",
        "granularity": "day",
        "timezone": "UTC",
        "lookback_seconds": 172800,
        "max_span_seconds": 2592000,
        "request": {
          "start": {
            "encoding": "date_parts",
            "bound": "inclusive",
            "pad": "none",
            "parameters": { "year": "year1", "month": "month1", "day": "day1" }
          },
          "end": {
            "encoding": "date_parts",
            "bound": "exclusive",
            "pad": "none",
            "parameters": { "year": "year2", "month": "month2", "day": "day2" }
          }
        },
        "merge": {
          "materialization": "partition_replace",
          "partition": { "column": "valid", "key": "iso_date_prefix" },
          "row_identity": ["station", "valid"]
        }
      }
    },
    {
      "name": "station_metadata",
      "description": "The published station roster, one row per ASOS station.",
      "source_class": "user_url",
      "data_classification": "public",
      "closed": true,
      "locator": {
        "kind": "https_url",
        "display_locator": "https://mesonet.agron.iastate.edu/sites/networks.php?network=NY_ASOS&format=csv&nohtml=on"
      },
      "rights_claim": {
        "claimed_basis": "public_domain_asserted",
        "claim_evidence_digest": "sha256:fc6ec632f676ccb16126dc24d86f07ffc4bb1051fc7c299d8b91762628050510"
      },
      "connector": {
        "adapter_id": "public.https",
        "credential_mode": "none",
        "origin": "https://mesonet.agron.iastate.edu"
      },
      "limits": { "max_source_bytes": 1048576 }
    }
  ],
  "transform": {
    "engine": "duckdb_sql",
    "steps": [
      {
        "step_id": "readings",
        "description": "Carry the publisher's own valid column through, cast the readings into the declared types, and drop rows with no timestamp.",
        "sql": "select station, valid, cast((try_cast(tmpf as decimal(6,2)) - 32) * 5 / 9 as decimal(6,2)) as air_temp_c, try_cast(relh as decimal(5,2)) as relative_humidity_pct from asos_observations where valid is not null and valid <> ''"
      },
      {
        "step_id": "joined",
        "description": "Attach the station name, read the publisher's timestamp as a UTC instant, and return the declared columns.",
        "sql": "select r.station, m.station_name, cast(concat(r.valid, ':00+00') as timestamp with time zone) as valid, r.air_temp_c, r.relative_humidity_pct from readings r join (select stid as station, name as station_name from station_metadata) m on m.station = r.station"
      }
    ]
  },
  "checks": [
    {
      "check_id": "grain_unique",
      "kind": "key_uniqueness",
      "enforcement": "required",
      "columns": ["station", "valid"],
      "description": "One row per station and observation time."
    },
    {
      "check_id": "temperature_present",
      "kind": "null_ceiling",
      "enforcement": "required",
      "columns": ["air_temp_c"],
      "max_null_ppm": 20000,
      "description": "At most two per cent of readings may be missing a temperature."
    },
    {
      "check_id": "humidity_in_range",
      "kind": "value_range",
      "enforcement": "required",
      "columns": ["relative_humidity_pct"],
      "min_value": "0",
      "max_value": "100"
    },
    {
      "check_id": "hourly_continuity",
      "kind": "continuity",
      "enforcement": "advisory",
      "columns": ["valid"],
      "max_gap_seconds": 7200,
      "description": "Two consecutive observations more than two hours apart is a reporting outage."
    },
    {
      "check_id": "enough_rows",
      "kind": "row_expectation",
      "enforcement": "required",
      "min_rows": 24,
      "max_rows": 1000000
    }
  ],
  "units": [
    { "column": "air_temp_c", "unit": "Cel" },
    { "column": "relative_humidity_pct", "unit": "%" }
  ],
  "timezone": "UTC"
}
```

In that document `grain` is `["station", "valid"]`, and `grain_unique` names the same pair. Step `joined` is the last step, so its result is the table. `asos_observations` declares a request window whose incremental action re-requests only the last two days plus whatever is new. Its `merge.partition.column` names `valid`, a column of the acquired relation, and the table declares a column of that name as well. `station_metadata` declares `closed: true`, which describes predecessor reuse. A closed source may ride beside a request window in a strict refresh plan. This plan names two sources, so the window's partition column has to be a column the table declares. It is one here, so Studio admits the refresh and the table is partitioned by the day it reads off that column.

Step `readings` carries `valid` through under the publisher's own name, and step `joined` reads it as a UTC instant. Deriving an `observed_at` from it instead would leave the table unpartitioned and the refresh refused. Both sources bind all-text, so every declared type is an explicit cast. `air_temp_c` is declared `decimal`, so the statement ends in a `cast(… as decimal(6,2))`. A bare division would come back `DOUBLE` and the run would refuse it. `humidity_in_range` states `min_value` and `max_value` as strings, because the document carries no fractional number.

## What registration refuses

| Status and code | When |
| --- | --- |
| `422 CONTRACT_VALIDATION_FAILED` | the document does not satisfy the JSON Schema |
| `422 RECIPE_DOCUMENT_INCOHERENT` | the document is internally inconsistent: a grain naming an undeclared column, duplicate column, check or step ids, a chart a type cannot carry, a credential that does not match `credential_mode`, a source declared `prohibited` |
| `422 RECIPE_DOCUMENT_NOT_CANONICAL` | the document cannot be canonicalized, because of a float or an oversized integer |
| `422 DATASET_ID_REQUIRED` | category enforcement was requested and `dataset.id` is absent |
| `422 DATASET_CATEGORY_REQUIRED` | the bound dataset has no saved primary category |
| `422 SECRET_UNKNOWN` | a `credential.secret_name` is not enrolled in the workspace |
| `409 DATASET_MISMATCH` | `dataset.id` names a dataset in another workspace |

A `RECIPE_DOCUMENT_INCOHERENT` answer carries one detail per problem, each with a JSON Pointer into the document and a reason.

Two classes of refusal land later, at run time rather than at registration: the transform dialect (`TRANSFORM_*`, `RUN_RECIPE_INVALID`) and the check dialect (`DECLARED_CHECK*`). Both cost a queued run. [Transforms](/docs/reference/transforms/) and [Checks](/docs/reference/checks/) list them.