# How a build works Source: https://mostlyright.md/docs/start/how-it-works/ A recipe becomes a run. When the run succeeds, it writes an immutable table version. ## The shape of a build A successful run writes a new table version. The version contains one or more Parquet parts and records their digests. It never changes. The run also writes a receipt that records how the version was built. ```text dataset ─────────────────────────────────────────────────────────────┐ (a container of tables; one primary category, one description) │ │ recipe document ──register──▶ recipe_id + recipe_digest │ (sources, table plan, SQL, checks, units, timezone) │ │ │ │ mr-data run --recipe … --digest … --mode sample|full|refresh │ ▼ │ run ── acquire ─▶ parse ─▶ transform ─▶ check ─▶ persist │ │ │ │ succeeded │ ▼ │ table version (a manifest over one or more Parquet parts) │ │ │ │ first succeeded run promotes the table automatically │ ▼ │ live pointer ──▶ newest passing version ◀── refresh cadence ───────┘ │ ├─▶ dataset page (app.mostlyright.md/datasets/…) ├─▶ Public Dataset API (bounded JSON + Parquet download) └─▶ MCP (an AI tool reading the same version) ``` ## The objects | Object | What it is | How you get at it | | --- | --- | --- | | Dataset | A container of tables, with a name, one primary category, a description and a decision record. No build happens at this level. | `mr-data dataset create`, `dataset show`, `dataset note` | | Table | One built table inside a dataset. Identity is `(workspace, dataset, table name)`, and it stays stable across recipe revisions. | `mr-data table TABLE_ID` | | Recipe document | One JSON object: dataset identity, question, sources, table plan, transform plan, declared checks, declared units, timezone. The digest covers this object. | the file you write | | `recipe_id` and `recipe_digest` | The server canonicalizes the document, digests the canonical form and registers every entity in one transaction. `recipe_digest` is bare lowercase hex. Registration is idempotent by that digest. | `mr-data recipe recipe.json` | | Run | One execution of one recipe digest against one table. It records its mode, limits and status. A successful run also records `table_version_id` and `candidate_digest`. | `mr-data run`, `status`, `watch`, `runs` | | Run mode | `sample`, `full`, `refresh`, `backfill`, `compact`, `replay`. One run command, six modes. `--mode MODE` always works, and four modes also have a shorthand flag. | `--sample`, `--full`, `--refresh`, `--backfill`, `--mode compact`, `--mode replay` | | Table version | What a succeeded run writes: a manifest over one or more Parquet parts, with a row count, a total size and a digest. Immutable. | `mr-data parts TABLE_ID`, `verify RUN_ID` | | Parts | The Parquet files that make up a version. Several runs may have written one version's parts, and a single manifest names all of them. | `mr-data parts`, `download --kind table_part` | | Live pointer | Which version readers get. It tracks the newest passing version on its own. Pin it to freeze one version, unpin to resume tracking. | `mr-data pin`, `unpin`, `demote` | | Promote | It does not make a table readable. It records the refresh cadence and the reasoning behind it, and it puts a withdrawn table back. | `mr-data promote TABLE --cadence … --why …` | | Stream subscription | A recorder keeps a `wss` connection open and creates a new version at each checkpoint instead of executing the recipe in a worker. | `mr-data stream`, [Live streams](/docs/build/live-streams/) | | Secret reference | A source needing a credential sets `credential_mode: "opaque_reference"` and a sibling `credential: {"secret_name": "…"}`. The value never appears in a recipe, a receipt, an event or a log. The worker redeems a one-time handle at acquisition. | `mr-data keys`, [Source credentials](/docs/build/credentials/) | The [recipe reference](/docs/reference/recipe/) covers every field. ## The run state machine Seven states, one machine: ```text mr-data run │ ├──▶ awaiting_confirmation ──── --confirm ─────┐ │ │ ├──▶ awaiting_sample_approval ── released by ──┤ │ an editor │ │ ▼ └────────────────────────────────────────▶ queued ──▶ running │ succeeded ◀─────────┤ failed ◀─────────┤ cancelled ◀─────────┘ (cancel is legal while a run is queued, held or running) ``` | State | What it means | | --- | --- | | `queued` | The backend accepted the run, and it waits for a worker. | | `running` | A worker is executing the run and reporting progress. | | `awaiting_confirmation` | Over the projection threshold. The run exists, and it needs confirming before it starts. | | `awaiting_sample_approval` | An existing legacy preview hold or a repair hold. New full builds do not create sample/full pairs. | | `succeeded` | Terminal. Includes `outcome` and, unless the outcome was `unchanged`, `table_version_id`. | | `failed` | Terminal. Includes the required `failure_code`, `failure_detail` and `failed_stage`. | | `cancelled` | Terminal. `mr-data run --cancel RUN_ID` is legal while a run is queued, held or running. | A succeeded `refresh` can report `outcome: "unchanged"`. Every source's bytes were identical to the predecessor's, so the worker stopped after acquisition. The worker ran no transform and wrote no version. The table's last-refresh time advances and the live pointer does not move. Every other mode always writes a version. On a sample, a row or window clamp truncates and the run succeeds, and `coverage.truncated` reports it. A run cannot remain `running` forever. Worker failures produce typed errors. A separate watchdog also marks a run as failed when its execution disappears and records an infrastructure `failure_code`. ## The five worker stages ```text acquire ──▶ parse ──▶ transform ──▶ check ──▶ persist ``` | Stage | What happens | What fails here | | --- | --- | --- | | `acquire` | The worker fetches each source from the one host it declares, under its own `limits`. | Publisher errors, redirects, media types, byte ceilings, credentials | | `parse` | The fetched bytes become a relation, from CSV directly or through the reader the source pinned. | Reader mismatches, malformed documents, decode budgets | | `transform` | The `duckdb_sql` steps run in order. The last step's result is the table. | SQL errors, missing columns, non-deterministic functions | | `check` | The declared checks run inline over the result. A required check that fails fails the run. | `CHECK_FAILED` and the check's own detail | | `persist` | The worker writes and uploads the Parquet parts, the manifest, the preview, the column profile and the receipt. | Upload and signing faults | `failed_stage` on a failed run is one of these five. The run's event log records one durable event type per stage: `run_progress_acquire`, `run_progress_parse`, `run_progress_transform`, `run_progress_check` and `run_progress_persist`, plus `run_succeeded`, `run_failed` and `run_narrative_appended`. Progress ticks arrive per source, per step and per check, so `mr-data watch` can say "2 of 7 sources" instead of "acquiring". [Run states and errors](/docs/reference/run-states-and-errors/) lists the codes by stage. ## What is immutable, and what moves Immutable: - A table version and every part in it. A later run writes a new version and never edits one. - A recipe digest. Different bytes are a different digest. - A run record, once terminal, and the receipt it wrote. A receipt names the `recipe_digest` that ran, the source digests, the timings and the check results. Moves: - The live pointer, to the newest passing version, unless you pin it. - The refresh schedule, which the platform adjusts from what each refresh observes about the source, unless you lock it. - Dataset metadata: name, description, topics, licence, who can read it. - Table name, with `mr-data table rename`. Runs, versions and the schedule stay attached to the identifier, which does not move. A revised recipe yields a new digest against the same table. There is no `_v2` table and no renaming. Failed runs and abandoned revisions cost nothing. ## Ceilings and the confirmation gate The plan is flat. A run adds nothing to your bill. The gate below holds a run because of its size, not its price. The refusal carries a `projected_cost`, which is an internal estimate of what the platform spends on infrastructure. You are never billed for it. Limits supplied with a run are called `clamps`: `--max-rows`, `--max-source-bytes` and `--window`. Their behavior depends on the run mode. [Run and inspect](/docs/build/run-and-inspect/) explains each combination. Source limits are declared under `sources[].limits`: `max_source_bytes`, `max_rows` and `max_requests`. They are part of the digested document and bind every run of it. ### The confirmation gate and progressive checkpoint Every new full build uses progressive acquisition. A long build exposes an inspection checkpoint after about five minutes, then continues automatically in the same run using the pages it has already captured. The checkpoint is read-only, not a second approval gate or a runtime limit. Small builds may finish before the checkpoint is useful. | Gate | State | Refusal | Who settles it | | --- | --- | --- | --- | | Run confirmation | `awaiting_confirmation` | HTTP `409 CONFIRM_REQUIRED`, reported by the CLI as `THIN_STUDIO_CONFIRM_REQUIRED` with `{run_id, projected_bytes, projected_runtime_s, projected_cost}` | Any authenticated caller in the paying workspace. `mr-data run … --confirm` re-states the request and authorizes the projection already on the record. | | Existing legacy preview approval | `awaiting_sample_approval` | Existing pairs only; new full admissions never dispatch a separate preview. | Any editor credential of the workspace, once the legacy preview has sealed a checked table and the build is authorized. `POST /v4/runs/{run_id}:approve-full` keeps its `expected_version` fence. Repair holds require a person in a signed-in browser. `--confirm` does not settle either hold. | `mr-data run` prints the projection and a `confirm_command` that approves it, then exits `2`. An existing legacy held full run's record names the sample as `sample_run_id`, and the sample names the full as `full_run_id`. ## Refresh, and where the cadence comes from The first succeeded run makes a table live and dispatches no catch-up. After that: 1. `mr-data promote TABLE --cadence … --why …` records a seed schedule and the reasoning, runs a catch-up backfill from the last successful run's bookmark, and attaches the recurring schedule. 2. Each refresh compares the sha256 of every source's acquired bytes against the predecessor's digest for that source. It reports one observation per source on success. 3. The platform adjusts the applied schedule from those observations. 4. A failed refresh never advances the live pointer, and the platform tells the workspace the cause. A schedule runs only where the recipe can refresh. Every source has to declare a continuation: a `window` carrying a `request`, `closed: true`, a `collection`, or a recorded stream. The plan's shape has to be materializable too: every source a recorded stream, or at least one collection-free request window, with `closed` sources allowed beside it. Anything else answers `RESYNC_REQUIRED` before acquisition, and the table keeps serving the version its last run sealed. [Publish and keep it fresh](/docs/build/publish-and-refresh/) carries the whole gate. A table whose source is a daily snapshot has a cadence floor. The window partitions on the run's own acquisition day, so a day nobody runs is a hole no later run can fill. The learned schedule may only tighten past the window's granularity, never loosen. An accepted schedule is not evidence a refresh ran. `mr-data table TABLE_ID` reports `last_refresh`, `next_refresh_at` and `last_failure`. `mr-data reschedule TABLE --cadence … --why …` changes the schedule without starting a run or spending anything. `--lock` stops the loop adjusting it, and `--unlock` resumes. A second `promote` on a live table is an idempotent replay that ignores the new cadence, so use `reschedule`. Both commands accept three cadence forms: a five-field cron expression, an interval, or the word `source` to let the platform choose. An interval reads `every 6h` or `every 30m from 2026-09-03T12:20:00Z`. Both commands require `--why`, unless you asked for `source`. ## Next - [Build your first dataset](/docs/start/first-dataset/): the same lifecycle as commands you can paste. - [Write a recipe](/docs/build/write-a-recipe/): the document, field by field. - [Run and inspect](/docs/build/run-and-inspect/): modes, ceilings, and reading a run back. - [Publish and keep it fresh](/docs/build/publish-and-refresh/): promotion, cadence, pinning. - [Limits and ceilings](/docs/reference/limits/): the numbers the contract enforces. - [Run states and errors](/docs/reference/run-states-and-errors/): every state, event and code.