Skip to content

How a build works

A recipe becomes a run. When the run succeeds, it writes an immutable table version.

The shape of a build

Section titled “The shape of a build”

A successful run writes a new table version. The version contains one or more Parquet parts and records their digests. It never changes. The run also writes a receipt that records how the version was built.

  dataset ─────────────────────────────────────────────────────────────┐
   (a container of tables; one primary category, one description)      │
                                                                       │
   recipe document ──register──▶ recipe_id + recipe_digest             │
   (sources, table plan, SQL, checks, units, timezone)                 │
        │                                                              │
        │ mr-data run --recipe … --digest … --mode sample|full|refresh │
        ▼                                                              │
   run ── acquire ─▶ parse ─▶ transform ─▶ check ─▶ persist            │
        │                                                              │
        │ succeeded                                                    │
        ▼                                                              │
   table version  (a manifest over one or more Parquet parts)          │
        │                                                              │
        │ first succeeded run promotes the table automatically         │
        ▼                                                              │
   live pointer ──▶ newest passing version  ◀── refresh cadence ───────┘
        │
        ├─▶ dataset page        (app.mostlyright.md/datasets/…)
        ├─▶ Public Dataset API  (bounded JSON + Parquet download)
        └─▶ MCP                 (an AI tool reading the same version)

The objects

Section titled “The objects”
Object What it is How you get at it
Dataset A container of tables, with a name, one primary category, a description and a decision record. No build happens at this level. mr-data dataset create, dataset show, dataset note
Table One built table inside a dataset. Identity is (workspace, dataset, table name), and it stays stable across recipe revisions. mr-data table TABLE_ID
Recipe document One JSON object: dataset identity, question, sources, table plan, transform plan, declared checks, declared units, timezone. The digest covers this object. the file you write
recipe_id and recipe_digest The server canonicalizes the document, digests the canonical form and registers every entity in one transaction. recipe_digest is bare lowercase hex. Registration is idempotent by that digest. mr-data recipe recipe.json
Run One execution of one recipe digest against one table. It records its mode, limits and status. A successful run also records table_version_id and candidate_digest. mr-data run, status, watch, runs
Run mode sample, full, refresh, backfill, compact, replay. One run command, six modes. --mode MODE always works, and four modes also have a shorthand flag. --sample, --full, --refresh, --backfill, --mode compact, --mode replay
Table version What a succeeded run writes: a manifest over one or more Parquet parts, with a row count, a total size and a digest. Immutable. mr-data parts TABLE_ID, verify RUN_ID
Parts The Parquet files that make up a version. Several runs may have written one version’s parts, and a single manifest names all of them. mr-data parts, download --kind table_part
Live pointer Which version readers get. It tracks the newest passing version on its own. Pin it to freeze one version, unpin to resume tracking. mr-data pin, unpin, demote
Promote It does not make a table readable. It records the refresh cadence and the reasoning behind it, and it puts a withdrawn table back. mr-data promote TABLE --cadence … --why …
Stream subscription A recorder keeps a wss connection open and creates a new version at each checkpoint instead of executing the recipe in a worker. mr-data stream, Live streams
Secret reference A source needing a credential sets credential_mode: "opaque_reference" and a sibling credential: {"secret_name": "…"}. The value never appears in a recipe, a receipt, an event or a log. The worker redeems a one-time handle at acquisition. mr-data keys, Source credentials

The recipe reference covers every field.

The run state machine

Section titled “The run state machine”

Seven states, one machine:

  mr-data run
        │
        ├──▶ awaiting_confirmation ──── --confirm ─────┐
        │                                              │
        ├──▶ awaiting_sample_approval ── released by ──┤
        │                                an editor     │
        │                                              ▼
        └────────────────────────────────────────▶ queued ──▶ running
                                                                 │
                                             succeeded ◀─────────┤
                                                failed ◀─────────┤
                                             cancelled ◀─────────┘
                                            (cancel is legal while a run is
                                             queued, held or running)
State What it means
queued The backend accepted the run, and it waits for a worker.
running A worker is executing the run and reporting progress.
awaiting_confirmation Over the projection threshold. The run exists, and it needs confirming before it starts.
awaiting_sample_approval An existing legacy preview hold or a repair hold. New full builds do not create sample/full pairs.
succeeded Terminal. Includes outcome and, unless the outcome was unchanged, table_version_id.
failed Terminal. Includes the required failure_code, failure_detail and failed_stage.
cancelled Terminal. mr-data run --cancel RUN_ID is legal while a run is queued, held or running.

A succeeded refresh can report outcome: "unchanged". Every source’s bytes were identical to the predecessor’s, so the worker stopped after acquisition. The worker ran no transform and wrote no version. The table’s last-refresh time advances and the live pointer does not move. Every other mode always writes a version.

On a sample, a row or window clamp truncates and the run succeeds, and coverage.truncated reports it.

A run cannot remain running forever. Worker failures produce typed errors. A separate watchdog also marks a run as failed when its execution disappears and records an infrastructure failure_code.

The five worker stages

Section titled “The five worker stages”
acquire ──▶ parse ──▶ transform ──▶ check ──▶ persist
Stage What happens What fails here
acquire The worker fetches each source from the one host it declares, under its own limits. Publisher errors, redirects, media types, byte ceilings, credentials
parse The fetched bytes become a relation, from CSV directly or through the reader the source pinned. Reader mismatches, malformed documents, decode budgets
transform The duckdb_sql steps run in order. The last step’s result is the table. SQL errors, missing columns, non-deterministic functions
check The declared checks run inline over the result. A required check that fails fails the run. CHECK_FAILED and the check’s own detail
persist The worker writes and uploads the Parquet parts, the manifest, the preview, the column profile and the receipt. Upload and signing faults

failed_stage on a failed run is one of these five. The run’s event log records one durable event type per stage: run_progress_acquire, run_progress_parse, run_progress_transform, run_progress_check and run_progress_persist, plus run_succeeded, run_failed and run_narrative_appended. Progress ticks arrive per source, per step and per check, so mr-data watch can say “2 of 7 sources” instead of “acquiring”.

Run states and errors lists the codes by stage.

What is immutable, and what moves

Section titled “What is immutable, and what moves”

Immutable:

  • A table version and every part in it. A later run writes a new version and never edits one.
  • A recipe digest. Different bytes are a different digest.
  • A run record, once terminal, and the receipt it wrote. A receipt names the recipe_digest that ran, the source digests, the timings and the check results.

Moves:

  • The live pointer, to the newest passing version, unless you pin it.
  • The refresh schedule, which the platform adjusts from what each refresh observes about the source, unless you lock it.
  • Dataset metadata: name, description, topics, licence, who can read it.
  • Table name, with mr-data table rename. Runs, versions and the schedule stay attached to the identifier, which does not move.

A revised recipe yields a new digest against the same table. There is no _v2 table and no renaming. Failed runs and abandoned revisions cost nothing.

Ceilings and the confirmation gate

Section titled “Ceilings and the confirmation gate”

The plan is flat. A run adds nothing to your bill. The gate below holds a run because of its size, not its price. The refusal carries a projected_cost, which is an internal estimate of what the platform spends on infrastructure. You are never billed for it.

Limits supplied with a run are called clamps: --max-rows, --max-source-bytes and --window. Their behavior depends on the run mode. Run and inspect explains each combination.

Source limits are declared under sources[].limits: max_source_bytes, max_rows and max_requests. They are part of the digested document and bind every run of it.

The confirmation gate and progressive checkpoint

Section titled “The confirmation gate and progressive checkpoint”

Every new full build uses progressive acquisition. A long build exposes an inspection checkpoint after about five minutes, then continues automatically in the same run using the pages it has already captured. The checkpoint is read-only, not a second approval gate or a runtime limit. Small builds may finish before the checkpoint is useful.

Gate State Refusal Who settles it
Run confirmation awaiting_confirmation HTTP 409 CONFIRM_REQUIRED, reported by the CLI as THIN_STUDIO_CONFIRM_REQUIRED with {run_id, projected_bytes, projected_runtime_s, projected_cost} Any authenticated caller in the paying workspace. mr-data run … --confirm re-states the request and authorizes the projection already on the record.
Existing legacy preview approval awaiting_sample_approval Existing pairs only; new full admissions never dispatch a separate preview. Any editor credential of the workspace, once the legacy preview has sealed a checked table and the build is authorized. POST /v4/runs/{run_id}:approve-full keeps its expected_version fence. Repair holds require a person in a signed-in browser. --confirm does not settle either hold.

mr-data run prints the projection and a confirm_command that approves it, then exits 2. An existing legacy held full run’s record names the sample as sample_run_id, and the sample names the full as full_run_id.

Refresh, and where the cadence comes from

Section titled “Refresh, and where the cadence comes from”

The first succeeded run makes a table live and dispatches no catch-up. After that:

  1. mr-data promote TABLE --cadence … --why … records a seed schedule and the reasoning, runs a catch-up backfill from the last successful run’s bookmark, and attaches the recurring schedule.
  2. Each refresh compares the sha256 of every source’s acquired bytes against the predecessor’s digest for that source. It reports one observation per source on success.
  3. The platform adjusts the applied schedule from those observations.
  4. A failed refresh never advances the live pointer, and the platform tells the workspace the cause.

A schedule runs only where the recipe can refresh. Every source has to declare a continuation: a window carrying a request, closed: true, a collection, or a recorded stream. The plan’s shape has to be materializable too: every source a recorded stream, or at least one collection-free request window, with closed sources allowed beside it. Anything else answers RESYNC_REQUIRED before acquisition, and the table keeps serving the version its last run sealed. Publish and keep it fresh carries the whole gate.

A table whose source is a daily snapshot has a cadence floor. The window partitions on the run’s own acquisition day, so a day nobody runs is a hole no later run can fill. The learned schedule may only tighten past the window’s granularity, never loosen.

An accepted schedule is not evidence a refresh ran. mr-data table TABLE_ID reports last_refresh, next_refresh_at and last_failure.

mr-data reschedule TABLE --cadence … --why … changes the schedule without starting a run or spending anything. --lock stops the loop adjusting it, and --unlock resumes. A second promote on a live table is an idempotent replay that ignores the new cadence, so use reschedule.

Both commands accept three cadence forms: a five-field cron expression, an interval, or the word source to let the platform choose. An interval reads every 6h or every 30m from 2026-09-03T12:20:00Z. Both commands require --why, unless you asked for source.

Next

Section titled “Next”