# Fix a failed run

Source: https://mostlyright.md/docs/build/troubleshooting/

Start with `failure_code`, `failure_detail` and `failed_stage`, then fix the recipe or command.

Every failed run includes `failure_code`, `failure_detail` and `failed_stage`. You will find the same
values on the run record, terminal event and command error. Start with those fields regardless of
which command reported the failure.

```bash
mr-data status RUN_ID --json     # the triple, plus coverage and clamps
mr-data checks RUN_ID --json     # how each declared check came out
mr-data runs --json --limit 20   # the triple on every recent run
```

`status` and `checks` exit non-zero on a failure, so a script gating on a build cannot read one as a success.

## Where a failure happened

`failed_stage` is one of five, and it is the first field to check. The worker keeps one cursor for the run's current stage. It reads that cursor when an error occurs, so the field records where the run actually was.

| Stage | What happens there |
| --- | --- |
| `acquire` | The worker reads the job document, specifies the declared checks, turns the document into an executable plan, then binds each declared source to a request and fetches it. |
| `parse` | The worker reads the fetched bytes as a table and writes them with their shape recorded. |
| `transform` | The worker restores a predecessor when the mode needs one, runs the statements over the fetched relations, then writes the result. |
| `check` | The worker evaluates each declared check over the table it wrote. |
| `persist` | The worker derives, uploads and registers the version's artifacts. |

The worker validates the declared checks before it fetches a byte. Kind, parameters, domains, duplicate identifiers and every column name against the table's own declared columns follow from the document alone. A recipe naming a column that does not exist fails at `acquire`, not after the whole fetch.

The predecessor sits in `transform`, not `acquire`. A refresh computes a new version against a previous one. A run that cannot establish the previous one has not failed to reach a source.

`WORKER_TERMINATED` belongs to no stage of its own. The platform can reclaim a worker at any moment, so that refusal carries whatever stage the cursor held when the signal arrived.

## Before changing anything

Studio caches a terminal run against `(recipe, digest, mode, clamps)`. Re-running that exact cache key replays the stored result rather than making a new attempt, a stored failure included. It returns the same failure and consumes no worker. Changing a clamp forces a new attempt. `--full --max-rows 50000` is a different cache key from a bare `--full`.

`SANDBOX_MEMORY_BOUNDARY` is not about the source, despite the address in the message. It reads `source NAME was refused at the acquisition boundary: SANDBOX_MEMORY_BOUNDARY`. The worker sandbox started without a delegated memory ceiling. The fault is intermittent and affects individual instances, so the same recipe may fail and then succeed minutes later without a source change. Re-run a recipe that has succeeded before. If it fails again with the same code, wait and retry on a different clamp. The source, URL, Reader and recipe document are not the cause.

Warming is not failure. Studio scales to zero, so the first cloud command of a session wakes it. Cloud answers 503 with a `retry-after` header, and the client waits and retries on its own, up to twice, up to thirty seconds a wait. A pause of about twenty seconds before the first receipt is that. A 503 with no `retry-after` is the different answer: Studio is not configured for this workspace, and the client stops immediately.

## The refusals an author meets

### Starting the run

| Code | What it means | What to change |
| --- | --- | --- |
| `RESYNC_REQUIRED` | 409. The recipe cannot continue its sealed predecessor, so no refresh is admitted. The `details` name the blocker: a source with no declared continuation, a plan whose shape has no bounded materializer, or a missing predecessor. It happens before acquisition, and the table keeps serving its current version. | Read the blocker. `mr-data recipe explain RECIPE.json` lists every reason, each with a rule code and Studio's suggested repair, before you change anything. Declare a `window` with a `request` on the source that has none. A closed-only plan, a page collection, a snapshot source, and a window beside a recorded stream each need an explicit table resync, or a declared `rebuild` that reads the table whole on an interval ([Publish and keep it fresh](/docs/build/publish-and-refresh/#rebuild-a-table-that-cannot-refresh)). |
| `RECIPE_DIGEST_MISMATCH` | 409. The registered recipe does not have the digest the command passed. Studio checks the digest here and never derives it. | Use the `recipe_digest` the registration receipt printed for the bytes you mean. The refusal's `details` names the registered digest. After editing the document, register it again and take the new digest. |
| `RUN_CLAMPS_REQUIRED` | A `sample` run must state at least one clamp. | Add `--max-rows` or `--max-source-bytes`. The client also counts `--window`, but the current worker refuses a window clamp on `sample` with `RUN_CLAMPS_INVALID`, so use a row or byte ceiling. Two thousand rows a source is the number to start from before any measurement. |
| `RUN_CLAMPS_INVALID` | The clamp is outside what this deployed worker accepts, whatever the run schema allows. A `--window` on `sample` or `full` is one of these, because the range such a run covers comes from the recipe and not from a clamp. | Report the code. Do not retry the same run in another mode. |
| `RUN_MODE_INVALID` | The mode is outside what this deployed worker accepts. `backfill` is one of these on the current worker. | Report the code and choose a supported mode. |
| `THIN_ARGUMENT_INVALID` | The command line itself: a `--status` or `--mode` outside the closed sets, or `--sources-from` on a mode other than `replay`. The client rejects it before it sends anything. | Fix the argument. |
| `CONFIRM_REQUIRED` | 409, arriving as `THIN_STUDIO_CONFIRM_REQUIRED` with `http_status: 409`. The run is over the projection threshold, Studio holds it `awaiting_confirmation`, and nothing has executed. The plan is flat, so this gate is about the size of the run and not about a charge. The error body includes `run_id`, `projected_bytes`, `projected_runtime_s`, `projected_cost`. | Run the same command again with `--confirm` on the end. The held answer prints it back as `confirm_command`. Any authenticated caller of the paying workspace may settle it. `mr-data run` exits `2` on this state rather than `0`. |
| `DAILY_SPEND_CEILING_EXCEEDED` | 409. The workspace has spent its daily budget. The error names what went against what ceiling, so a budget that ran out is distinguishable from one nobody raised. | Wait, or have an owner raise the ceiling. `--confirm` does not settle it. |
| `awaiting_sample_approval` | Not an error, a run state. The platform has atomically created a bounded preview and a linked immutable full run, and dispatched only the preview. | Read `sample_run_id` off the status and interrogate that preview. Then release the full run: any editor of the workspace may, once the preview has sealed a table, so an agent can on your word — or open the run's page and release it in a signed-in browser. The release is fenced on the run's `expected_version`. `--confirm` is not an alternative: it settles the confirmation gate, which is a different state. |

### Acquiring the sources

| Code | What it means | What to change |
| --- | --- | --- |
| `ACQUISITION_FAILED` | The fetch did not deliver. The detail names the source and what the transport said. | Read the detail. A 404 or a moved endpoint is a recipe edit. A 429 or a 5xx is the publisher, and a retry on a different clamp is the honest response. |
| `ACQUISITION_FORMAT_UNMAPPED` | The response's media type is one the worker does not name, and it rejects rather than coercing it. The transform reads one input encoding, so a source whose bytes are some other format has to name the Reader family that opens them. | Pin a Reader through `connector.parameters`, or use a source that serves a format the direct path accepts. The error names the media type it was handed. |
| `CLAMP_EXCEEDED` | The run crossed a clamp. On `full`, `refresh` and `backfill`, crossing any clamp fails the run. `max_source_bytes` fails in every mode, including sample, because the worker cannot truncate bytes that have already crossed the wire. | Where the source really delivers more than the ceiling, do not lower the ceiling, which fetches no less data. Split the window across several sources and union them in a first transform step. |
| `SANDBOX_MEMORY_LIMIT` | The confined read ran out of memory on this source. It is the largest single source, not the total. | Narrow the projection where the source is wide. Split the window across several sources and union them in a first transform step where it is long. |
| `READER_BUDGET` | The source crossed a Reader budget. The refusal names the size and the ceiling. A fetch ceiling and a decoded-output ceiling are different numbers, and [Limits](/docs/reference/limits/) carries both. | Pin a Reader where an unpinned `json` or `parquet` source hit 16 MiB. Narrow the projection where the decoded table was the half that did not fit. |
| `SANDBOX_MEMORY_BOUNDARY` | The worker sandbox, not the source. | Re-run a recipe that succeeded before. If that fails too, wait and retry on a different clamp. |
| `CLAIM_KIND_UNKNOWN` | `rights_claim.claimed_basis` is not one the worker recognises. | Use one of `unknown`, `prohibited`, `permission_asserted`, `public_domain_asserted`, `contractual_license_asserted`, `terms_of_service_asserted`. |
| `RUN_CREDENTIAL_SCOPE_INVALID` | The credential the source names is not one this run may redeem. | Check `mr-data keys list` and the `secret_name` spelling, or the connection grant on the dataset. |
| `COLLECTION_DISCOVERY_FAILED` | The collection listing failed and the run has nothing to continue from. | Read the refusal code inside the detail. A first run whose index is unreachable is a source problem, not an empty publisher. |
| `COLLECTION_MERGE_INVALID` | A contradiction between the run and its own predecessor: different columns, a ledger under a different header, or a discovery body that is not the declared shape. | The predecessor and this revision disagree. Register the corrected document and run `full` to start a fresh lineage. |
| `WINDOW_ROWS_OUT_OF_RANGE` | A publisher returned rows outside the window the run asked for. The worker refuses the response instead of clipping it because clipping would make a publisher that ignores the window look compliant. | The `bound` on `window.request.start` or `.end` is about the publisher's parameter, not the engine's. Settle it by comparing one windowed pull against a full pull over the same days, row for row. |
| `SNAPSHOT_BEFORE_START_AT` | The acquisition's own receipt instant falls before `window.start_at`. | `start_at` is a floor for a request. Nothing moves an observation up to it, because that would stamp rows with a day nobody read them on. |
| `SNAPSHOT_PARTITIONS_EXHAUSTED` | The merge would exceed `snapshot.max_partitions`. Studio checks it before it uploads anything. | The table has outlived its declared ceiling. Start a new table. The old one stays readable. |
| `SNAPSHOT_RELATION_EMPTY` | A snapshot slice decoded to no rows at all, a fetch that returned nothing rather than a day the publisher had nothing to say on. | Writing it would erase the partition already written for that day. Fix the source or the Reader pin. |

### Declaring and running the checks

| Code | What it means | What to change |
| --- | --- | --- |
| `DECLARED_CHECKS_INVALID` | The `checks` array as a whole is not admissible. | Read the pointer in the detail. |
| `DECLARED_CHECK_INVALID` | One check's shape is wrong for its kind. `check X reads exactly one column and names 3` is the commonest. | One check per column. `null_ceiling`, `value_range` and `continuity` each take exactly one column. `row_expectation` takes none, and naming one refuses. Bounds are strings, not numbers: `"min_value": "-50"`. |
| `DECLARED_CHECK_DUPLICATE` | Two checks share a `check_id`. | Rename one. |
| `DECLARED_CHECK_COLUMN_UNKNOWN` | A check names a column the table plan does not declare. | Fix the spelling, or declare the column. The worker decides this from the document alone, before any fetch. |
| `DECLARED_CHECK_UNREADABLE` | The worker could not evaluate the check over the table it wrote. | Read the detail. A table too large to scan inside the budget records the check as `not_evaluated` with a reason rather than as a pass. |
| `CHECK_FAILED` | A required check did not pass, and that is the run failing, with the check named. The check pass runs after the worker writes the table and before it registers any artifact, so a failed required check leaves nothing uploaded. | `mr-data checks RUN_ID --json` reports the triple in full. Either the data is genuinely not what the check says, or the check was wrong. A continuity check over a sample cut at a per-source row ceiling fails on seams that exist nowhere in the data. Raise the ceiling above the largest source, or make the check `advisory` where it is a report rather than a gate. |

### Transforming

| Code | What it means | What to change |
| --- | --- | --- |
| `TRANSFORM_COLUMN_TYPE_MISMATCH` | The statement returns an engine type the declared column does not allow. The run binds every source as all-text CSV, so each declared type is an explicit cast. | `declared timestamp and the statement returns TIMESTAMP` names the same word twice because the difference is the zone. Build it with `to_timestamp(CAST(x AS BIGINT))`, or `CAST(concat(ts, ':00+00') AS TIMESTAMP WITH TIME ZONE)`. `AT TIME ZONE` is unavailable, because the engine does not load ICU. `decimal` takes `DECIMAL(p,s)` and never `FLOAT` or `DOUBLE`. |
| `TRANSFORM_STATEMENT_NOT_SELECT` | A step is not a `SELECT`. The worker refuses `ATTACH`, `COPY`, `INSTALL`, `PRAGMA`, anything that writes, and a step that reads a file or a URL. | Reach a source by naming its relation, not by opening its address again. |
| `TRANSFORM_STATEMENT_NOT_SINGLE` | A step carries more than one statement. | One statement per step. Add another step instead. |
| `TRANSFORM_STEP_DUPLICATE` | Two steps share a `step_id`. | Rename one. |
| `TRANSFORM_STEP_SHADOWS_SOURCE` | A `step_id` equals a source's `name`. The worker rejects it rather than resolving it, because which of the two the SQL meant would depend on registration order. | Rename the step. |
| `TRANSFORM_ENGINE_UNCONFINED` | The statement reads something the plan does not fix: `current_date`, `current_timestamp`, `localtime`, `now()`, `today()`, `random()`. A later run re-derives the table and compares it byte for byte. A clock-dependent step may agree with itself all day, then fail on the next refresh. | Remove it. Qualify a column a publisher spelled like one of those: `observations.current_date`. |
| `TRANSFORM_PLAN_INVALID` | The plan as a whole is not admissible: `engine` is not `duckdb_sql`, or there are no steps. | `engine: "none"` is a registered value no run executes. |
| `PREDECESSOR_UNAVAILABLE` | A refresh could not establish the previous version it computes against. | The first run after adding a window must establish a complete predecessor relation before a refresh can merge a slice into it. Run `full` first. |
| `PREDECESSOR_HOST_REFUSED` | The predecessor could be named but not read. | Read the detail. This is infrastructure, not the document. |
| `TABLE_UPSERT_UNCLUSTERED` | A `delta_upsert` refresh's rewrite set is over 64 parts, which is a full rewrite of a table that may be forty gibibytes. | The refusal names both fixes: declare `partition_fields` and refresh with `partition_replace`, or cluster the table on its identity key. |

### Persisting

| Code | What it means | What to change |
| --- | --- | --- |
| `ARTIFACT_PERSIST_FAILED` | The worker could not derive, upload or register an artifact. A run whose recipe declares sources cannot upload a receipt saying it fetched nothing, so an empty fetch record with sources declared lands here. | Read the detail. Where it names the empty record, the sources did not deliver, and the earlier stage's detail says why. |
| `TABLE_VERSION_UNCOMMITTABLE` | The version could not be committed. | Read the detail. |
| `RUN_ARTIFACT_SESSION_INVALID` | The run cannot use the upload session it was given. | Infrastructure. Retry on a different clamp. |
| `RUN_EXECUTION_FAILED` | The run died in a way the worker could name but not attribute to the document. | Retry on a different clamp. If it repeats identically, it is not the recipe. |

### Deaths only Studio can observe

A run whose execution vanishes never reports a failure of its own, so Studio's controller terminalizes it with a full triple. Its `failed_stage` is the last stage the run actually reported, and `acquire` only when it reported none.

`mr-data status RUN_ID --diagnose --json` says how each attempt of the run ended. A cause it cannot read from Studio's own record, such as a revision roll or memory pressure, is marked as an inference. `mr-data run --retry RUN_ID` resumes a failed run whose triple carries `room_fault: true`, and a per-row source sends only the requests still owed. A declared `rebuild` that dies this way is retried on its own, up to three times.

| Code | What it means | What to change |
| --- | --- | --- |
| `EXECUTION_LOST` | No live producer attempt remains. | Nothing in the document. Resume it with `mr-data run --retry RUN_ID`, or start a new attempt by changing a clamp. |
| `EXECUTION_LEASE_EXPIRED` | Every live attempt's lease ran out, or the run's log was silent longer than the stale-after window. | The same. That sweep never touches a run `awaiting_confirmation`, because nobody had started work on it. |
| `WORKER_TERMINATED` | The platform reclaimed the worker. It carries whatever stage the cursor held. | The same. |

## Needs server 4.14

Check first: run `mr-data recipe explain RECIPE.json --json` and read `studio_capabilities`. The
refusals below come only from features whose capability is true. When a capability is false, the
feature is not available on this server yet, and a snapshot table keeps the one day its last full
run read.

| Code | What it means | What to change |
| --- | --- | --- |
| `ROW_FILTER_INVALID` | A per-row collection's `collection.discovery.row_filter` names a column its upstream does not have, returns something other than a boolean, or calls a refused function such as `now()` or `random()`. The worker binds it at run start, before any request of the run. | Read the upstream's columns in the detail, and fix the expression. |
| `SNAPSHOT_HISTORY_EXCEEDS_SOURCE_CEILING` | A snapshot refresh's merged history would pass `limits.max_source_bytes`. It fails before it seals, and the live version and its history stay readable. | The history ceiling is fixed at registration, and raising it discards the history. Register a revision with a larger ceiling only when losing the held days is acceptable. |
| `SNAPSHOT_HISTORY_WOULD_BE_DISCARDED` | 409. The table holds snapshot history, and a `full` or `sample` would replace every held day with one. It names each snapshot source and the run that holds the history. Nothing started. | Refresh instead. To replace the history on purpose, add `--discard-snapshot-history` to `mr-data run --full` or `mr-data table resync`. The discarded days cannot be fetched again. |

## Client refusals

These are the thin client's own, raised before or around a call. They all begin `THIN_`.

| Code | What it means | What to change |
| --- | --- | --- |
| `THIN_UNKNOWN_COMMAND` | That name is not a command. `mr-data --help` is the whole list. | Do not retry it under a different install, and do not report its absence as a defect. |
| `THIN_ARGUMENT_INVALID` | An argument outside a closed set. | Fix the argument. |
| `THIN_REQUEST_INVALID` | The request the arguments compose is not one that can be sent: two spellings of one body, a repeated `--format`, a missing sub-command. | Fix the arguments. |
| `THIN_RECIPE_DOCUMENT_UNREADABLE` | The recipe path is not there, is not a plain file, or is over the size ceiling (1,048,576 bytes). | Check the path. |
| `THIN_RECIPE_DOCUMENT_INVALID` | Not strict JSON, not one JSON object, a fractional number, a missing required member, no sources, or a `dataset.id` that is not a lowercase canonical UUID. The sentence names the position. | Fix the document. No fractional number may appear anywhere in it. |
| `THIN_AUTHENTICATION_FAILED` | Cloud answered 401 and rejected the stored device credential. | `mr-data login`. Hosted commands need an `mr_cli_` device credential, and the client refuses a read-only live data key here rather than failing three calls later. |
| `THIN_SUBSCRIPTION_REQUIRED` | Cloud answered 402: this workspace has no hosted execution access. Not raised while usage is metered: building is then free. | Billing, not the recipe. |
| `THIN_RATE_LIMITED` | Cloud answered 429 and asked this command to retry later. | Wait. Do not loop. |
| `THIN_AUTHORIZATION_DENIED` | 401 or 403 on the call itself: the credential does not carry the authority this call needs. On a receipt read it means the credential may not see that record. | Switch to an authorized account, or request access. |
| `THIN_NOT_FOUND` | 404: no such resource for that path. On `receipt` and `checks` it is the ordinary state of a run that has not produced that record yet. | Wait, or check the identifier. |
| `THIN_STUDIO_UNAVAILABLE` | Studio is not configured for this workspace, or stayed unavailable across every attempt. | Ask the workspace owner to enable hosted builds. |
| `THIN_STUDIO_UNREACHABLE` | Studio did not answer at all. | Transport. Retry. |
| `THIN_STUDIO_REFUSED` | Studio refused and named no code this client recognises. The HTTP status rides on the refusal. | Read the status. |
| `THIN_STUDIO_<CODE>` | Any Studio refusal code, folded under this prefix with Studio's own sentence kept. `THIN_STUDIO_CONFIRM_REQUIRED` and `THIN_STUDIO_NOTHING_TO_PROMOTE` are the two an author meets. | Act on the inner code. |
| `THIN_RESPONSE_INVALID` | Studio answered something this client cannot read: a run with no identifier, a registration with no digest. | Report it. |
| `THIN_RESPONSE_TOO_LARGE` | The answer was over the client's own byte ceiling. | Narrow the request. |
| `THIN_NO_CHECK_RESULTS` | The run ended before its declared checks ran, or Studio holds it as succeeded and its log carries no terminal record. | Read `mr-data status` for the state. |
| `THIN_DOWNLOAD_DIGEST_MISMATCH` | A fetched artifact's bytes are not the digest Studio recorded. | Nothing is written. Report it, because this is a real integrity failure and not a retryable blip. |
| `THIN_VERSION_PART_UNREACHABLE` | A part of the version could not be fetched. A download refuses naming how many, rather than leaving a folder holding some of the table. | Retry, and report it if it persists. |
| `THIN_DESTINATION_EXISTS` | The output folder already holds that file. Nothing is ever overwritten. | Use a different `--output`. |
| `THIN_CLOUD_STEP_UP_REQUIRED` | Publishing a dataset publicly needs a verification only a signed-in browser can finish. | Open the dashboard address the refusal names and share it from that page. |
| `THIN_CLOUD_PUBLIC_SHARING_DISABLED` | The organization does not allow datasets to be shared publicly. | An owner decides this. |
| `THIN_CLOUD_PUBLISHER_PROFILE_REQUIRED` | The organization has no publisher name yet, and a public dataset is published under one. | Set it in the team's settings. |
| `THIN_GRAMMAR_NOT_SERVED` | The recipe or the command states a member or flag this Studio does not serve yet, by its own refresh explanation, or this Studio predates that route. Nothing was registered or started. | Wait for the Studio deploy, or remove the member. |
| `THIN_DATASET_SLUG_UNRESOLVED` | `mr-data dataset show` was given a slug, an address or an identifier that names no dataset this workspace can read and no public dataset. | `mr-data dataset list` names every dataset the workspace holds, by its identifier. |
| `THIN_COMMAND_HOSTED_PENDING` | The command names a capability gap rather than doing anything: `acquire-slices` is the one. | Choose a supported route, or report the capability unavailable. Do not improvise a replacement. |
| `THIN_COMMAND_IS_WORKER_INTERNAL` | `export-hosted-candidate` is a backend operator step. | Not part of any build. |

## Two refusals from the table commands

| Code | What it means | What to change |
| --- | --- | --- |
| `NOTHING_TO_PROMOTE` | There is no succeeded run to make live yet. | Start one: `mr-data run --recipe RECIPE_ID --digest HEX --full`. `mr-data checks` tells you whether it passed. |
| `TABLE_ARCHIVED` | The table was retired, so nothing can make it live, and no run against it will change that. | Nothing built was deleted. Every run, artifact and version is still there and still queryable by its run. To publish this work, register a recipe naming a new table in the same dataset and promote that one. |

## The full list

[Run states and errors](/docs/reference/run-states-and-errors/) carries every run state, every event type, and the complete refusal and worker vocabulary. [Limits](/docs/reference/limits/) carries the ceilings.