Fix a failed run

Start with failure_code, failure_detail and failed_stage, then fix the recipe or command.

Every failed run includes failure_code, failure_detail and failed_stage. You will find the same values on the run record, terminal event and command error. Start with those fields regardless of which command reported the failure.

mr-data status RUN_ID --json     # the triple, plus coverage and clamps
mr-data checks RUN_ID --json     # how each declared check came out
mr-data runs --json --limit 20   # the triple on every recent run

status and checks exit non-zero on a failure, so a script gating on a build cannot read one as a success.

Where a failure happened

failed_stage is one of five, and it is the first field to check. The worker keeps one cursor for the run’s current stage. It reads that cursor when an error occurs, so the field records where the run actually was.

Stage	What happens there
acquire	The worker reads the job document, specifies the declared checks, turns the document into an executable plan, then binds each declared source to a request and fetches it.
parse	The worker reads the fetched bytes as a table and writes them with their shape recorded.
transform	The worker restores a predecessor when the mode needs one, runs the statements over the fetched relations, then writes the result.
check	The worker evaluates each declared check over the table it wrote.
persist	The worker derives, uploads and registers the version’s artifacts.

The worker validates the declared checks before it fetches a byte. Kind, parameters, domains, duplicate identifiers and every column name against the table’s own declared columns follow from the document alone. A recipe naming a column that does not exist fails at acquire, not after the whole fetch.

The predecessor sits in transform, not acquire. A refresh computes a new version against a previous one. A run that cannot establish the previous one has not failed to reach a source.

WORKER_TERMINATED belongs to no stage of its own. The platform can reclaim a worker at any moment, so that refusal carries whatever stage the cursor held when the signal arrived.

Before changing anything

Studio caches a terminal run against (recipe, digest, mode, clamps). Re-running that exact cache key replays the stored result rather than making a new attempt, a stored failure included. It returns the same failure and consumes no worker. Changing a clamp forces a new attempt. --full --max-rows 50000 is a different cache key from a bare --full.

SANDBOX_MEMORY_BOUNDARY is not about the source, despite the address in the message. It reads source NAME was refused at the acquisition boundary: SANDBOX_MEMORY_BOUNDARY. The worker sandbox started without a delegated memory ceiling. The fault is intermittent and affects individual instances, so the same recipe may fail and then succeed minutes later without a source change. Re-run a recipe that has succeeded before. If it fails again with the same code, wait and retry on a different clamp. The source, URL, Reader and recipe document are not the cause.

Warming is not failure. Studio scales to zero, so the first cloud command of a session wakes it. Cloud answers 503 with a retry-after header, and the client waits and retries on its own, up to twice, up to thirty seconds a wait. A pause of about twenty seconds before the first receipt is that. A 503 with no retry-after is the different answer: Studio is not configured for this workspace, and the client stops immediately.

The refusals an author meets

Starting the run

Code	What it means	What to change
RESYNC_REQUIRED	409. The recipe cannot continue its sealed predecessor, so no refresh is admitted. The details name the blocker: a source with no declared continuation, a plan whose shape has no bounded materializer, or a missing predecessor. It happens before acquisition, and the table keeps serving its current version.	Read the blocker. Declare a window with a request on the source that has none. A closed-only plan, a plan carrying a collection, and a window beside a recorded stream each need an explicit table resync instead. mr-data recipe readiness classifies every source before you change anything.
RECIPE_DIGEST_MISMATCH	409. The registered recipe does not have the digest the command passed. Studio checks the digest here and never derives it.	Use the recipe_digest the registration receipt printed for the bytes you mean. The refusal’s details names the registered digest. After editing the document, register it again and take the new digest.
RUN_CLAMPS_REQUIRED	A sample run must state at least one clamp.	Add --max-rows or --max-source-bytes. The client also counts --window, but the current worker refuses a window clamp on sample with RUN_CLAMPS_INVALID, so use a row or byte ceiling. Two thousand rows a source is the number to start from before any measurement.
RUN_CLAMPS_INVALID	The clamp is outside what this deployed worker accepts, whatever the run schema allows. A --window on sample or full is one of these, because the range such a run covers comes from the recipe and not from a clamp.	Report the code. Do not retry the same run in another mode.
RUN_MODE_INVALID	The mode is outside what this deployed worker accepts. backfill is one of these on the current worker.	Report the code and choose a supported mode.
THIN_ARGUMENT_INVALID	The command line itself: a --status or --mode outside the closed sets, or --sources-from on a mode other than replay. The client rejects it before it sends anything.	Fix the argument.
CONFIRM_REQUIRED	409, arriving as THIN_STUDIO_CONFIRM_REQUIRED with http_status: 409. The run is over the projection threshold, Studio holds it awaiting_confirmation, and nothing has executed. The plan is flat, so this gate is about the size of the run and not about a charge. The error body includes run_id, projected_bytes, projected_runtime_s, projected_cost.	Run the same command again with --confirm on the end. The held answer prints it back as confirm_command. Any authenticated caller of the paying workspace may settle it. mr-data run exits 2 on this state rather than 0.
DAILY_SPEND_CEILING_EXCEEDED	409. The workspace has spent its daily budget. The error names what went against what ceiling, so a budget that ran out is distinguishable from one nobody raised.	Wait, or have an owner raise the ceiling. --confirm does not settle it.
awaiting_sample_approval	Not an error, a run state. The platform has atomically created a bounded preview and a linked immutable full run, and dispatched only the preview.	Read sample_run_id off the status and interrogate that preview. Then release the full run: any editor of the workspace may, once the preview has sealed a table, so an agent can on your word — or open the run’s page and release it in a signed-in browser. The release is fenced on the run’s expected_version. --confirm is not an alternative: it settles the confirmation gate, which is a different state.

Acquiring the sources

Code	What it means	What to change
ACQUISITION_FAILED	The fetch did not deliver. The detail names the source and what the transport said.	Read the detail. A 404 or a moved endpoint is a recipe edit. A 429 or a 5xx is the publisher, and a retry on a different clamp is the honest response.
ACQUISITION_FORMAT_UNMAPPED	The response’s media type is one the worker does not name, and it rejects rather than coercing it. The transform reads one input encoding, so a source whose bytes are some other format has to name the Reader family that opens them.	Pin a Reader through connector.parameters, or use a source that serves a format the direct path accepts. The error names the media type it was handed.
CLAMP_EXCEEDED	The run crossed a clamp. On full, refresh and backfill, crossing any clamp fails the run. max_source_bytes fails in every mode, including sample, because the worker cannot truncate bytes that have already crossed the wire.	Where the source really delivers more than the ceiling, do not lower the ceiling, which fetches no less data. Split the window across several sources and union them in a first transform step.
SANDBOX_MEMORY_LIMIT	The confined read ran out of memory on this source. It is the largest single source, not the total.	Narrow the projection where the source is wide. Split the window across several sources and union them in a first transform step where it is long.
READER_BUDGET	The source crossed a Reader budget. The refusal names the size and the ceiling. A fetch ceiling and a decoded-output ceiling are different numbers, and Limits carries both.	Pin a Reader where an unpinned json or parquet source hit 16 MiB. Narrow the projection where the decoded table was the half that did not fit.
SANDBOX_MEMORY_BOUNDARY	The worker sandbox, not the source.	Re-run a recipe that succeeded before. If that fails too, wait and retry on a different clamp.
CLAIM_KIND_UNKNOWN	rights_claim.claimed_basis is not one the worker recognises.	Use one of unknown, prohibited, permission_asserted, public_domain_asserted, contractual_license_asserted, terms_of_service_asserted.
RUN_CREDENTIAL_SCOPE_INVALID	The credential the source names is not one this run may redeem.	Check mr-data keys list and the secret_name spelling, or the connection grant on the dataset.
COLLECTION_DISCOVERY_FAILED	The collection listing failed and the run has nothing to continue from.	Read the refusal code inside the detail. A first run whose index is unreachable is a source problem, not an empty publisher.
COLLECTION_MERGE_INVALID	A contradiction between the run and its own predecessor: different columns, a ledger under a different header, or a discovery body that is not the declared shape.	The predecessor and this revision disagree. Register the corrected document and run full to start a fresh lineage.
WINDOW_ROWS_OUT_OF_RANGE	A publisher returned rows outside the window the run asked for. The worker refuses the response instead of clipping it because clipping would make a publisher that ignores the window look compliant.	The bound on window.request.start or .end is about the publisher’s parameter, not the engine’s. Settle it by comparing one windowed pull against a full pull over the same days, row for row.
SNAPSHOT_BEFORE_START_AT	The acquisition’s own receipt instant falls before window.start_at.	start_at is a floor for a request. Nothing moves an observation up to it, because that would stamp rows with a day nobody read them on.
SNAPSHOT_PARTITIONS_EXHAUSTED	The merge would exceed snapshot.max_partitions. Studio checks it before it uploads anything.	The table has outlived its declared ceiling. Start a new table. The old one stays readable.
SNAPSHOT_RELATION_EMPTY	A snapshot slice decoded to no rows at all, a fetch that returned nothing rather than a day the publisher had nothing to say on.	Writing it would erase the partition already written for that day. Fix the source or the Reader pin.

Declaring and running the checks

Code	What it means	What to change
DECLARED_CHECKS_INVALID	The checks array as a whole is not admissible.	Read the pointer in the detail.
DECLARED_CHECK_INVALID	One check’s shape is wrong for its kind. check X reads exactly one column and names 3 is the commonest.	One check per column. null_ceiling, value_range and continuity each take exactly one column. row_expectation takes none, and naming one refuses. Bounds are strings, not numbers: "min_value": "-50".
DECLARED_CHECK_DUPLICATE	Two checks share a check_id.	Rename one.
DECLARED_CHECK_COLUMN_UNKNOWN	A check names a column the table plan does not declare.	Fix the spelling, or declare the column. The worker decides this from the document alone, before any fetch.
DECLARED_CHECK_UNREADABLE	The worker could not evaluate the check over the table it wrote.	Read the detail. A table too large to scan inside the budget records the check as not_evaluated with a reason rather than as a pass.
CHECK_FAILED	A required check did not pass, and that is the run failing, with the check named. The check pass runs after the worker writes the table and before it registers any artifact, so a failed required check leaves nothing uploaded.	mr-data checks RUN_ID --json reports the triple in full. Either the data is genuinely not what the check says, or the check was wrong. A continuity check over a sample cut at a per-source row ceiling fails on seams that exist nowhere in the data. Raise the ceiling above the largest source, or make the check advisory where it is a report rather than a gate.

Transforming

Code	What it means	What to change
TRANSFORM_COLUMN_TYPE_MISMATCH	The statement returns an engine type the declared column does not allow. The run binds every source as all-text CSV, so each declared type is an explicit cast.	declared timestamp and the statement returns TIMESTAMP names the same word twice because the difference is the zone. Build it with to_timestamp(CAST(x AS BIGINT)), or CAST(concat(ts, ':00+00') AS TIMESTAMP WITH TIME ZONE). AT TIME ZONE is unavailable, because the engine does not load ICU. decimal takes DECIMAL(p,s) and never FLOAT or DOUBLE.
TRANSFORM_STATEMENT_NOT_SELECT	A step is not a SELECT. The worker refuses ATTACH, COPY, INSTALL, PRAGMA, anything that writes, and a step that reads a file or a URL.	Reach a source by naming its relation, not by opening its address again.
TRANSFORM_STATEMENT_NOT_SINGLE	A step carries more than one statement.	One statement per step. Add another step instead.
TRANSFORM_STEP_DUPLICATE	Two steps share a step_id.	Rename one.
TRANSFORM_STEP_SHADOWS_SOURCE	A step_id equals a source’s name. The worker rejects it rather than resolving it, because which of the two the SQL meant would depend on registration order.	Rename the step.
TRANSFORM_ENGINE_UNCONFINED	The statement reads something the plan does not fix: current_date, current_timestamp, localtime, now(), today(), random(). A later run re-derives the table and compares it byte for byte. A clock-dependent step may agree with itself all day, then fail on the next refresh.	Remove it. Qualify a column a publisher spelled like one of those: observations.current_date.
TRANSFORM_PLAN_INVALID	The plan as a whole is not admissible: engine is not duckdb_sql, or there are no steps.	engine: "none" is a registered value no run executes.
PREDECESSOR_UNAVAILABLE	A refresh could not establish the previous version it computes against.	The first run after adding a window must establish a complete predecessor relation before a refresh can merge a slice into it. Run full first.
PREDECESSOR_HOST_REFUSED	The predecessor could be named but not read.	Read the detail. This is infrastructure, not the document.
TABLE_UPSERT_UNCLUSTERED	A delta_upsert refresh’s rewrite set is over 64 parts, which is a full rewrite of a table that may be forty gibibytes.	The refusal names both fixes: declare partition_fields and refresh with partition_replace, or cluster the table on its identity key.

Persisting

Code	What it means	What to change
ARTIFACT_PERSIST_FAILED	The worker could not derive, upload or register an artifact. A run whose recipe declares sources cannot upload a receipt saying it fetched nothing, so an empty fetch record with sources declared lands here.	Read the detail. Where it names the empty record, the sources did not deliver, and the earlier stage’s detail says why.
TABLE_VERSION_UNCOMMITTABLE	The version could not be committed.	Read the detail.
RUN_ARTIFACT_SESSION_INVALID	The run cannot use the upload session it was given.	Infrastructure. Retry on a different clamp.
RUN_EXECUTION_FAILED	The run died in a way the worker could name but not attribute to the document.	Retry on a different clamp. If it repeats identically, it is not the recipe.

Deaths only Studio can observe

A run whose execution vanishes never reports a failure of its own, so Studio’s controller terminalizes it with a full triple. Its failed_stage is the last stage the run actually reported, and acquire only when it reported none.

Code	What it means	What to change
EXECUTION_LOST	No live producer attempt remains.	Nothing in the document. Start a new attempt by changing a clamp.
EXECUTION_LEASE_EXPIRED	Every live attempt’s lease ran out, or the run’s log was silent longer than the stale-after window.	The same. That sweep never touches a run awaiting_confirmation, because nobody had started work on it.
WORKER_TERMINATED	The platform reclaimed the worker. It carries whatever stage the cursor held.	The same.

Client refusals

These are the thin client’s own, raised before or around a call. They all begin THIN_.

Code	What it means	What to change
THIN_UNKNOWN_COMMAND	That name is not a command. mr-data --help is the whole list.	Do not retry it under a different install, and do not report its absence as a defect.
THIN_ARGUMENT_INVALID	An argument outside a closed set.	Fix the argument.
THIN_REQUEST_INVALID	The request the arguments compose is not one that can be sent: two spellings of one body, a repeated --format, a missing sub-command.	Fix the arguments.
THIN_RECIPE_DOCUMENT_UNREADABLE	The recipe path is not there, is not a plain file, or is over the size ceiling (1,048,576 bytes).	Check the path.
THIN_RECIPE_DOCUMENT_INVALID	Not strict JSON, not one JSON object, a fractional number, a missing required member, no sources, or a dataset.id that is not a lowercase canonical UUID. The sentence names the position.	Fix the document. No fractional number may appear anywhere in it.
THIN_AUTHENTICATION_FAILED	Cloud answered 401 and rejected the stored device credential.	mr-data login. Hosted commands need an mr_cli_ device credential, and the client refuses a read-only live data key here rather than failing three calls later.
THIN_SUBSCRIPTION_REQUIRED	Cloud answered 402: this workspace has no hosted execution access.	Billing, not the recipe.
THIN_RATE_LIMITED	Cloud answered 429 and asked this command to retry later.	Wait. Do not loop.
THIN_AUTHORIZATION_DENIED	401 or 403 on the call itself: the credential does not carry the authority this call needs. On a receipt read it means the credential may not see that record.	Switch to an authorized account, or request access.
THIN_NOT_FOUND	404: no such resource for that path. On receipt and checks it is the ordinary state of a run that has not produced that record yet.	Wait, or check the identifier.
THIN_STUDIO_UNAVAILABLE	Studio is not configured for this workspace, or stayed unavailable across every attempt.	Ask the workspace owner to enable hosted builds.
THIN_STUDIO_UNREACHABLE	Studio did not answer at all.	Transport. Retry.
THIN_STUDIO_REFUSED	Studio refused and named no code this client recognises. The HTTP status rides on the refusal.	Read the status.
THIN_STUDIO_<CODE>	Any Studio refusal code, folded under this prefix with Studio’s own sentence kept. THIN_STUDIO_CONFIRM_REQUIRED and THIN_STUDIO_NOTHING_TO_PROMOTE are the two an author meets.	Act on the inner code.
THIN_RESPONSE_INVALID	Studio answered something this client cannot read: a run with no identifier, a registration with no digest.	Report it.
THIN_RESPONSE_TOO_LARGE	The answer was over the client’s own byte ceiling.	Narrow the request.
THIN_NO_CHECK_RESULTS	The run ended before its declared checks ran, or Studio holds it as succeeded and its log carries no terminal record.	Read mr-data status for the state.
THIN_DOWNLOAD_DIGEST_MISMATCH	A fetched artifact’s bytes are not the digest Studio recorded.	Nothing is written. Report it, because this is a real integrity failure and not a retryable blip.
THIN_VERSION_PART_UNREACHABLE	A part of the version could not be fetched. A download refuses naming how many, rather than leaving a folder holding some of the table.	Retry, and report it if it persists.
THIN_DESTINATION_EXISTS	The output folder already holds that file. Nothing is ever overwritten.	Use a different --output.
THIN_CLOUD_STEP_UP_REQUIRED	Publishing a dataset publicly needs a verification only a signed-in browser can finish.	Open the dashboard address the refusal names and share it from that page.
THIN_CLOUD_PUBLIC_SHARING_DISABLED	The organization does not allow datasets to be shared publicly.	An owner decides this.
THIN_CLOUD_PUBLISHER_PROFILE_REQUIRED	The organization has no publisher name yet, and a public dataset is published under one.	Set it in the team’s settings.
THIN_COMMAND_HOSTED_PENDING	The command names a capability gap rather than doing anything: acquire-slices is the one.	Choose a supported route, or report the capability unavailable. Do not improvise a replacement.
THIN_COMMAND_IS_WORKER_INTERNAL	export-hosted-candidate is a backend operator step.	Not part of any build.

Two refusals from the table commands

Code	What it means	What to change
NOTHING_TO_PROMOTE	There is no succeeded run to make live yet.	Start one: mr-data run --recipe RECIPE_ID --digest HEX --full. mr-data checks tells you whether it passed.
TABLE_ARCHIVED	The table was retired, so nothing can make it live, and no run against it will change that.	Nothing built was deleted. Every run, artifact and version is still there and still queryable by its run. To publish this work, register a recipe naming a new table in the same dataset and promote that one.

The full list

Run states and errors carries every run state, every event type, and the complete refusal and worker vocabulary. Limits carries the ceilings.