Make builds efficient

Decide what each source can reuse before the first build. Changing a recipe later changes its digest and can require a full resync. For paid sources, that can repeat provider charges.

Explain, forecast, run, diagnose

mr-data recipe explain recipe.json --json
mr-data recipe recipe.json --json
mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json
mr-data watch RUN_ID --json
mr-data status RUN_ID --diagnose --json

Replace the IDs with returned values. Run each step after reading the previous result. explain exits 2 when a finding blocks refresh; a snapshot-only first build can still be intentional, but it does not promise scheduled updates. forecast exits 3 when projected work exceeds quota or one attempt’s runtime. Resize before starting the build.

Read studio_capabilities: only literal true means a feature is available. An older deployment can return a local explanation or a forecast based on declared ceilings. source: local_mirror, basis: declared, basis: unknown, and null expected values are limitations, not measured performance. Report a disagreement between client and server findings instead of choosing the answer you prefer.

Keep work that has not changed

Feature	When it helps	Constraint
table.partition	A table has a non-null date-like or low-cardinality column and no window already partitions it.	Keeps unchanged partition bytes on later full builds. It does not make a table eligible for refresh. A table can hold at most 4,096 parts; use monthly rather than daily partitions for a long history.
request.batch	A per-row GET API accepts several IDs in one query parameter.	Up to 100 members per call; the Reader must expose the key used to separate responses. POST bodies and platform credentials do not support this batch shape.
Source reference	A bulk dump is joined to a windowed source.	Reuses sealed bytes between checks; a changed dump recomputes the table. It cannot also be closed, windowed, a collection or a stream.
A request window	The publisher accepts a bounded date range.	State lookback, start and merge identity from the source’s actual behavior.
closed: true	A source is immutable.	Do not use it to label a mutable snapshot as incremental.

fast_features reports table.partition, batch, reference, resync_only_sources, pace, concurrency, parallel_pagination and ranged_parquet. Each missed feature includes a suggested fix. Check whether its assumptions fit the publisher before applying it. For an existing live table, obtain authorization for any changed semantics or paid resync.

Fetch at the publisher’s pace

State pages.min_interval_milliseconds from the publisher’s documented request limit. When none is documented, start with 1000 ms and record the assumption. Registration refuses a gap below 16 ms; that lower bound is not a recommended rate.

Leave pages.concurrency out so the worker can size its width. Keep an explicit value only when the publisher limits concurrent connections or a run refusal requires it. The gap can apply per egress address; a per-key or per-account publisher limit therefore needs care when requests are spread across addresses.

For a keyed JSON API whose first page supplies a total page count, inspect the counted parallel-pagination support in authenticated.https.api_key@2.1.0. For public Parquet with byte-range support, use the parquet.tabular@1.1.0 Reader and select the columns you need. Confirm deployed support with recipe explain and normalize Reader options with reader normalize before registration.

Compare the forecast with the run

On a running collection, status and watch expose source rates and remaining-time estimates. Quote the reported numbers; replayed answers can make the early rate look faster than the actual request rate.

Read the receipt’s per-source bytes, rows, window and truncation. If a supported one-window refresh seals no receipt, read status --json coverage instead. Unknown coverage is not proof of completeness.

Before scheduling, run:

mr-data recipe readiness --table TABLE_ID --current-only --json

Read the current revision’s blockers. A held schedule needs a supported continuation, a declared rebuild, or an intentional explicit resync. Publish and refresh explains those choices.