# Documents and PDFs Source: https://mostlyright.md/docs/recipes/document-extraction/ Turn one selected table inside a captured PDF into a dataset through a certified projection. This document registers, and `mr-data` cannot produce the eight digests it needs. The capture store, the certified decoder and the operator command that performs capture, table selection and `handoff_projection` all run inside the backend worker. This client exposes no `document` command and no local stand-in. The eight digests here have to come from an operator who ran that pipeline. Everything else here is exact, and the recipe validates against the v4 schema as printed. ## The question What figures does the published document state, row by row, and which retained projection does each row come from? ## Why a PDF is handled differently A Reader opens a format with an explicit row contract and turns it into exact rows. A PDF carries no such contract. Its apparent rows and tables are visual hypotheses. Documents go through a separate capture pipeline. It records evidence first, then accepts one explicitly selected table projection into ordinary dataset processing. The pipeline's state words are claims, not progress labels: | State | What is true | What is not implied | | --- | --- | --- | | captured | the exact raw bytes and a content-addressed receipt exist | the file was readable or safe to decode | | text extracted | native text and spans can be reproduced from the source digest, page selection, engine name and version, and configuration digest | the text is a correct reading of the visual layout | | OCR extracted | a pinned OCR engine transcribed retained rendered images, with boxes and confidence | the transcription is correct | | table candidate | the pipeline retained bounded cells, geometry, hypotheses, ambiguity and page provenance | it is a dataset or an approved reading of a table | | value evidence | the pipeline retained exact spans inside one selected region | it is a number, a normalized value or a row | | certified table projection | one retained candidate plus an exact projection policy produced CSV or Parquet | any unselected surrounding content became data | Only a certified table projection can become a dataset. It enters through `handoff_projection` and the `document.projection@1.0.0` adapter. ## The operator's steps, before the recipe The pipeline accepts `application/pdf`, `image/png`, `image/jpeg` and `image/tiff`. It refuses office containers. Certified execution runs on Linux only, inside the worker sandbox, with no network authority. 1. Capture the original bytes once with their declared media type, and keep the returned receipt digest, which addresses the record of what was captured. A capture proves raw-byte retention, not extraction. 2. Run `inspect_pdf` before choosing a decoding path, and read the retained admission and text-layer evidence. 3. Run `extract_native` for the exact pages the claim needs. The spans carry boxes and reading order, so the selection stays inspectable. 4. Run `ocr_pages` for pages without usable native text, under an explicit policy. The policy decides which pages fall back, not a model guess, and native and OCR spans stay separate artifacts. 5. Run `table_candidates` against one page, one bounding box and one span kind, never a mix of native and OCR. 6. Run `project_table` with the candidate's exact digest, explicit `header_rows` and non-empty `body_rows`, an output format, and explicit ambiguity and merged-cell policies. No default selects the rows that look tabular. 7. Run `handoff_projection` to get the closed set of digests the recipe carries. An exported CSV acquired again is ordinary `user_file` input, and equal bytes do not restore document provenance. Footnote-bearing table evidence stays readable. This version does not project it, because no policy can guess its selection or placement. ## The recipe ```json title="recipe.json" { "dataset": { "name": "Certified document figures, one projection" }, "question": { "text": "What figures does the published document state, row by row, and which retained projection does each row come from?" }, "table": { "name": "reported_figures", "description": "One row per body row of the projected table.", "grain": [ "row_ordinal" ], "columns": [ { "name": "row_ordinal", "type": "integer", "nullable": false, "description": "The one-based position of this row inside the projected table.", "presentation": { "chart": "none" } }, { "name": "line_item", "type": "string", "nullable": false, "description": "The label the document prints in the first column of the table.", "presentation": { "chart": "top_values" } }, { "name": "period", "type": "string", "nullable": true, "description": "The period heading the figure sits under, exactly as the document prints it." }, { "name": "reported_value", "type": "string", "nullable": true, "description": "The figure as the document prints it, including any thousands separator or sign the publisher used.", "presentation": { "chart": "none" } } ] }, "sources": [ { "name": "annual_report_table", "description": "One table an operator selected out of a captured annual report and projected to CSV.", "source_class": "user_file", "data_classification": "internal", "locator": { "kind": "document_capture", "display_locator": "document-capture://receipt/sha256:4444444444444444444444444444444444444444444444444444444444444444" }, "rights_claim": { "claimed_basis": "permission_asserted", "claim_evidence_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555", "claim_note": "The document was supplied to this workspace by its publisher for internal analysis." }, "connector": { "adapter_id": "document.projection@1.0.0", "credential_mode": "none", "parameters": [ { "name": "document.receipt_digest", "value": "sha256:4444444444444444444444444444444444444444444444444444444444444444" }, { "name": "document.raw_document_digest", "value": "sha256:6666666666666666666666666666666666666666666666666666666666666666" }, { "name": "document.projection_manifest_digest", "value": "sha256:7777777777777777777777777777777777777777777777777777777777777777" }, { "name": "document.projection_content_digest", "value": "sha256:8888888888888888888888888888888888888888888888888888888888888888" }, { "name": "document.candidate_manifest_digest", "value": "sha256:9999999999999999999999999999999999999999999999999999999999999999" }, { "name": "document.projection_policy_digest", "value": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }, { "name": "document.media_type", "value": "text/csv" }, { "name": "document.source_version", "value": "mostlyright-document-projection-source.v1" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 100000, "max_requests": 1 } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "reported_figures", "sql": "select cast(row_ordinal as integer) as row_ordinal, line_item, period, reported_value from annual_report_table", "description": "Carry the projected columns through under the names the projection policy gave them. The projection is sealed as CSV, so every column reaches the transform as text." } ] }, "checks": [ { "check_id": "one_row_per_projected_row", "kind": "key_uniqueness", "enforcement": "required", "columns": [ "row_ordinal" ] }, { "check_id": "line_item_present", "kind": "null_ceiling", "enforcement": "required", "columns": [ "line_item" ], "max_null_ppm": 0 }, { "check_id": "rows_present", "kind": "row_expectation", "enforcement": "required", "min_rows": 1 } ], "units": [], "timezone": "UTC" } ``` ## Part by part ### The source `locator.kind` is `document_capture` and `adapter_id` is `document.projection@1.0.0`. The receipt records that adapter name as the route the source arrived through. The run fetches nothing. It redeems the projection from the operator's capture store. `credential_mode` is `none`. The source names no `origin`, because there is no host. ### The eight reserved parameters `declared_source` is closed and carries no member for a capture address, so the eight digests ride on `connector.parameters`. That array already folds into the source authority digest, so an edit in transit moves the recipe digest. | Parameter | What it addresses | | --- | --- | | `document.receipt_digest` | the retained receipt the projection belongs to | | `document.raw_document_digest` | the original captured bytes | | `document.projection_manifest_digest` | the projection artifact, as bytes plus source, engine, configuration, pages, region, bounds and parents | | `document.projection_content_digest` | the projection's bytes alone | | `document.candidate_manifest_digest` | the table candidate the projection was made from | | `document.projection_policy_digest` | the exact header, body, ambiguity and merged-cell policy | | `document.media_type` | `text/csv` or `application/vnd.apache.parquet` | | `document.source_version` | `mostlyright-document-projection-source.v1` | All eight or none. A source that names half the set cannot be replayed from its own record, so a partial set fails with `RUN_RECIPE_INVALID` naming the missing members. The engine also refuses a parameter inside the reserved `document.` namespace that matches none of the eight, instead of ignoring it. A typo would otherwise pass as a digest the worker never read. The adapter reopens the named receipt from its configured capture store and re-verifies all eight digests before parsing the retained CSV inside the worker sandbox. Deployment configuration sets the store's location. No recipe and no run document can select it. ### Transform The projection is stored as CSV, so every column reaches the SQL as text and each declared type needs an explicit cast. The column names come from the projection policy, chosen at `project_table` time rather than here. ### Columns `reported_value` is declared `string`, not `decimal`. The pipeline keeps the source text as printed. It does not parse `1,234`, remove a percent sign, infer a unit, or turn prose into a number. Cast to a number in the transform, where the cast is visible and can fail, and keep the printed text beside it. The receipt cites what the document said. ### Checks `key_uniqueness` on `row_ordinal` and a `null_ceiling` of zero on `line_item` catch a projection whose rows did not come out as the policy expected. ## What the receipt records Every derived artifact records the source content digest and the primary engine. It records the ordered decoder chain with binary and OCR model digests, the canonical configuration digest, and the sandbox policy and bounds digest. It records the page and region selection too, with parent manifest digests and its own content digest, media type and size. Read an artifact, or follow its parentage, by manifest digest. The content digest addresses its bytes. The manifest digest addresses those bytes plus everything that produced them. Two rendered pages can have the same image bytes and different manifest digests. ## Traps Do not turn a whole page into a one-column table to extract one number. Name the page and box that hold the printed token, then run `select_spans` against the retained spans artifact. Under `ambiguity_policy: "refuse"` the region must contain one span. Under `"preserve"` the pipeline retains every matching span in source order, so a reviewer can see the region was ambiguous. Do not merge native and OCR spans into a hybrid table. Retain the OCR spans and their image geometry first, then build the candidate against those. The 128 MiB source ceiling is a retention limit, not an extraction guarantee. Certified operations use a 32 MiB JSON transport envelope and inline the source as base64. A source at or above 24 MiB refuses before that copy is allocated. The usable size drops further when the operation also carries digests or retained evidence. An instruction can narrow a certified ceiling and can never widen one. Confidence is evidence, not proof. It never upgrades an OCR claim to a native-text claim.