# Readers

Source: https://mostlyright.md/docs/reference/readers/

A Reader turns fetched bytes into rows. Select an exact `family_id@family_version` in the recipe.

## How a recipe pins a Reader

A v4 source names its Reader on the connector, as three reserved parameters. All three, or none.

```json title="the Reader pin"
"connector": {
  "adapter_id": "public.https",
  "credential_mode": "none",
  "origin": "https://example.org",
  "parameters": [
    { "name": "reader.family_id",      "value": "json.tabular" },
    { "name": "reader.family_version", "value": "1.1.0" },
    { "name": "reader.decode_options", "value": "{\"columns\":[{\"name\":\"icao_id\",\"pointer\":\"/icaoId\",\"required\":true}],\"document_format\":\"json\",\"expand\":[],\"records_pointer\":\"\"}" }
  ]
}
```

`declared_source` has no member for a decoder. The pin rides on `connector.parameters`, which the source authority digest already folds in. Registration refuses a fourth name under `reader.` rather than ignoring it.

`reader.decode_options` is the canonical JSON text of the options the family itself accepts, with every family-defaulted key filled in. A stated subset digests differently from the pin the worker reconstructs, so registration refuses it and quotes the spelling that works. The value runs to 262144 bytes (256 KiB), and every other connector parameter value runs to 2048.

Reader implementations run on the backend and are not included in the thin CLI installation. Use the complete settings shown for the pinned family below. Serialize the admitted options with sorted keys and compact separators; this canonicalizes JSON, not family defaults. A canonical-options refusal supplies the complete expected object.

```python
import json

value = json.dumps({"encoding": "utf-8"}, sort_keys=True, separators=(",", ":"))
# Complete options for text.lines@1.0.0.
```

A column that may be absent from a record must state `"required": false`. Publishers routinely omit a key rather than sending null, and a missing required column refuses the record.

Studio checks the envelope at registration without loading Reader code. The `(family_id, family_version)` pair must be one of the coordinates this page lists. The options must be an object, and the bytes must be canonical JSON inside 256 KiB. The worker runs family-specific admission before it fetches anything.

## Where a Reader runs

A Reader needs a memory ceiling of its own, and the warm session worker gives a child none. Admission stamps a Reader-pinned source for the batch shape and refuses it on the warm shape before any fetch. A generic API response always names a Reader inside its frozen configuration, so it takes the same shape.

Three source shapes accept a certified Reader pin:

- a hash-pinned `user.file` artifact
- a credential-free public HTTPS source
- a bounded ordered public HTTPS collection (`public.https.collection@2.0.0`). It applies one exact pin to every member and combines equal-schema normalized rows after all members succeed

The worker narrows the response media type to the exact pinned family before it makes the request. Fetched source bytes never leave the confined operation: retrieval, decode and parse are one call into the worker sandbox, and only normalized bytes come back.

On macOS the worker sandbox refuses with `SANDBOX_MEMORY_BOUNDARY`, because there is no cgroup to bound a Reader's memory with. It refuses every other sandbox operation with `SANDBOX_OS_BOUNDARY`. A single-address credential-free public HTTPS acquisition runs in the hosted Linux sandbox from any machine. Bounded collections have no other route.

## Picking a version

For an unreviewed source, name the family's highest certified version. A base version rejects a publisher that merely declined to describe its file. A weak-label widening requires a new family version.

Where a major version has moved, the highest version is a different output contract. `weather.grib2@2.0.0` emits columns `1.0.0` does not. A version change is a recipe revision.

## The registry

A Reader coordinate is the exact `(family_id, family_version)` pair, and resolution takes nothing else. There is no plugin scan, entry point or environment override.

| Coordinate | Accepted media types | Settings | Optional dependency |
| --- | --- | --- | --- |
| `archive.gzip@1.0.0` | `application/gzip` `application/x-gzip` | `member_format`, plus optional `header: true`. `member` is invalid. | none |
| `archive.gzip@1.1.0` | `application/gzip` `application/octet-stream` `application/x-gzip` | Same, plus a response served as `application/octet-stream`. | none |
| `archive.tar@1.0.0` | `application/x-tar` | Same as `archive.zip@1.0.0`. | none |
| `archive.tar@1.1.0` | `application/octet-stream` `application/x-tar` | Same, plus a response served as `application/octet-stream`. | none |
| `archive.tar@2.0.0` | `application/gzip` `application/x-gzip` `application/x-tar` `application/octet-stream` | Explicit compression and encoding; exact member or member list; see physical lines below. | none |
| `text.lines@1.0.0` | `text/csv` `text/plain` `text/tab-separated-values` `application/octet-stream` | Explicit encoding; physical line number and original line text. | none |
| `archive.zip@1.0.0` | `application/x-zip-compressed` `application/zip` | `member`, `member_format` (`csv` or `tsv`), and optional `header: true`. | none |
| `archive.zip@1.1.0` | `application/x-zip-compressed` `application/zip` | Same, plus `member_format: semicolon` for publishers such as DWD. | none |
| `archive.zip@1.2.0` | `application/octet-stream` `application/x-zip-compressed` `application/zip` | Same as `archive.zip@1.1.0`, plus a response served as `application/octet-stream`. | none |
| `archive.zip@1.3.0` | `application/octet-stream` `application/x-zip-compressed` `application/zip` | Same as `archive.zip@1.2.0`; explicit `encoding: cp1252` or `utf-8`, with optional zero-cell record skipping. | none |
| `delimited_text@1.0.0` | `text/csv` `text/tab-separated-values` | `delimiter` (comma or tab), optional `header: true`, optional `encoding: utf-8`. | none |
| `delimited_text@1.1.0` | `text/csv` `text/plain` `text/tab-separated-values` | Same, plus a response served as `text/plain`, common for public agency CSV such as IEM. | none |
| `delimited_text@2.0.0` | `application/csv` `application/octet-stream` `application/vnd.ms-excel` `application/vnd.sdmx.data+csv` `application/x-csv` `text/csv` `text/plain` `text/tab-separated-values` `text/x-csv` | Explicit delimiter, encoding, quoting, header presence, skip_rows, skip_blank_rows, and indexed columns with output names and optional expected_header. The setting unselected_columns requires empty cells unless it is set to allow. | none |
| `geojson.vector@1.0.0` | `application/geo+json` `application/json` | Required property projection. Optional `include_id` and `include_geometry` retain Feature identifiers and RFC 7946 geometries. | none |
| `geopackage.vector@1.0.0` | `application/geopackage+sqlite3` `application/x-sqlite3` | `table`, `geometry_column` and explicit property `columns`. Only EPSG:4326 GeoPackage features are accepted. | none |
| `geoparquet.vector@1.0.0` | `application/vnd.apache.parquet` | `geometry_column` and explicit property `columns`. Only primary WKB GeoParquet geometry is accepted. | none |
| `html.tabular@1.0.0` | `text/html` | Absolute record element path and relative text/attribute field selectors. A field is required unless it states `required: false`. | none |
| `html.web_extract@1.0.0` | `text/html` | `mode: dom` with closed CSS-like record/field selectors, or `mode: json` with an inert JSON script selector and RFC 6901 field pointers. | none |
| `html.web_extract@1.1.0` | `text/html` | Same settings as 1.0.0. Extracted text follows DOM order, with one space at block-element boundaries and `<br>`, and none between inline runs. | none |
| `html.web_extract@1.2.0` | `text/html` | Adds bounded integer `:nth-child(N)` and `:nth-of-type(N)`, plus field `scope: ancestor` with `ancestor_selector`. Requires deployed certification. | none |
| `image.rendered@1.0.0` | `image/jpeg` `image/png` | `format` plus a bounded `x`/`y`/`width`/`height` rendered-image crop. | `Pillow` |
| `json.tabular@1.0.0` | `application/json` `application/x-ndjson` | `columns` with output names and RFC 6901 pointers, plus optional `document_format` (`json` or `ndjson`), `records_pointer` and sequential `expand` pointers. A column is required unless it states `required: false`. | none |
| `json.tabular@1.1.0` | `application/json` `application/x-ndjson` `text/plain` | Same, plus a response served as `text/plain`, common for public agency JSON and NDJSON. | none |
| `json.tabular@1.2.0` | Same as 1.1.0 | Optional expansion entries `{ "pointer": "/x", "required": false }` skip null or missing arrays; present non-arrays still refuse. | none |
| `meteorology.bufr@1.0.0` | `application/bufr` `application/octet-stream` | Exact `message_index` and `subset_index` for the certified cloud fields. | `eccodes` |
| `meteorology.grib@1.0.0` | `application/octet-stream` `application/wmo-grib` `application/wmo-grib2` | Exact GRIB1/GRIB2 message selector plus canonical-decimal named nearest points. | `eccodes` |
| `microdata.sas_xpt@1.0.0` | `application/x-sas-transport` | Required ordered stored-column projection. SAS Transport XPT only, raw values only. | `pyreadstat` |
| `microdata.spss_sav@1.0.0` | `application/x-spss-sav` | Required ordered stored-column projection. Raw values only, no labels or display formats. | `pyreadstat` |
| `microdata.stata_dta@1.0.0` | `application/x-stata` | Required ordered stored-column projection. Raw values only, no labels or display formats. | `pyreadstat` |
| `official.pcaxis_px@1.0.0` | `application/x-pcaxis` `text/plain` | `dimensions` pins STUB-then-HEADING order, with `value_column` and optional per-code `unit`. UTF-8 numeric PX tables only. | none |
| `official.sdmx_json@1.0.0` | `application/json` | `dimensions` pins every series/observation dimension in order, with optional selected `attributes` (including units) and `value_column`. Compact one-dataset SDMX-JSON only. | none |
| `parquet.tabular@1.0.0` | `application/vnd.apache.parquet` | Optional `columns` selects a bounded ordered Parquet projection. | none |
| `parquet.tabular@1.1.0` | Same as 1.0.0 | Required `{path, as}` columns, dotted struct paths and optional `all` row predicates. Public HTTPS reads only the footer and selected chunks by range. | none |
| `protobuf.tabular@1.0.0` | `application/octet-stream` `application/protobuf` `application/vnd.google.protobuf` | Digest-pinned base64 FileDescriptorSet, fully-qualified message type, `single` or `length_delimited` framing, and fields with an explicit repeated `explode` or `refuse` choice. | `protobuf` |
| `raster.geospatial@1.0.0` | `image/j2c` `image/jp2` `image/tiff` `image/x-jp2` | `format`, exact native `crs`, and an `x`/`y`/`width`/`height` window. | `rasterio` |
| `satellite.goes_abi_l2@1.0.0` | `application/netcdf` `application/x-netcdf` | `variable: CMI` and exact `x`/`y` inclusive-exclusive pixel ranges. | `netCDF4` |
| `satellite.himawari_hsd@1.0.0` | `application/octet-stream` `application/x-himawari-hsd` | Certified HSD v1.3 single-segment B13 plus an exact native `x`/`y`/`width`/`height` radiance window. | `numpy` |
| `satellite.himawari_hsd_complete@1.0.0` | `application/zip` | Complete HSD v1.3 scan ZIP: `band_id`, exact ordered `segment_members`, and a native `x`/`y`/`width`/`height` radiance window. | `numpy` |
| `satellite.msg_seviri_l15@1.0.0` | `application/zip` | Exact MSG prologue, eight ordered VIS/IR segments and epilogue. Recorded channel, quality, navigation and calibration, plus a bounded radiance window. | `pyPublicDecompWT` |
| `satellite.sentinel_safe@1.0.0` | `application/x-zip-compressed` `application/zip` | SAFE ZIP with exact spectral/quality member names, native CRS, quality layer, and an `x`/`y`/`width`/`height` window. | `rasterio` |
| `satellite.xrit_image@1.0.0` | `application/zip` | Exact ZIP prologue, eight ordered HRIT/LRIT image segments and epilogue. Recorded channel, grid and navigation, plus a bounded raw-count window. | `pyPublicDecompWT` |
| `scientific.fits@1.0.0` | `application/fits` `application/octet-stream` | `band_id` plus an exact native `x`/`y`/`width`/`height` image window. | `astropy` |
| `scientific.hdf4@1.0.0` | `application/x-hdf4` | Linux x86_64/macOS arm64 HDF4 SD only: exact `dataset_name`, same-rank `ranges`, recorded shape/dtype/semantic attributes, and explicit raw values. | `pyhdf` |
| `scientific.hdf5@1.0.0` | `application/x-hdf5` | Exact `dataset_path`, same-rank `ranges`, recorded shape/dtype/chunks/compression/semantic attributes, and explicit raw-value representation. | `h5py` |
| `scientific.hdf_eos2@1.0.0` | `application/x-hdf4` | Linux x86_64/macOS arm64 HDF-EOS2 Grid/Swath raw field: exact object/field metadata, same-rank `ranges`, recorded SD attributes, and explicit raw values. | `pyhdf` |
| `scientific.netcdf@1.0.0` | `application/netcdf` `application/x-netcdf` | NetCDF-3/4 with exact root `variable`, same-rank `ranges`, recorded shape/dtype/chunks/compression/semantic attributes, and explicit raw-value representation. | `netCDF4` |
| `scientific.zarr@1.0.0` | `application/zip` | ZIP-packaged Zarr v2/v3 with exact `array_path`, same-rank `ranges`, recorded shape/dtype/chunks/compression/semantic attributes, and explicit raw-value representation. | `zarr` |
| `spreadsheet.xlsx@1.0.0` | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | `sheet`, `header_row`, and `range`. The heading row must be the first row of the range. | none |
| `spreadsheet.xlsx@1.1.0` | `application/octet-stream` `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | Same, plus a response served as `application/octet-stream`. | none |
| `weather.grib2@1.0.0` | `application/octet-stream` `application/wmo-grib2` | `variable`, `points`, and `admission_allowlist_version`. | `mostlyright-grib` |
| `weather.grib2@2.0.0` | `application/octet-stream` `application/wmo-grib2` | Same settings as 1.0.0. The output also includes time, level, matched coordinates and message digest. | `mostlyright-grib` |
| `weather.grib2@3.0.0` | `application/octet-stream` `application/wmo-grib2` | Exact collection message selector, `variable_name`, `points`, and allowlist version. | `mostlyright-grib` |
| `xbrl.facts@1.0.0` | `application/xhtml+xml` `application/xml` | `document_kind` (`xbrl` or constrained XML `ixbrl`) and exact namespace/name fact projection. Contexts, units and metadata remain evidence fields, not joins. | none |
| `xml.tabular@1.0.0` | `application/rss+xml` `application/xml` `text/xml` | Absolute record element path, declared namespace prefixes, and relative field selectors ending in `text()` or `@attribute`. A field is required unless it states `required: false`. | none |

The runtime loads an optional dependency lazily, so a family can be described without importing it. First use without a compatible binding returns `READER_DEPENDENCY_UNAVAILABLE`, and the rest of the CLI stays usable.

## Settings, family by family

Each block is the `decode_options` object, before canonical form turns it into the `reader.decode_options` string. One record is one output row unless the family says otherwise.

### Delimited text

One record is one data row. `delimited_text@1.0.0` and `@1.1.0` read a file whose first row is the heading.

```json
{ "delimiter": ",", "header": true, "encoding": "utf-8" }
```

`delimited_text@2.0.0` reads a file with a publisher preamble, no heading, or padded columns. The recipe selects columns by index, and `expected_header` asserts what the heading says.

```json
{
  "skip_rows": 2,
  "skip_blank_rows": true,
  "columns": [
    { "index": 0, "name": "Date", "expected_header": "Date" },
    { "index": 1, "name": "food_index", "expected_header": "Food Price Index" },
    { "index": 2, "name": "meat_index", "expected_header": "Meat" }
  ]
}
```

### CSV downloads served as octet-stream

Some publishers, including ČHMÚ's hourly air-quality files, serve CSV as
`application/octet-stream`. A request `Accept: text/csv` does not override the response
`Content-Type`. This label does not make the file a ZIP archive.

`delimited_text@2.0.0` accepts this response type and applies its strict text, binary-container,
markup and budget checks. Direct CSV and the older delimited-text pins have narrower admission.
Use `normalize_reader_options` to confirm the deployed coordinate, options and
the accepted media types in the matching `reader_capabilities.readers` entry, then test
the same source with its original coverage and bounds. A normalized option set proves option
validation, not that the source was read. Existing approval and spend holds still apply.

Preserve headings exactly when using `expected_header`: a publisher may include spaces after
commas. Rename output columns explicitly rather than pretending those spaces were absent.

### JSON and NDJSON

One record is one element of the array `records_pointer` names. `records_pointer: ""` is the RFC 6901 pointer for a root-level array. `expand` walks a nested array sequentially, so one parent record becomes one row per child.

```json
{
  "document_format": "json",
  "records_pointer": "/stations",
  "expand": ["/observations"],
  "columns": [
    { "name": "station_id", "pointer": "/id", "required": true },
    { "name": "time", "pointer": "/observations/time", "required": true },
    { "name": "rain", "pointer": "/observations/rain", "required": true }
  ]
}
```

NDJSON is the same family with `document_format: "ndjson"` and a root records pointer.

```json
{
  "document_format": "ndjson",
  "records_pointer": "",
  "expand": [],
  "columns": [
    { "name": "station", "pointer": "/station", "required": true },
    { "name": "valid", "pointer": "/valid", "required": true },
    { "name": "tmpf", "pointer": "/tmpf", "required": false }
  ]
}
```

### Physical lines and compressed TAR files

These two coordinates require the Studio API and worker release containing them. Updating the CLI alone does not add Reader support to a deployment; a deployment that has not received that release will refuse the new coordinate.

Use `text.lines@1.0.0` for publisher text with preambles, inconsistent delimiters or
unbalanced quotes. Set `encoding` explicitly to `utf-8` or `cp1252`. Output is
`line_number, raw_line`; numbering starts at 1. Blank lines, tabs, quotes and a BOM are
preserved. Only CR, LF and CRLF terminators are removed. No field parsing or repairs occur.

`archive.tar@2.0.0` adds an explicit `compression` setting (`none` or `gzip`) and requires
`encoding`. Earlier TAR coordinates remain unchanged. To read one file, set `member` to its
exact name and `member_format` to `csv`, `tsv`, `semicolon` or `lines`. Delimited forms require
`header: true`; for preambles or inconsistent rows, use `lines` and parse in the recipe.

For a quarterly bundle, `member_format: lines` alternatively accepts `members`, an explicit
list of 1–1024 unique exact names. `member` and `members` are mutually exclusive. The canonical
list is sorted; output follows archive order and carries `member_name, line_number, raw_line`.
A missing, duplicate, sparse, linked or nested selected file refuses the whole decode. There is
no wildcard selection or directory extraction. An unrelated file is not added automatically.

Gzip inflation is streamed and checked against the total uncompressed bytes and expansion ratio,
including unselected files and the trailer. Only the selected file and canonical output are
retained, not the whole inflated TAR. Row, cell and output limits apply across all selected files.
This avoids downloading one quarterly archive separately for every daily file. It does not
increase the final dataset row limit; plan the final panel within the build limits. A sample
row cap does not bypass archive integrity or reader limits.

```json
{"compression":"gzip","encoding":"utf-8","member_format":"lines",
 "members":["prezzo_alle_8-20240101.csv","prezzo_alle_8-20240102.csv"]}
```

Exact names must come from the publisher's archive inventory. The example illustrates the
option shape, not a claim that every publisher uses those names.

### Containers

One record is one data row of the named member. The worker opens the container, reads one member, and decodes that member as delimited text. `archive.gzip` holds one stream, so `member` is invalid there.

```json title="archive.zip"
{ "header": true, "member": "produkt_klima_tag_00044_akt.txt", "member_format": "semicolon" }
```

```json title="archive.tar"
{ "header": true, "member": "cities.tsv", "member_format": "tsv" }
```

```json title="archive.gzip"
{ "header": true, "member_format": "csv" }
```

### Spreadsheets

One record is one row of the named range. The heading row must be the first row of the range.

```json
{ "sheet": "Cities", "header_row": 1, "range": "A1:C3" }
```

Receipts identify stored calculated values, so a workbook that shipped computed cells says so.

### Parquet

One record is one Parquet row. `columns` is optional and selects a bounded ordered projection of at most 512 names.

```json
{ "columns": ["symbol", "price"] }
```

### XML

One record is one match of `record_selector`, an absolute element path. Field selectors are relative and end in `text()` or `@attribute`. The recipe declares namespace prefixes, and nothing guesses them.

```json
{
  "record_selector": "/sec:edgarSubmission/sec:offeringData",
  "namespaces": { "sec": "https://www.sec.gov/edgar/formd" },
  "columns": [
    { "name": "issuer_name", "selector": "sec:issuer/sec:issuerName/text()" },
    { "name": "state", "selector": "sec:issuer/sec:issuerAddress/@stateOrCountry" }
  ]
}
```

### HTML tables

One record is one match of `record_selector` over the parsed document.

```json
{
  "record_selector": "/table/tr",
  "columns": [
    { "name": "id", "selector": "@data-id" },
    { "name": "name", "selector": "th/text()" },
    { "name": "count", "selector": "td/text()" }
  ]
}
```

### HTML pages

`html.web_extract` has two modes. `mode: dom` selects records with a closed CSS-like selector grammar. A column with `scope: "document"` reads once per page, and a column with `scope: "record"` reads per match.

```json title="mode: dom"
{
  "mode": "dom",
  "record_selector": "article.story",
  "columns": [
    { "name": "headline", "selector": "h1.headline" },
    { "name": "report", "selector": "div.report" },
    { "name": "price", "selector": "p.price" }
  ]
}
```

`mode: json` reads an inert JSON script block and pulls fields out of it with RFC 6901 pointers.

```json title="mode: json"
{
  "mode": "json",
  "document_selector": "script[type=\"application/ld+json\"]",
  "document_policy": "refuse",
  "records_pointer": "/itemListElement",
  "columns": [
    { "name": "title", "pointer": "/name" },
    { "name": "score", "pointer": "/aggregateRating/ratingValue" },
    { "name": "link", "pointer": "/url" }
  ]
}
```

`1.1.0` changes the text rule and nothing else. Extracted text follows DOM order, with one space at block-element boundaries and at `<br>`, and none between inline runs. For a new source, use `1.2.0` when the deployment certifies it; it retains this text behavior and adds positional and ancestor selectors. `1.0.0` exists so recipes sealed under it still replay.

### Protocol buffers

One record is one message, or one exploded repeated field. The descriptor set rides inline as digest-pinned base64, and every repeated field states `explode` or `refuse`.

```json
{
  "message_type": ".transit_realtime.FeedMessage",
  "framing": "single",
  "descriptor_set": { "base64": "CqJNChNndGZzLXJlYWx0aW1lLnByb3Rv…", "sha256": "…" },
  "columns": [
    { "name": "entity_id", "path": "entity.id", "repeated": "explode" },
    { "name": "trip_id", "path": "entity.trip_update.trip.trip_id", "repeated": "explode" }
  ]
}
```

### Geospatial vector

One record is one Feature. `geojson.vector` keeps the Feature id and the RFC 7946 geometry only when asked.

```json title="geojson.vector"
{
  "columns": [
    { "name": "place", "property": "name" },
    { "name": "population", "property": "population", "required": false }
  ],
  "include_geometry": true,
  "include_id": true
}
```

```json title="geoparquet.vector"
{ "geometry_column": "geom", "columns": [{ "name": "place", "field": "name" }] }
```

```json title="geopackage.vector"
{ "table": "places", "geometry_column": "geom", "columns": [{ "name": "place", "field": "name" }] }
```

The reader accepts only primary WKB GeoParquet geometry, and only EPSG:4326 GeoPackage features.

### Official statistics

One record is one observation. `dimensions` pins the order, and nothing infers it from the file.

```json title="official.sdmx_json"
{
  "dimensions": [
    { "id": "REF_AREA", "name": "area" },
    { "id": "TIME_PERIOD", "name": "period" }
  ],
  "attributes": [
    { "scope": "series", "id": "UNIT_MEASURE", "name": "unit" },
    { "scope": "observation", "id": "CONF_STATUS", "name": "status" }
  ],
  "value_column": "value"
}
```

```json title="official.pcaxis_px"
{
  "dimensions": [
    { "id": "Region", "name": "region" },
    { "id": "ContentsCode", "name": "measure" },
    { "id": "Year", "name": "year" }
  ],
  "unit": { "dimension": "ContentsCode", "name": "unit" },
  "value_column": "value"
}
```

Compact one-dataset SDMX-JSON only. UTF-8 numeric PX tables only.

### Statistical microdata

One record is one case. The projection is required and ordered. The values are the stored ones, with no labels and no display formats.

```json title="microdata.stata_dta, .spss_sav, .sas_xpt"
{ "columns": ["score", "id"] }
```

### Financial filings

One record is one fact. Contexts, units and metadata are evidence fields beside the fact, not joins.

```json
{
  "document_kind": "xbrl",
  "facts": [
    {
      "namespace": "urn:example:us-gaap",
      "name": "RevenueFromContractWithCustomerExcludingAssessedTax"
    }
  ]
}
```

`document_kind: "ixbrl"` reads a constrained inline-XBRL XML document instead.

### Weather

`weather.grib2@1.0.0` and `@2.0.0` open one extracted GRIB2 record. They refuse a multi-record GFS or HRRR file. Supply the exact record bytes from the publisher's index and byte range. `@2.0.0` emits the same rows plus time, level, matched coordinates and the message digest.

```json title="weather.grib2@1.0.0 and @2.0.0"
{
  "admission_allowlist_version": 1,
  "variable": {
    "discipline": 0,
    "name": "temperature_2m",
    "parameter_category": 0,
    "parameter_number": 0,
    "surface_type": 103,
    "surface_value": 2
  },
  "points": [
    { "id": "prague", "latitude": "50.0755", "longitude": "14.4378" },
    { "id": "denver", "latitude": "39.7392", "longitude": "-104.9903" }
  ]
}
```

`weather.grib2@3.0.0` accepts a bounded multi-record collection. It selects one message by its recorded field, surface, run and valid-time values, then reuses the version-2 decoder. Zero matches and multiple matches both refuse.

```json title="weather.grib2@3.0.0"
{
  "variable_name": "temperature_2m",
  "admission_allowlist_version": 1,
  "collection_profile_version": 1,
  "selector": {
    "discipline": 0,
    "parameter_category": 0,
    "parameter_number": 0,
    "surface_type": 103,
    "surface_value": { "numerator": 2, "denominator": 1 },
    "run_time": "2026-08-05T12:00:00Z",
    "valid_time": "2026-08-05T12:00:00Z"
  },
  "points": [{ "id": "north_west", "latitude": "21.1381", "longitude": "237.2805" }]
}
```

One record is one point. Latitudes and longitudes are canonical decimal strings, never floats, because the pin is digested.

### Meteorology through ecCodes

```json title="meteorology.grib"
{
  "profile_version": 1,
  "selector": {
    "edition": 1,
    "param_id": 167,
    "type_of_level": "surface",
    "level": 0,
    "data_date": 20070323,
    "data_time": 1200,
    "step_range": "0"
  },
  "points": [{ "id": "origin", "latitude": "0", "longitude": "0" }]
}
```

```json title="meteorology.bufr"
{ "message_index": 0, "subset_index": 0 }
```

### Scientific arrays

One record is one cell of the selected window. `ranges` is same-rank with the array. `expected_metadata` is fixed: the reader refuses a file whose shape, dtype, chunking, compression or semantic attributes differ from what the pin declares.

```json title="scientific.hdf5"
{
  "dataset_path": "measurements/temperature",
  "ranges": [{ "start": 0, "stop": 2 }, { "start": 1, "stop": 3 }],
  "expected_metadata": {
    "shape": [2, 3],
    "dtype": "int32",
    "attributes": { "units": "K", "long_name": "air_temperature" },
    "chunks": null,
    "compression": "none"
  },
  "value_representation": "raw"
}
```

`scientific.netcdf` takes `variable` in place of `dataset_path`. `scientific.zarr` takes `array_path`. `scientific.hdf4` takes `dataset_name`. `scientific.hdf_eos2` takes `dataset_name` plus `eos_object_type` (`grid` or `swath`), `eos_object_name` and `field_name`.

```json title="satellite.goes_abi_l2"
{ "variable": "CMI", "y": { "start": 0, "stop": 2 }, "x": { "start": 1, "stop": 3 } }
```

### Raster and rendered images

One record is one pixel for `raster.geospatial`. `image.rendered` emits one row for the whole crop.

```json title="raster.geospatial"
{ "format": "geotiff", "crs": "EPSG:4326", "x": 1, "y": 1, "width": 2, "height": 2 }
```

```json title="image.rendered"
{ "format": "png", "x": 1, "y": 0, "width": 2, "height": 2 }
```

```json title="scientific.fits"
{ "band_id": "195", "x": 0, "y": 1, "width": 2, "height": 1 }
```

### Satellite segments

```json title="satellite.sentinel_safe"
{
  "band": "B02",
  "raster_member": "GRANULE/TILE/IMG_DATA/TILE_B02_10m.tif",
  "quality_layer": "SCL",
  "quality_member": "GRANULE/TILE/IMG_DATA/TILE_SCL_10m.tif",
  "crs": "EPSG:4326",
  "x": 1, "y": 0, "width": 2, "height": 2
}
```

```json title="satellite.himawari_hsd"
{ "band_id": "B13", "x": 0, "y": 0, "width": 2, "height": 1 }
```

```json title="satellite.himawari_hsd_complete"
{
  "band_id": "B13",
  "segment_members": ["segment-01.dat", "segment-02.dat", "segment-03.dat"],
  "x": 0, "y": 1, "width": 2, "height": 3
}
```

`satellite.xrit_image` and `satellite.msg_seviri_l15` take a prologue member, exactly eight ordered segment members and an epilogue member. They also take the channel and grid constants (`band_id`, `bits_per_pixel`, `cfac`, `coff`, `lfac`, `loff`, `columns`, `lines_per_segment`) and a bounded window.

```json title="satellite.msg_seviri_l15"
{
  "band_id": 9,
  "bits_per_pixel": 10,
  "cfac": -13642337, "coff": 1856,
  "lfac": -13642337, "loff": 1856,
  "columns": 3712,
  "lines_per_segment": 464,
  "prologue_member": "prologue.hrit",
  "segment_members": [
    "segment-01.hrit", "segment-02.hrit", "segment-03.hrit", "segment-04.hrit",
    "segment-05.hrit", "segment-06.hrit", "segment-07.hrit", "segment-08.hrit"
  ],
  "epilogue_member": "epilogue.hrit",
  "x": 0, "y": 0, "width": 2, "height": 8
}
```

## Budgets

Every family declares default budgets. A recipe may narrow them per field and never widen them. Registration refuses a cap that names a field the contract does not have, by name.

| Budget group | Families | `max_input_bytes` | `max_output_bytes` | `max_uncompressed_bytes` | `max_declared_cells` | `max_rows` | `max_columns` |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Bulk | `archive.*`, `delimited_text`, `json.tabular`, `parquet.tabular`, `geojson.vector`, `geoparquet.vector`, `geopackage.vector`, `official.*` | 256 MiB | 4 GiB | 4 GiB | 400,000,000 | 10,000,000 | 1,024 |
| Streaming tabular | `xml.tabular`, `html.tabular`, `protobuf.tabular` | 16 MiB | 16 MiB | 16 MiB | 4,000,000 | 1,000,000 | 256 |
| DOM | `html.web_extract` | 16 MiB | 16 MiB | 16 MiB | 4,000,000 | 250,000 | 256 |
| Spreadsheet | `spreadsheet.xlsx` | 16 MiB | 16 MiB | 32 MiB | 250,000 | 250,001 | 256 |
| Microdata | `microdata.*` | 16 MiB | 64 MiB | 64 MiB | 5,000,000 | 100,000 | 128 |
| Filings | `xbrl.facts` | 8 MiB | 32 MiB | 32 MiB | 1,000,000 | 100,000 | 6 |
| Contract default | `weather.grib2` (all versions) | 16 MiB | 16 MiB | 128 MiB | 10,000,000 | 1,000,000 | 1,024 |

The scientific, satellite and raster families each declare their own:

| Coordinate | `max_input_bytes` | `max_output_bytes` | `max_uncompressed_bytes` | `max_declared_cells` | `max_rows` | `max_columns` |
| --- | --- | --- | --- | --- | --- | --- |
| `satellite.goes_abi_l2@1.0.0` | 128 MiB | 64 MiB | 256 MiB | 4,000,000 | 1,000,000 | 4 |
| `scientific.hdf5@1.0.0` | 128 MiB | 64 MiB | 128 MiB | 1,000,000 | 250,000 | 33 |
| `scientific.netcdf@1.0.0` | 128 MiB | 64 MiB | 128 MiB | 1,000,000 | 250,000 | 33 |
| `scientific.hdf4@1.0.0` | 64 MiB | 32 MiB | 64 MiB | 500,000 | 125,000 | 33 |
| `scientific.hdf_eos2@1.0.0` | 64 MiB | 32 MiB | 64 MiB | 500,000 | 125,000 | 33 |
| `scientific.zarr@1.0.0` | 64 MiB | 64 MiB | 96 MiB | 1,000,000 | 250,000 | 33 |
| `raster.geospatial@1.0.0` | 128 MiB | 64 MiB | 256 MiB | 100,000,000 | 1,000,000 | 11 |
| `satellite.sentinel_safe@1.0.0` | 128 MiB | 64 MiB | 192 MiB | 100,000,000 | 250,000 | 22 |
| `image.rendered@1.0.0` | 64 MiB | 64 MiB | none | 1,000,000 | 1 | 7 |
| `scientific.fits@1.0.0` | 128 MiB | 64 MiB | none | 5,000,000 | 1,000,000 | 1,024 |
| `satellite.himawari_hsd@1.0.0` | 128 MiB | 64 MiB | 128 MiB | 1,750,000 | 250,000 | 7 |
| `satellite.himawari_hsd_complete@1.0.0` | 512 MiB | 64 MiB | 128 MiB | 1,750,000 | 250,000 | 7 |
| `satellite.xrit_image@1.0.0` | 128 MiB | 32 MiB | 64 MiB | 1,000,000 | 250,000 | 4 |
| `satellite.msg_seviri_l15@1.0.0` | 128 MiB | 32 MiB | 64 MiB | 1,000,000 | 250,000 | 11 |
| `meteorology.bufr@1.0.0` | 128 MiB | 64 MiB | none | 1,000,000 | 1,000,000 | 1,024 |
| `meteorology.grib@1.0.0` | 128 MiB | 64 MiB | 256 MiB | 1,500,000 | 100,000 | 15 |

A run narrows `max_output_bytes` further. The decoded output of one source stops at the lowest of the family's number, 1 GiB, and one third of the decode child's address space. So a bulk family's 4 GiB is the family's own number and 1 GiB is what a run reaches.

A cell reading none means the family states no value of its own, and the contract default applies: `max_uncompressed_bytes` 128 MiB, `max_expansion_ratio` 100, `max_container_members` 1,024. Two families set the member count below that default: `satellite.sentinel_safe` to 256, and `satellite.himawari_hsd_complete` to 99. One sets it above, and that is `scientific.zarr` at 4,096. `meteorology.grib` states 1,024, which restates the default rather than narrowing it, and so does `scientific.zarr`'s `max_expansion_ratio` of 100. Only the microdata and filing families narrow the expansion ratio, to 4. `max_nesting_depth` is fixed at 1 and cannot be set. It is a security invariant, not a budget.

Routing and resource feasibility are separate facts. A source may route to the right family and still exceed its uncompressed, output or declared-cell ceilings. It then reports `READER_BUDGET`, the truthful answer, rather than `MEDIA_TYPE`. A source's file and its decoded table both have to fit.

## Previewing a Reader before registration

`mr-data peek` against a run reads the preview the run already wrote. Its `--reader`, `--reader-options` and `--format` flags are accepted and reported as having no effect. No command tries a Reader against a URL.

Settle a Reader by pinning it in the recipe, registering, then taking a sample without `--window` under `--max-rows`. Read the columns back with `peek` and the decode evidence with `receipt`. A wrong pin is a typed refusal on that run, naming the source. See [Probe a source](/docs/build/probe-sources/) for the reading path.

## Refusals

Reader settings are closed. Unknown settings, unsupported wrappers, missing archive members, invalid sheets or ranges and unsupported delimiters all produce typed refusals. So do invalid UTF-8, missing headings, multi-record weather files, unapproved grids or packing, lossy codecs, uncovered points, truncated files and corrupt values. Change the recipe or the source. Do not widen validation, and do not report a refused build as an empty result.

| Code | Meaning | What the author does |
| --- | --- | --- |
| `READER_FAMILY_UNKNOWN` | the pin names a family the registry does not hold | select a registered certified Reader family |
| `READER_CONTRACT` | the pin does not name this family, or the pin is malformed | fix `reader.family_id` / `reader.family_version` |
| `READER_OPTIONS` | one setting is outside the family's closed option set: an unknown key, the wrong type, an uncertified value | correct the closed Reader options and run the same acquisition again |
| `READER_ADMISSION` | the settings are outside what this family accepts | re-derive `decode_options` through the family's own `validate_options` |
| `READER_BUDGET` | the input, output, cell, row or column ceiling was crossed | narrow the projection or the window, or ask the publisher for a smaller slice |
| `READER_DECODE` | the bytes did not decode under these settings | check the member, sheet, range, delimiter or selector |
| `READER_DEPENDENCY_UNAVAILABLE` | the family's optional binding is not installed on this worker | the coordinate is not usable on this deployment, so pick another family |
| `READER_MEMBER_MISSING` | the named archive member is not in the container | inspect the archive members and set the member option to the exact intended file |
| `READER_MEMBER_AMBIGUOUS` | the container holds the named member more than once, and a recipe names exactly one | re-pack the archive, or name a member that appears once |
| `READER_MEMBER_TYPE` | the named entry is not a regular file: a directory, symbolic link, hard link, fifo, device or socket, or an encrypted entry | name a regular file. A symbolic link stores the path it points at, not data |
| `READER_NESTED_CONTAINER` | the member is itself an archive or an executable, and containers open one layer deep | extract the inner container at the publisher, or name a member that is not one |
| `READER_SHEET_MISSING` | the workbook does not hold the named sheet | name a sheet the workbook holds. The refusal lists them |
| `READER_RANGE` | the declared window is not a readable range: a header cell that is empty or not text, or a repeated column name | move the header row or the window. A blank or duplicate heading refuses rather than getting an auto-name |
| `READER_GRID_UNKNOWN` | the weather message is on a grid template this product has never been certified against | use a message on a certified grid. Approving one means adding the pair to the allowlist and re-certifying |
| `READER_PACKING_UNKNOWN` | the weather message uses a packing template this product does not unpack, or pairs a certified grid with a certified packing that has never been opened together | use a certified pair |
| `READER_OUTPUT` | a decoded value cannot be represented in the output: it scales outside binary64, or a masked point reached the scaling rule | narrow the window away from the affected points, or take the source at a different resolution |
| `READER_WARMUP` | the hosted Reader's pre-refresh warm-up failed or returned an invalid result | retry. A warm-up that keeps failing is a deployment fault rather than a recipe one |
| `SANDBOX_MEMORY_BOUNDARY` | no cgroup to bound the Reader child | the run reached a shape that cannot host a Reader |
| `SANDBOX_OS_BOUNDARY` | the worker sandbox has no route on this platform | run it hosted |
| `MEDIA_TYPE` | the response media type is not one this family accepts | pin the version whose accepted list carries it |

Every source receipt records the source, the byte digest, the row digest, the Reader coordinate and the settings digest.

Refusal codes for the rest of a run are on [Run states and errors](/docs/reference/run-states-and-errors/). [Limits and ceilings](/docs/reference/limits/) collects the numeric bounds.