mostlyright.core.schemas.satellite_patch

Satellite crop-tier schema (schema.satellite.patch.v1).

This module defines:

validate_satellite_patch_site_id : The site-identity check. ColumnSpec supports only dtype/nullable/enum and no regex, so the site-id alphabet cannot live as a column pattern; it is enforced by this function at the record-build site, the same way mostlyright.core.schemas.satellite.validate_satellite_station() enforces the scalar schema’s identity.

PATCH_PAYLOAD_VARIABLES : The crop payload registry: one entry per persisted (product_code, variable) pair, each carrying the variable’s native on-disk storage width and the source object that width was read from.

SatellitePatchSchema : The schema.satellite.patch.v1 shape contract — wide packed native-dtype payload columns, their per-row decode companions, per-row window metadata, an explicit 3-D payload shape and pressure axis, and an entitlement tier field.

decode_packed_value / patch_payload_index : The decode and addressing contracts, expressed as executable arithmetic rather than prose, so a reader does not have to reconstruct them from the writer.

These rows exist only in the data path. There is no route, no mostlyright.* verb, no TypeScript equivalent, and no key issuance: tier makes the rows ready for entitlement checks, and nothing reads it yet.

Storage contract — values are stored packed

A crop payload holds the raw integers exactly as the source variable holds them. A packed LVT of 32768 is stored as 32768, not as the 242.5 K it means. Storing packed is what preserves the native-width saving: a float32 cast measures +6% on a uint16 payload and 8x on an int8 categorical, and the wide one-list-column-per-variable layout measures 27 to 36 percent smaller than a long layout. Storing decoded values would force float columns and give up that saving.

Because a packed value is not a physical value, the decode inputs travel with the row. They are per-file attributes, not registry constants — the same variable ships different scale_factor and units across eras — so every payload variable carries five per-row companions: {column}_scale_factor, {column}_add_offset, {column}_fill_raw, {column}_unsigned, and {column}_units.

Decode order, matching _goes_extract._apply_scale_offset:

1. Recognise fills against the packed value first: raw == {column}_fill_raw means missing. Fills stay packed inside the payload list rather than becoming nulls — a null element inside a list<uint16> would force a widening or a parallel mask column.
2. Then apply the _Unsigned reinterpretation when {column}_unsigned is true (a signed raw is reread as the same bits unsigned: an int16 -31738 becomes 33798).
3. Then physical = raw * scale_factor + add_offset. A null scale_factor means the raw is already the value (categoricals, quality flags).

The declared element width is the width the source variable uses on disk, read from real objects, never a logical width inferred after reinterpretation. Where an older era stored a variable signed with _Unsigned='true' and the current era stores it unsigned outright, the two are the same 16 bits, so the packed value round-trips bit-exactly under the declared width and {column}_unsigned records which convention the source object used.

Payload order — C-order, (level, row, col)

A 2-D variable’s payload has window_h * window_w elements in (row, col) C-order. A 3-D profile variable’s payload has n_levels * window_h * window_w elements in (level, row, col) C-order, so:

index = ((level * window_h) + row) * window_w + col

pressure_levels_hpa carries exactly n_levels values in the same order the payload’s level axis is flattened in. n_levels and pressure_levels_hpa are null for a 2-D variable.

Which variables are persisted — primary keys plus deduplicated DQF companions

Quality flags are not registry keys: dqf_variable is a field on ProductVariable. The persisted set is therefore the primary keys of the GOES registry unioned with the Himawari registry, plus their product-qualified dqf_variable companions, deduplicated by that same union — so ACM/BCM/Cloud_Probabilities collapse to one ("ABI-L2-ACMC", "DQF") companion while ABI-L2-AODC contributes both DQF and AE_DQF. Product-qualifying the companion keeps ABI-L2-TPWC’s DQF_Overall from colliding with ABI-L2-LVMPC’s.

The registry is pure data: packages/core gains no weather import at runtime, and no Arrow or parquet import at all (the writer lives in services/). tests/test_satellite_patch_registry_drift.py imports both extractor registries behind the [satellite] skip guard and pins this mapping against them, along with the observed on-disk dtype per entry.

Module Attributes

Attribute	Description
SATELLITE_PATCH_TIER_VALUES	Entitlement classes carried in the data path.
PATCH_DECODE_COMPANION_SUFFIXES	The five per-row decode companions every payload column carries.
SATELLITE_PATCH_PLATFORM_VALUES	The satellite platforms a crop row can come from — the same enum the scalar schema.satellite.v1 carries, so a crop row and a pixel row name the same platform with the same string.

Functions

Function	Description
decode_packed_value(raw, *, storage_dtype[, …])	Recover the physical value of one packed payload element.
patch_payload_column(product_code, variable)	The payload column name for one (product_code, variable) pair.
patch_payload_index(level, row, col, *, …)	The flat payload index of (level, row, col) in C-order.
validate_satellite_patch_site_id(site_id)	Validate a crop row’s site identity against the widened alphabet.

Classes

Class	Description
PatchPayloadVariable(product_code, variable, …)	One persisted crop-payload variable.
SatellitePatchSchema()	schema.satellite.patch.v1 — wide packed native-dtype crop rows.

mostlyright.core.schemas.satellite_patch.PATCH_DECODE_COMPANION_SUFFIXES : tuple[str, …]

The five per-row decode companions every payload column carries. Generated mechanically from PATCH_PAYLOAD_VARIABLES, so a payload column cannot come into existence without its decode contract.

class mostlyright.core.schemas.satellite_patch.PatchPayloadVariable(product_code, variable, storage_dtype, is_dqf_companion, is_3d_profile, evidence)

Bases: object

One persisted crop-payload variable.

• Parameters:
  • product_code (str)
  • variable (str)
  • storage_dtype (str)
  • is_dqf_companion (bool)
  • is_3d_profile (bool)
  • evidence (str)

product_code

The source product (e.g. "ABI-L2-ACMC").

variable

The source variable name as it appears in the NetCDF object.

storage_dtype

The variable’s native on-disk width — a canonical scalar dtype tag. Read from a real source object, never inferred and never defaulted.

is_dqf_companion

True when this entry is a product-qualified dqf_variable companion rather than a primary registry key.

is_3d_profile

True when the payload carries one value per pressure level per pixel (n_levels and pressure_levels_hpa populated).

evidence

Where storage_dtype was observed — a source object key or prefix. An entry with no evidence is an error, not a default.

evidence: str

is_3d_profile: bool

is_dqf_companion: bool

product_code: str

storage_dtype: str

variable: str

mostlyright.core.schemas.satellite_patch.SATELLITE_PATCH_PLATFORM_VALUES : tuple[str, …]

The satellite platforms a crop row can come from — the same enum the scalar schema.satellite.v1 carries, so a crop row and a pixel row name the same platform with the same string.

mostlyright.core.schemas.satellite_patch.SATELLITE_PATCH_TIER_VALUES : tuple[str, …]

Entitlement classes carried in the data path. Nothing reads this column yet: there is no control plane, no key issuance, and no enforcement. It exists so the rows are ready for entitlement checks without rewriting every partition later.

open — redistributable with no entitlement (NODD is a US-government : work, public domain under 17 U.S.C. §105).

entitled — served only to an entitled key. internal — never served; the source’s licence forbids redistribution.

class mostlyright.core.schemas.satellite_patch.SatellitePatchSchema

Bases: Schema

schema.satellite.patch.v1 — wide packed native-dtype crop rows.

One row per (site_id, satellite, product, scan_start_utc). Each row carries the crop window for that product’s variables: one list column per payload-registry variable at the source’s native storage width, its five per-row decode companions, the resolved window geometry, and — for a 3-D profile product — the level count and the pressure axis.

A row populates only the payload columns of its own product; every other payload column is null. That wide layout measures 27 to 36 percent smaller than a long one, and an all-null parquet column is nearly free.

This schema is deliberately separate from schema.satellite.v1: the settlement-feeding weather/satellite/ partitions stay byte-identical, and nothing here can reach the label path.

columns: ClassVar[list[ColumnSpec]]

schema_id: ClassVar[str]

mostlyright.core.schemas.satellite_patch.decode_packed_value(raw, , storage_dtype, scale_factor=None, add_offset=None, fill_raw=None, unsigned=False)

Recover the physical value of one packed payload element.

The executable form of the module docstring’s decode contract, in the same order _goes_extract._apply_scale_offset applies it:

1. raw == fill_raw (compared on the packed value, before any reinterpretation) returns None — the element is missing.
2. unsigned reinterprets a signed raw as the same bits unsigned.
3. physical = raw * scale_factor + add_offset; a null scale_factor means the raw is already the value.

• Raises: ValueError – if storage_dtype is not an integer width (only packed integer payloads have a decode contract).
• Return type: float | None
• Parameters:
  • raw (int)
  • storage_dtype (str)
  • scale_factor (float | None)
  • add_offset (float | None)
  • fill_raw (int | None)
  • unsigned (bool)

mostlyright.core.schemas.satellite_patch.patch_payload_column(product_code, variable)

The payload column name for one (product_code, variable) pair.

Product-qualified and deterministic: ("ABI-L2-ACMC", "DQF") becomes "abi_l2_acmc__dqf", so ABI-L2-TPWC’s DQF_Overall and ABI-L2-LVMPC’s cannot collide.

• Return type: str
• Parameters:
  • product_code (str)
  • variable (str)

mostlyright.core.schemas.satellite_patch.patch_payload_index(level, row, col, , window_h, window_w)

The flat payload index of (level, row, col) in C-order.

index = ((level * window_h) + row) * window_w + col. A 2-D variable is the level == 0 slice, so its payload length is window_h * window_w.

• Return type: int
• Parameters:
  • level (int)
  • row (int)
  • col (int)
  • window_h (int)
  • window_w (int)

mostlyright.core.schemas.satellite_patch.validate_satellite_patch_site_id(site_id)

Validate a crop row’s site identity against the widened alphabet.

Returns the value unchanged when valid (so call sites can use it inline). Raises SchemaValidationError for anything outside ^[A-Z0-9]{4,12}$ — a lowercase id, a separator-bearing id, or an id shorter than 4 / longer than 12 characters must fail loudly rather than silently produce un-joinable rows or an unsafe partition path.

• Return type: str
• Parameters: site_id (str)