mostlyright.core

mostlyright.core

mostlyright.core — temporal safety + schema + format primitives.

The architectural spine of the mostlyright SDK. Originally ported from the mostlyright-mcp wave-1-core branch (lift provenance preserved in per-module docstrings); promoted from the _v02 reference into the canonical namespace in Phase 2 of the v0.1.0 plan.

Sub-modules:

• mostlyright.core.exceptions — structured exception hierarchy (MostlyRightError base + 6 subclasses + JSON-safe to_dict). MostlyRightMCPError remains importable as a deprecation alias.
• mostlyright.core.temporal — UTC-aware TimePoint wrapper; KnowledgeView + LeakageDetector (Phase 2 Wave 2).
• mostlyright.core.schema — declarative Schema framework with audit-log seam used by the source-identity Validator.
• mostlyright.core.schemas — three canonical schema instances (observation, forecast, settlement).
• mostlyright.core.formats — five lossless format serializers (dataframe / json / parquet / toon / csv).

class mostlyright.core.ColumnSpec(name, dtype, units, nullable, enum_values=None, notes=”)

Bases: object

Description of a single column in a Schema.

• Parameters:
  • name (str)
  • dtype (str)
  • units (str | None)
  • nullable (bool)
  • enum_values (tuple [str , … ] | None)
  • notes (str)

name

Column name as it appears in the canonical (metric) projection.

dtype

One of the canonical dtype tags. "enum" requires enum_values to be populated.

units

Free-form unit label ("celsius", "meters", "kt") or None for dimensionless columns.

nullable

Whether the column may contain NaN/NULL values.

enum_values

Allowed values when dtype == "enum". Must be None for non-enum columns.

notes

Free-form documentation surfaced in catalog metadata.

dtype : str

enum_values : tuple[str, ...] | None = None

name : str

notes : str = ”

nullable : bool

units : str | None

class mostlyright.core.KnowledgeView(df, as_of)

Bases: object

A filtered, knowledge-time-bounded view over a DataFrame.

Construction validates the shape of the input DataFrame. After construction, dataframe() returns a defensive copy of the rows where knowledge_time <= as_of.

Examples

Build a 3-row DataFrame with a tz-aware UTC knowledge_time column and view only the rows knowable as of 2025-01-02T12:00:00Z:

>>> import pandas as pd
>>> from mostlyright.core import KnowledgeView, TimePoint
>>> df = pd.DataFrame({
...     "knowledge_time": pd.to_datetime([
...         "2025-01-01T00:00:00Z",
...         "2025-01-02T00:00:00Z",
...         "2025-01-03T00:00:00Z",
...     ], utc=True),
...     "value": [10, 20, 30],
... })
>>> view = KnowledgeView(df, TimePoint("2025-01-02T12:00:00Z"))
>>> len(view.dataframe())
2

• Parameters:
  • df (pd.DataFrame | MostlyRightResult)
  • as_of (TimePoint)

property as_of : TimePoint

The as-of cutoff supplied at construction.

dataframe()

Return a defensive copy of the rows where knowledge_time <= as_of.

• Return type: DataFrame

class mostlyright.core.LeakageDetector(as_of)

Bases: object

Convenience wrapper for repeated detection against a fixed as_of.

• Parameters: as_of (TimePoint)

property as_of : TimePoint

check(df)

Run assert_no_leakage() against the bound as_of.

Accepts either a raw DataFrame or a MostlyRightResult wrapper (unwrapped inside assert_no_leakage()).

• Return type: None
• Parameters: df (DataFrame | MostlyRightResult)

check_issued_at(df)

Raise IssuedAtMissingError if any row has null issued_at.

Phase 20 OM-04 extension. Independent of as_of — the bound cutoff is irrelevant when the row carries no model-run time at all.

• Return type: None
• Parameters: df (DataFrame | MostlyRightResult)

exception mostlyright.core.LeakageError(message=”, , as_of, violating_count, sample_violations=None, source=None, request_id=None, error_code=None)

Bases: MostlyRightError

Temporal leakage detected — at least one row has knowledge_time greater than the asserted as_of cutoff. Carries the count and a small sample of violating rows for actionable surfacing.

• Parameters:
  • message (str)
  • as_of (str)
  • violating_count (int)
  • sample_violations (list [dict [str , Any ] ] | None)
  • source (str | None)
  • request_id (str | None)
  • error_code (str | None)
• Return type: None

default_error_code : str = ‘LEAKAGE_DETECTED’

Subclass override — the stable string enum surfaced via error_code.

exception mostlyright.core.MostlyRightError(message=”, , error_code=None, source=None, request_id=None)

Bases: Exception

Base class for all mostlyright structured errors.

error_code is a stable enum (e.g. "SOURCE_UNAVAILABLE") used by callers / agents to branch on without parsing message text. source is the source ID involved ("iem.archive" etc.) when applicable, and request_id correlates an MCP JSON-RPC request when applicable.

• Parameters:
  • message (str)
  • error_code (str | None)
  • source (str | None)
  • request_id (str | None)
• Return type: None

default_error_code : str = ‘MOSTLYRIGHT_ERROR’

Subclass override — the stable string enum surfaced via error_code.

to_dict()

Return a JSON-safe dict suitable for MCP error.data.

• Return type: dict[str, Any]

class mostlyright.core.MostlyRightResult(frame, source, retrieved_at, schema_id=None, qc=None, data_version=None)

Bases: object

Backend-neutral provenance wrapper for a DataFrame-returning call.

Both pandas-backend and polars-backend adapters return the same wrapper shape. frame holds the native frame; the remaining fields carry the provenance that v0.1.0 used to stamp on df.attrs.

• Parameters:
  • frame (FrameLike)
  • source (str)
  • retrieved_at (datetime)
  • schema_id (str | None)
  • qc (dict [str , Any ] | None)
  • data_version (DataVersion | None)

frame

The underlying DataFrame (pandas OR polars). Optional polars type is type-hinted via TYPE_CHECKING so default install does not require polars.

source

The canonical source identifier (e.g. "iem.live", "awc.live", "noaa_bdp"). Mirrors the v0.1.0 df.attrs["source"] contract.

retrieved_at

UTC timestamp of the fetch. MUST be tz-aware.

schema_id

Optional canonical schema ID (e.g. "schema.observation.v1"). None if the call returns heterogeneous rows (e.g. research() pairs).

qc

Optional QC summary (rules_fired counts, sidecar_paths) if the caller invoked qc=True. Mirrors df.attrs["qc"].

data_version

Optional DataVersion token for reproducibility.

Examples

>>> import pandas as pd
>>> from datetime import datetime, timezone
>>> from mostlyright.core.result import MostlyRightResult
>>> df = pd.DataFrame({"date": ["2025-01-01"], "value": [42]})
>>> result = MostlyRightResult(
...     frame=df,
...     source="iem.live",
...     retrieved_at=datetime(2025, 1, 1, tzinfo=timezone.utc),
... )
>>> result.source
'iem.live'
>>> result.frame_as_pandas().iloc[0]["value"]
42

data_version : DataVersion | None = None

frame : DataFrame | DataFrame

frame_as_pandas()

Return the underlying frame as a pandas DataFrame.

Pandas frames pass through unchanged. Polars frames are converted via pl.DataFrame.to_pandas(). Parity-locked modules call this before running their pandas-only pipelines (Phase 6 PLAN.md W4-T1).

Polars→pandas conversion may shift datetime resolution (us → ns on the v0.1.0 contract path) and may change nullable-int storage. Callers that need byte-equivalent round-trip across backends MUST consult the documented coercion rules in tests/fixtures/parity/coerce_pd3.py.

• Return type: DataFrame

legacy_df_with_attrs()

Return the wrapped pandas DataFrame with df.attrs populated.

Mirrors the v0.1.0 df.attrs-stamped shape that catalog adapters produced: attrs["source"], attrs["retrieved_at"] (ISO string), and attrs["qc"] / attrs["data_version"] when set. For callers that still consume df.attrs["source"] directly instead of the wrapper, this method bridges the legacy expectation for one release cycle (strict deprecation lands in v0.3).

• Raises: TypeError – if the wrapped frame is polars — the legacy shape was pandas-only by definition.
• Return type: DataFrame

qc : dict[str, Any] | None = None

retrieved_at : datetime

schema_id : str | None = None

source : str

to_dict()

JSON-safe dict representation for v0.2 MCP JSON-RPC serialization.

Excludes the frame body — callers that need to ship rows over MCP should serialize the frame via mostlyright.core.formats.* writers and attach the provenance via this method’s output.

• Return type: dict[str, Any]

exception mostlyright.core.PayloadTooLargeError(message=”, , declared_size, limit, accepted_modes=None, source=None, request_id=None, error_code=None)

Bases: MostlyRightError

The MCP server rejected an inline payload whose declared size exceeded the cap. accepted_modes advertises the alternatives (e.g. file-path mode per design.md §Q).

• Parameters:
  • message (str)
  • declared_size (int)
  • limit (int)
  • accepted_modes (list [str ] | None)
  • source (str | None)
  • request_id (str | None)
  • error_code (str)
• Return type: None

default_error_code : str = ‘PAYLOAD_TOO_LARGE’

Subclass override — the stable string enum surfaced via error_code.

class mostlyright.core.Schema

Bases: object

Declarative schema base class.

Subclasses declare:

• schema_id — stable identifier ("schema.observation.v1").
• COLUMNS — ordered list of ColumnSpec.
• IMPERIAL_RENAMES (optional) — metric → imperial column-name map; columns not mentioned keep their metric name.

The base class is intentionally light: no I/O, no DataFrame coupling. Validator and adapter layers consume COLUMNS to perform their work; Schema itself only holds the contract.

COLUMNS : ClassVar[list[ColumnSpec]] = []

IMPERIAL_RENAMES : ClassVar[dict[str, str]] = {}

classmethod column(name)

Look up a ColumnSpec by its (metric) name.

• Raises: KeyError – if name does not name a declared column.
• Return type: ColumnSpec
• Parameters: name (str)

classmethod column_names(mode=‘metric’)

Return the column names in declaration order for the given mode.

mode="metric" returns the canonical names declared in COLUMNS. mode="imperial" applies IMPERIAL_RENAMES (columns absent from the rename map keep their metric name).

• Raises: ValueError – if mode is not "metric" or "imperial".
• Return type: list[str]
• Parameters: mode (str)

classmethod from_dataframe(df, source, retrieved_at)

Infer a schema from a DataFrame (BYO data path). Deferred to v0.1.1.

Per docs/design.md Premise 5 (and §”Layer responsibilities”): schemas can be constructed declaratively (subclasses of Schema) OR inferred from a sample DataFrame. The declarative path ships in v0.1.0; the inference path (this method) ships in v0.1.1.

Track at:

• Return type: Schema
• Parameters:
  • df (DataFrame)
  • source (str)
  • retrieved_at (datetime)

classmethod register(source, retrieved_at, rows)

Register this schema against a concrete pull.

Records the source and retrieved_at provenance and produces a fresh SchemaRegistration carrying the audit log. The registered event is appended automatically.

NOTE

v0.1.0 is a per-call factory. Every call returns a brand-new SchemaRegistration with an independent audit log; there is no cross-call registry that deduplicates by (schema_id, source) or merges audit events across pulls.

Design §BB.4 describes an audit log that persists with the schema across calls; that persistence model lands in v0.1.1 alongside the catalog adapter, which owns the (schema, source) keying and on-disk storage decisions (parquet metadata, JSON sidecar). Callers in v0.1.0 that need a long-lived registration must hold onto the returned object themselves.

Track at:

• Parameters:
  • source (str) – Source identity in "<source>.<endpoint>" form ("iem.archive", "awc.live").
  • retrieved_at (datetime) – Wall-clock timestamp of the pull that produced the rows. MUST be timezone-aware.
  • rows (int) – Row count after schema validation.
• Raises:
  • TypeError – if source is not str, retrieved_at is not a datetime, or rows is not int.
  • ValueError – if source is empty, retrieved_at is naive, or rows is negative.
• Return type: SchemaRegistration

schema_id : ClassVar[str] = ”

class mostlyright.core.SchemaRegistration(schema, source, retrieved_at_min, retrieved_at_max, rows, _audit=)

Bases: object

Audit-trail container produced by Schema.register(...).

Records the (source, retrieved_at range, row count) provenance of a training-time pull and accumulates an append-only audit log. The audit log captures source-drift opt-outs and reproducibility-audit outcomes per docs/design.md §BB.4.

Validator wires audit events in via _append_audit(); this module intentionally does not import Validator (it is implemented elsewhere).

Audit-log seam (contract — read this). The single-underscore prefix on _audit and _append_audit() denotes module-private with one cross-module writer: the Validator (a sibling module) calls _append_audit to record source_drift_allowed and temporal_drift_audit events. Any caller OTHER than Schema.register() or the Validator that invokes _append_audit — or any direct mutation of _audit — is a contract violation: the audit log is meant to be append-only and stamped by trusted writers only. The list is intentionally not frozen so the Validator can append without indirection; this is the explicit seam, documented here so code review can catch unauthorised callers.

Deferred to v0.1.1: volatility_warning (per design §B/§P, surfaced by catalog_search and flagged on schemas whose registered rows fall within the 30-day volatile window of iem.archive) will land as an attribute on this dataclass populated by the catalog adapter at registration time. v0.1.0 carries no such attribute.

• Parameters:
  • schema (type [Schema ])
  • source (str)
  • retrieved_at_min (datetime)
  • retrieved_at_max (datetime)
  • rows (int)
  • _audit (list [dict [str , Any ] ])

audit_log()

Return a defensive copy of the chronologically-ordered audit log.

Both the outer list and each inner entry are shallow-copied so callers cannot mutate stored entries (e.g. reg.audit_log()[0]["event"] = "tampered" is a no-op against the underlying log). The list contains dict entries with at minimum event and ts (ISO-8601 UTC) keys. See _append_audit() for the supported event vocabulary.

• Return type: list[dict[str, Any]]

retrieved_at_max : datetime

retrieved_at_min : datetime

rows : int

schema : type[Schema]

source : str

exception mostlyright.core.SchemaValidationError(message=”, , schema_id, violations=None, quarantine_count=0, sample_violations=None, source=None, request_id=None, error_code=None)

Bases: MostlyRightError

A DataFrame failed schema validation. Carries the full violation list (capped at 10,000 — surplus written to file via §Q file-path mode by the SDK) and a small inline sample for MCP wire serialization (≤10 entries).

• Parameters:
  • message (str)
  • schema_id (str)
  • violations (list [dict [str , Any ] ] | None)
  • quarantine_count (int)
  • sample_violations (list [dict [str , Any ] ] | None)
  • source (str | None)
  • request_id (str | None)
  • error_code (str)
• Return type: None

default_error_code : str = ‘SCHEMA_VALIDATION_FAILED’

Subclass override — the stable string enum surfaced via error_code.

exception mostlyright.core.SourceMismatchError(message=”, , schema_source, data_source, role=None, catalog_warning=None, source=None, request_id=None, error_code=None)

Bases: MostlyRightError

The data’s source does not match the schema’s registered source, and the caller did not opt out via allow_source_drift. role (if set) identifies which leg of a pull_pairs request mismatched and uses the canonical long form: "observations" / "forecasts" / "settlement".

• Parameters:
  • message (str)
  • schema_source (str)
  • data_source (str)
  • role (str | None)
  • catalog_warning (str | None)
  • source (str | None)
  • request_id (str | None)
  • error_code (str)
• Return type: None

VALID_ROLES = frozenset({‘forecasts’, ‘observations’, ‘settlement’})

Canonical role-name vocabulary (design.md §R).

default_error_code : str = ‘SOURCE_MISMATCH’

Subclass override — the stable string enum surfaced via error_code.

exception mostlyright.core.SourceUnavailableError(message=”, , source=None, http_status=None, retryable=False, retry_after_s=None, underlying=”, url=None, request_id=None, error_code=None)

Bases: MostlyRightError

A source (HTTP endpoint, vendored parser, etc.) returned an error or was otherwise unreachable. Carries enough metadata for callers to decide whether to retry and after how long.

• Parameters:
  • message (str)
  • source (str | None)
  • http_status (int | None)
  • retryable (bool)
  • retry_after_s (float | None)
  • underlying (str)
  • url (str | None)
  • request_id (str | None)
  • error_code (str)
• Return type: None

default_error_code : str = ‘SOURCE_UNAVAILABLE’

Subclass override — the stable string enum surfaced via error_code.

exception mostlyright.core.TemporalDriftError(message=”, , schema_id, asserted_range, violating_rows, sample_violations=None, source=None, request_id=None, error_code=None)

Bases: MostlyRightError

Raised by the reproducibility audit (design.md §P) when one or more rows have retrieved_at outside the asserted range AND fall within the volatile window of now. Indicates the source materially re-amended historical rows since the schema’s registered capture.

• Parameters:
  • message (str)
  • schema_id (str)
  • asserted_range (tuple [str , str ])
  • violating_rows (int)
  • sample_violations (list [dict [str , Any ] ] | None)
  • source (str | None)
  • request_id (str | None)
  • error_code (str)
• Return type: None

default_error_code : str = ‘TEMPORAL_DRIFT’

Subclass override — the stable string enum surfaced via error_code.

class mostlyright.core.TimePoint(value)

Bases: object

A UTC-aware timestamp normalized to UTC for storage.

Construction accepts:

• datetime.datetime with tzinfo set (converted to UTC)
• pandas.Timestamp with tz set (converted to UTC)
• ISO 8601 string with a timezone (parsed and converted to UTC)

Rejects:

• naive datetime / pd.Timestamp (tzinfo / tz is None)
• date-only ISO strings (no time component, e.g. "2026-05-21")

Storage is a datetime with tzinfo=timezone.utc and microsecond precision preserved.

• Parameters: value (TimePointInput)

as_zone(tz)

Convert to a different IANA zone via zoneinfo.ZoneInfo.

Display helper only — the canonical storage stays UTC. Raises zoneinfo.ZoneInfoNotFoundError if tz is not a known IANA timezone.

• Return type: datetime
• Parameters: tz (str)

classmethod from_pandas(ts)

Construct from a tz-aware pandas.Timestamp.

Explicit alternative to passing a Timestamp to TimePoint(...) — useful when the caller wants the pandas-specific dispatch path to be unambiguous.

Raises ValueError if ts is pd.NaT (missing-data sentinel).

• Return type: TimePoint
• Parameters: ts (Timestamp)

iso()

Return an ISO 8601 UTC string.

Includes microseconds only when non-zero, matching standard datetime.isoformat() behavior (e.g. "2026-05-21T14:30:00+00:00" or "2026-05-21T14:30:00.123456+00:00").

• Return type: str

classmethod now()

Return a TimePoint for the current UTC time.

• Return type: TimePoint

to_utc()

Return the underlying UTC datetime (microseconds preserved).

• Return type: datetime

mostlyright.core.assert_no_leakage(df, as_of)

Raise LeakageError if any row’s knowledge_time > as_of.

Phase 6 W0-T6: accepts either a raw DataFrame or a MostlyRightResult; wrapped polars frames are converted to pandas via MostlyRightResult.frame_as_pandas() because the body of this function uses pd.Timestamp + iterrows semantics.

• Parameters:
  • df (DataFrame | MostlyRightResult) – DataFrame with a tz-aware UTC knowledge_time column, OR a MostlyRightResult wrapping such a frame.
  • as_of (TimePoint) – The as-of cutoff. Rows with strictly greater knowledge_time count as leakage.
• Raises:
  • SchemaValidationError – if knowledge_time is missing or not tz-aware UTC.
  • TypeError – if as_of is not a TimePoint.
  • LeakageError – if ≥1 rows have knowledge_time > as_of. Payload carries violating_count and a sample (≤10) of violations.
• Return type: None

Examples

• Return type: None
• Parameters:
  • df (DataFrame | MostlyRightResult)
  • as_of (TimePoint)

A leak-free frame passes silently:

>>> import pandas as pd
>>> from mostlyright.core import TimePoint, assert_no_leakage
>>> df = pd.DataFrame({
...     "knowledge_time": pd.to_datetime(["2025-01-01T00:00:00Z"], utc=True),
...     "value": [10],
... })
>>> assert_no_leakage(df, TimePoint("2025-01-02T00:00:00Z"))

A row past the cutoff raises LeakageError:

>>> from mostlyright.core import LeakageError
>>> leaky = pd.DataFrame({
...     "knowledge_time": pd.to_datetime(["2025-01-03T00:00:00Z"], utc=True),
...     "value": [99],
... })
>>> try:
...     assert_no_leakage(leaky, TimePoint("2025-01-02T00:00:00Z"))
... except LeakageError as err:
...     print(err.violating_count)
1

mostlyright.core.assert_source_identity(df, expected_source, , role=‘observations’)

Raise SourceMismatchError if any row’s source != expected.

A defense-in-depth per-row gate: mirrors the per-row check in mostlyright.core.validator.validate_dataframe() but at the source-pinned dispatch layer so callers get a role-flavored error message when a pinned single-source frame carries a foreign row.

• Parameters:
  • df (DataFrame) – The DataFrame to check. If it has no source column the function is a silent no-op (the Validator handles that case).
  • expected_source (str) – The source every row must carry.
  • role (str) – The role label surfaced on the raised error (default "observations").
• Raises: SourceMismatchError – when one or more rows carry a source other than expected_source.
• Return type: None

mostlyright.core.validate_dataframe(df, schema_id, , allow_source_drift=None)

Validate a DataFrame against the named canonical schema.

Phase 6 W0-T4: accepts either a raw pd.DataFrame (legacy v0.1.0 contract — must carry df.attrs["source"] / df.attrs["retrieved_at"]) or a MostlyRightResult wrapper (v0.2+ contract — provenance travels on the wrapper). When passed a wrapper, the validator unwraps to pandas via MostlyRightResult.legacy_df_with_attrs() and proceeds with the same logic so the four checks below run byte-identically for both shapes.

The Validator runs four checks, in order:

1. Source-identity invariant. Reads df.attrs["source"] and compares against schema_cls._registered_source. If they differ and allow_source_drift is None, raises SourceMismatchError. If allow_source_drift is supplied, the drift is permitted and the audit log records the reason.
2. Required-column presence. Non-nullable columns must be present in df.columns.
3. Per-column dtype. Dispatched on ColumnSpec.dtype.
4. Enum value membership. dtype="enum" columns must have values in ColumnSpec.enum_values. Sample of violating values capped at 10.

• Parameters:
  • df (DataFrame | MostlyRightResult) – The DataFrame to validate. Must carry df.attrs["source"], OR a MostlyRightResult wrapper whose source / retrieved_at populate the equivalent attrs on unwrap.
  • schema_id (str) – Canonical schema ID (e.g. "schema.observation.v1").
  • allow_source_drift (str | None) – Reason string. If supplied, source mismatch is allowed; audit log records the reason.
• Return type: SchemaRegistration
• Returns: A SchemaRegistration recording the validation. Carries the full audit log (registered event always; source_drift_allowed event when allow_source_drift is supplied).
• Raises:
  • SchemaValidationError – column / dtype / enum / null-sentinel violations.
  • SourceMismatchError – source identity violated without opt-out.

Examples

The source-identity invariant fires when df.attrs["source"] differs from the schema’s registered canonical source (here schema.observation.v1 is registered to iem.archive):

>>> import pandas as pd
>>> from mostlyright.core import SourceMismatchError, validate_dataframe
>>> df = pd.DataFrame({"date": pd.to_datetime(["2025-01-06"])})
>>> df.attrs["source"] = "awc.live"
>>> try:
...     validate_dataframe(df, "schema.observation.v1")
... except SourceMismatchError as err:
...     print(err.data_source, "!=", err.schema_source)
awc.live != iem.archive

A full passing example requires the canonical column set; see packages/core/tests/core/test_validator.py for end-to-end fixtures.

Modules

exceptions	Structured exception hierarchy for the mostlyright SDK and MCP server.
formats	Format converters for DataFrame interchange.
merge	Phase 2.1 — read-time merge policy for silver-tier observation ledgers.
result	MostlyRightResult — backend-neutral provenance wrapper.
schema	Declarative schema framework for mostlyright.
schemas	Canonical schemas shipped with mostlyright v0.1.
source_identity	Canonical home for the assert_source_identity source-pinning gate.
temporal	mostlyright.core.temporal — temporal-safety primitives.
validator	DataFrame validator with source-identity enforcement.