mostlyright.core.validator

mostlyright.core.validator

DataFrame validator with source-identity enforcement.

The Validator is the source-identity invariant gate: every canonical-schema DataFrame flowing through mostlyright must carry df.attrs["source"], and that source must match the schema’s registered canonical source unless the caller explicitly opts out via allow_source_drift.

This is the invariant that prevents the silent train/infer mismatch research() Mode 2 is designed against. When a parquet cache file is loaded the next session and the saved source differs from what the schema expects, the user learns immediately (loud SourceMismatchError) instead of finding out their model trained on AWC data and inferred on IEM.

Engine choice (D-02 — decided in Phase 2 Wave 1 Task 1.0 spike): jsonschema 4.x. Per-column dtype validation is done in pandas-native (faster, no serialization roundtrip); enum-value checks use jsonschema directly so the error format is consistent with future MCP wire serialization.

See docs/design.md §H (test bar) + §J (allow_source_drift semantics).

Functions

register_schema(schema_cls)	Register a Schema subclass by its schema_id.
validate_dataframe(df, schema_id, *[, …])	Validate a DataFrame against the named canonical schema.

mostlyright.core.validator.validate_dataframe(df, schema_id, , allow_source_drift=None)

Validate a DataFrame against the named canonical schema.

Phase 6 W0-T4: accepts either a raw pd.DataFrame (legacy v0.1.0 contract — must carry df.attrs["source"] / df.attrs["retrieved_at"]) or a MostlyRightResult wrapper (v0.2+ contract — provenance travels on the wrapper). When passed a wrapper, the validator unwraps to pandas via MostlyRightResult.legacy_df_with_attrs() and proceeds with the same logic so the four checks below run byte-identically for both shapes.

The Validator runs four checks, in order:

1. Source-identity invariant. Reads df.attrs["source"] and compares against schema_cls._registered_source. If they differ and allow_source_drift is None, raises SourceMismatchError. If allow_source_drift is supplied, the drift is permitted and the audit log records the reason.
2. Required-column presence. Non-nullable columns must be present in df.columns.
3. Per-column dtype. Dispatched on ColumnSpec.dtype.
4. Enum value membership. dtype="enum" columns must have values in ColumnSpec.enum_values. Sample of violating values capped at 10.

• Parameters:
  • df (DataFrame | MostlyRightResult) – The DataFrame to validate. Must carry df.attrs["source"], OR a MostlyRightResult wrapper whose source / retrieved_at populate the equivalent attrs on unwrap.
  • schema_id (str) – Canonical schema ID (e.g. "schema.observation.v1").
  • allow_source_drift (str | None) – Reason string. If supplied, source mismatch is allowed; audit log records the reason.
• Return type: SchemaRegistration
• Returns: A SchemaRegistration recording the validation. Carries the full audit log (registered event always; source_drift_allowed event when allow_source_drift is supplied).
• Raises:
  • SchemaValidationError – column / dtype / enum / null-sentinel violations.
  • SourceMismatchError – source identity violated without opt-out.

Examples

The source-identity invariant fires when df.attrs["source"] differs from the schema’s registered canonical source (here schema.observation.v1 is registered to iem.archive):

>>> import pandas as pd
>>> from mostlyright.core import SourceMismatchError, validate_dataframe
>>> df = pd.DataFrame({"date": pd.to_datetime(["2025-01-06"])})
>>> df.attrs["source"] = "awc.live"
>>> try:
...     validate_dataframe(df, "schema.observation.v1")
... except SourceMismatchError as err:
...     print(err.data_source, "!=", err.schema_source)
awc.live != iem.archive

A full passing example requires the canonical column set; see packages/core/tests/core/test_validator.py for end-to-end fixtures.