Source identity

Three providers carry the same METAR: AWC for the live feed, IEM ASOS for the indexed archive, GHCNh for the long historical tail. They publish on different cadences with different corrections, so two strategies that look identical on paper can diverge in production because one trained on IEM-as-of-yesterday and the other infers from AWC-as-of-right-now.

mostlyrightmd makes the choice explicit through one kwarg contract. source= always means provenance — WHO produced the row — and nothing else. The default source-blind path fuses across providers for a single settlement-reproduction view; a source= pin selects a single named provenance. Both stamp the source identity on every row so a fused training set cannot silently feed a source-pinned inference path — or vice versa — without raising.

No more numbered modes

Earlier releases described this as “Mode 1” (fused) vs “Mode 2” (single-source). That vocabulary is retired: what used to be a numbered “source-pinned mode” is just a source= pin, and what used to be the numbered “fused mode” is the default source-blind path. The kwargs say exactly what they mean. See the v1.13 migration guide.

The fused default — source-blind

The default dataset() call returns fused output: one row per LST settlement date with obs_high_f / obs_low_f computed across the merged METAR feed. When the fetched feeds overlap, a deterministic client-side dedup tiebreak runs in your process to keep one row, with order AWC > IEM > GHCNh and first-seen kept at equal rank. This is the right choice for settlement reproduction. source=None (the default) means best available.

Python

import mostlyright

df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12")
# obs_high_f, obs_low_f come from the AWC > IEM > GHCNh fused merge

TypeScript

import { dataset } from "mostlyright";

const rows = await dataset("KNYC", "2025-01-06", "2025-01-12");
// obs_high_f, obs_low_f come from the AWC > IEM > GHCNh fused merge

The fused path is the right choice when:

• You are reproducing a historical Kalshi settlement.
• You want maximum coverage (GHCNh fills gaps where IEM ASOS dropped a tick).
• You are training a model on the same fused tape every other production model trained on.

The source= pin — single provenance

For strategies where source-identity matters at trade time, pin the source. A source= pin selects raw observation rows from a single named feed. On obs(), pass granularity="observation" for the per-report rows.

Python

from mostlyright.weather import obs

df = obs(
    "KNYC", "2025-01-06", "2025-01-12",
    source="iem", granularity="observation",
)
# Every row's `source` is the truthful parser-emitted tag "iem"

TypeScript

import { obs } from "@mostlyrightmd/weather";

const rows = await obs("KNYC", "2025-01-06", "2025-01-12", {
  source: "iem",
  granularity: "observation", // TS default is already "observation"
});
// Every row's `source` is the truthful parser-emitted tag "iem"

The source vocabulary:

Source	Description
iem	IEM ASOS (parser-emitted tag, archive + live both)
iem.archive	IEM ASOS archive (canonical identity for backfilled rows)
iem.live	IEM ASOS live tick
awc	AWC METAR (parser-emitted tag)
awc.live	AWC live tick
ghcnh	NCEI GHCNh
ghcnh.archive	NCEI GHCNh archive (canonical identity)

A pin accepts either form on input. The per-row source overlay column carries the parser-emitted tag (iem, awc, ghcnh) so downstream validators see the truthful provenance, not a rewritten alias.

Passing a source the vertical does not recognize raises a loud typed error immediately, before any network call — never a silent fallback to a different provider.

Retrain event

A source= pin is a retrain event, not a find-replace. Pinning source= takes the source-isolated exact-window fetch path, which returns the FULL single-source coverage before the fusion merge. That is a different row composition than the fused default (which dedups across providers). If you pin a feature that previously used the fused default, retrain the model on the pinned data — do not assume byte-stability across the switch. See the v1.13 migration guide.

delivery= is a separate axis

source= (provenance) and delivery= (where the computation runs — local default, or the opt-in hosted seam) are strictly orthogonal. Hosted rows are byte-identical to the local path. A vertical that overloads source= to mean transport is a bug.

axis	question it answers	example values
source=	WHO produced it	None, "awc", "iem", "noaa_goes"
delivery=	WHERE it is computed	"local", "hosted"

Frame-vs-row identity

There are two source tags, and they mean different things:

• Per-row source — the truthful, durable parser/provider tag on each row (awc / iem / ghcnh). It survives every DataFrame operation because it is a real column.
• Frame-level df.attrs["source"] — a pandas-experimental, fragile tag on the whole frame. attrs is dropped by concat / merge / groupby, so treat it as advisory, not load-bearing. (TypeScript carries the equivalent identity as a non-enumerable frameSource property on the returned array.)

The fused observation frame carries the frame tag merged.live_v1 — a distinct merged-policy identity, not any single provider’s name. merged.live_v1 is rejected by every single-source pinned schema (schema.observation.v1 raises SourceMismatchError) and is accepted only by schema.observation.merged.v1, which validates the per-row source as membership in {awc, iem, ghcnh}. This is what stops a fused frame from silently passing through a pinned-source validator.

SourceMismatchError — the loud guard

The validator stamps the schema’s expected source at registration time and compares every DataFrame’s carried identity against it. If you build a training set from iem and then accidentally route the result through a schema whose registered source is ghcnh, the validator raises SourceMismatchError instead of silently joining rows that should never have met.

Python

from mostlyright.core import validate_dataframe, SourceMismatchError
from mostlyright.weather import obs

df = obs("KNYC", "2025-01-06", "2025-01-12",
         source="iem", granularity="observation")

# Validator pins the expected source from the schema:
try:
    result = validate_dataframe(df, schema_id="schema.observation.v1")
except SourceMismatchError as err:
    print(err.schema_source)   # what the schema expected
    print(err.data_source)     # what the DataFrame actually carries
    print(err.role)            # "observations" | "forecasts" | "settlement"

TypeScript

import { validateRows } from "@mostlyrightmd/core/validator";
import { SourceMismatchError } from "@mostlyrightmd/core";
import { obs } from "@mostlyrightmd/weather";

const rows = await obs("KNYC", "2025-01-06", "2025-01-12", { source: "iem" });

try {
  const result = validateRows(rows, "schema.observation.v1");
} catch (err) {
  if (err instanceof SourceMismatchError) {
    console.log(err.schemaSource);  // what the schema expected
    console.log(err.dataSource);    // what the rows actually carry
    console.log(err.role);          // "observations" | "forecasts" | "settlement"
  }
}

To opt out (e.g. for a one-time manual backfill that intentionally crosses sources), pass allow_source_drift="<reason>":

result = validate_dataframe(
    df,
    schema_id="schema.observation.v1",
    allow_source_drift="manual backfill from GHCNh into the IEM schema",
)
# audit_log carries both "registered" + "source_drift_allowed" entries

The opt-out is intentionally verbose: the reason string is required, and it shows up in the audit log so a code review can spot it.

Train / trade symmetry

The contract exists so that a feature backtests the same way it trades. Pin the same provenance on both sides of the seam:

import mostlyright
from mostlyright.weather import obs
from mostlyright.core import validate_dataframe

# TRAIN — pinned historical, per-report grain
train = obs("KNYC", "2024-01-01", "2024-12-31",
            source="awc", granularity="observation")
validate_dataframe(train, schema_id="schema.observation.v1")

# TRADE — the SAME provenance, live
tick = mostlyright.live.stream("KNYC", source="awc")

Because both sides pin source="awc", the source identity is consistent across the train/infer seam. If you accidentally route a fused (merged.live_v1) frame through a single-source pinned schema, validate_dataframe raises SourceMismatchError at load time — loud, before the mismatch can silently corrupt a model.

See also

• Temporal safety — KnowledgeView and assert_no_leakage.
• → v1.13 migration — dataset(), obs() grain, and the source-pin retrain event.
• Cache directory — moving from ~/.tradewinds/ to ~/.mostlyright/.