dataset()

dataset() is the SDK’s primary composer. One call joins multi-source observations, NWS CLI climate, optional forecasts, and optional covariates into one settlement-ready row per day. Same signature in Python and TypeScript. research() is a fully working alias bound to the same callable.

research() is still here

dataset() is the canonical composer. research() is a fully working alias with the same signature and byte-identical output. As of v1.15 they are distinct thin wrappers over one shared body (the D-22 alias split — see Choosing the label), but nothing you call today returns different data. New code should call dataset(). See the v1.15 migration guide.

Quickstart

Python

import mostlyright

df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12")
print(df.head())

TypeScript

import { dataset } from "mostlyright";

const rows = await dataset("KNYC", "2025-01-06", "2025-01-12");
console.log(rows[0]);

Output is one row per LST settlement date in [from_date, to_date], fused across AWC + IEM + GHCNh observations + NWS CLI climate.

Two layers: composer + domain tables

dataset() sits on top of the raw, source-identified domain tables. Reach for the domain tables when you want the ingredients directly; reach for dataset() when you want the SDK to own the leakage-prone day alignment for you.

Layer	Surface	Returns
Composer	dataset() / research()	one leakage-safe training row per LST settlement day
Domain table — observations	obs()	daily extremes or per-report METAR/SPECI rows
Domain table — climate	climate()	NWS CLI daily settlement labels
Domain table — forecasts	forecasts() / forecast_nwp()	per settlement-day forecast envelope

The full contract for source= / delivery= / grain across every table lives in Source identity.

Choosing the label

As of v1.15 dataset() has an explicit label axis: the label= kwarg routes which settlement target (“label”) columns the composed frame carries. The core is a venue-free alignment engine — the label you pick is what makes it settlement-ready.

label=	Label columns	Settles for	Notes
"cli"	cli_high_f, cli_low_f, cli_report_type	Kalshi NHIGH/NLOW (NWS CLI)	Today’s bytes — the byte-identical legacy climate join. The implicit default.
"daily_extremes"	daily_extremes_tmax_c, daily_extremes_tmin_c, daily_extremes_tmean_c, daily_extremes_n_obs, daily_extremes_source_tmax, daily_extremes_source_tmin	Polymarket (WU / NOAA-WRH extremes)	Whole-°C precision, station-local calendar; WR-05 low-coverage nulling (rows preserved). Works for international stations.
None	(none — features only)	nothing (pure feature table)	Works for any catalog station worldwide, no market required. The 2.0 default.
(a DataFrame)	your own columns	your own target	Bring-your-own — the SDK enforces the leakage discipline for you (see below).

Python

import mostlyright

# Kalshi / NWS CLI settlement columns (today's default output).
cli = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", label="cli")

# What Polymarket settles on — WU/NOAA-WRH daily extremes, in °C.
poly = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", label="daily_extremes")

# Pure features — no label columns; works for any catalog station worldwide.
feats = mostlyright.dataset("EGLL", "2025-01-06", "2025-01-12", label=None)

TypeScript

import { dataset } from "mostlyright";

const cli = await dataset("KNYC", "2025-01-06", "2025-01-12", { label: "cli" });
const poly = await dataset("KNYC", "2025-01-06", "2025-01-12", { label: "daily_extremes" });
const feats = await dataset("EGLL", "2025-01-06", "2025-01-12", { label: null });

The D-22 default-label bridge — pin label="cli" for byte-stability

Calling dataset() without an explicit label= still returns the cli_* columns byte-identically, but emits a one-time FutureWarning:

dataset() label default changes from 'cli' to None in 2.0; pass label='cli' to keep settlement labels or label=None to silence

The default flips to label=None (features-only) at 2.0, after a ≥2-minor bridge window. If you reproduce historical settlements, pin label="cli" explicitly — it keeps the cli_* columns forever, silences the warning, and does not depend on the pre-2.0 default. Parity byte-stability is guaranteed by that explicit pin, not by a warnings filter.

research() never warns. The legacy alias keeps its implicit cli label with no new warning until it retires at 2.0. See the v1.15 migration guide.

Bring your own label (BYO)

Pass a DataFrame instead of a recipe name and the SDK becomes your aligner — you own the label values, it owns the leakage discipline. It buckets your rows to the station’s LST settlement calendar, LEFT-joins them (rows are preserved, missing days NaN), and guards against collisions with the reserved obs_ / cli_ / wu_ / noaa_wrh_ prefixes and registered feature namespaces. A misaligned frame raises the typed LabelAlignmentError (with a stable reason enum), never silently.

import pandas as pd

# Minimum contract: a `date` column (or a datetime/date-named index).
my_labels = pd.DataFrame({
    "date": ["2025-01-06", "2025-01-07"],
    "my_target": [1, 0],
})

df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", label=my_labels)
# `my_target` is LEFT-joined by settlement day; label-less days are NaN.

Two sharp edges worth internalizing (both enforced, never silent):

• As-of / publication discipline. By default labels_as_known="settled" treats each label as final. Pass labels_as_known="publication_lag" with a published_at column and the aligner holds a day’s label out until it would have been published by that day’s market close — no published_at column raises rather than silently treating everything as settled.
• tz-aware vs tz-naive dates. A tz-aware timestamp buckets by the station settlement calendar (a 02:00Z label rolls into the prior LST day); a tz-naive date / plain ISO string is treated as an already-LST calendar date. Be explicit about which you mean.

For multi-station panels (dataset(stations=[...])) and registering your own feature columns, see Extending dataset().

The full kwarg surface

dataset(station, from_date, to_date, ...) accepts these composable kwargs (parity surface across both SDKs unless noted):

Kwarg	Default	Description
label	(implicit "cli")	The settlement target — "cli" | "daily_extremes" | None | a BYO DataFrame. See Choosing the label. Omitting it on dataset() warns (D-22).
labels_as_known	"settled"	BYO publication discipline — "settled" or "publication_lag" (needs a published_at column).
stations	None	Multi-station panel (long-format). Mutex with station / city / contract / contracts. See Extending dataset().
features	None	Explicit feature list — ["forecasts", "satellite", "cwop", "trades"] + registered contributors. Replaces the include_* flags.
include_forecast	False	Deprecated alias of features=["forecasts"]. Attach fcst_* columns. IEM MOS wired both SDKs; per-NWP-model Python only.
forecast_model	None	Single forecast model name (e.g. "gfs"). Requires include_forecast=True / features=["forecasts"].
forecast_models	None	Multi-model forecast fan-out. Requires forecasts. Mutex with forecast_model.
include_satellite	False	Deprecated alias of features=["satellite"]. LEFT-join labels-only sat_* columns. Needs the [satellite] extra. Python.
include_cwop	False	Deprecated alias of features=["cwop"]. LEFT-join labels-only cwop_* columns. Needs the [cwop] extra. Python.
qc	False	Run QC passes, surface QC flags.
tz_override	None	IANA tz override for stations not in the canonical registry.
sources	None	Multi-source subset. Mutex with source.
source	None	Single-source pin (provenance). Mutex with sources.
backend	"pandas"	Python: "pandas" or "polars". TS: accepted but "polars" raises (DataAvailabilityError).
return_type	"dataframe"	"dataframe" | "list" | "wrapper". (TS canonical "dataframe"; "frame" is a back-compat alias.)

include_* flags are deprecated

As of v1.15, include_forecast / include_satellite / include_cwop / include_trades are deprecated aliases of features=[...] — they still work byte-identically for at least two more minors, but each True flag emits a DeprecationWarning. Migrate to features=["forecasts", "satellite", ...]. contract= / contracts= / include_trades= on the core dataset() are deprecated toward the markets delegators. See the v1.15 migration guide.

as_dataframe= is deprecated

as_dataframe= still works this release (True → "dataframe", False → "list") and emits a DeprecationWarning when passed explicitly. An explicit return_type= always wins. Migrate to return_type= — see the v1.13 migration guide.

Validation — errors fire BEFORE network

Mutually-exclusive kwargs raise TypeError before any HTTP or cache I/O:

# All three raise TypeError immediately
mostlyright.dataset("KNYC", "...", "...", sources=["awc"], source="iem")       # mutex
mostlyright.dataset("KNYC", "...", "...", forecast_model="gfs", forecast_models=["gfs"])  # mutex
mostlyright.dataset("KNYC", "...", "...", forecast_model="gfs")                # missing include_forecast

// Same shape in TypeScript — TypeError, not Error
await dataset("KNYC", "...", "...", {
  sources: ["awc"],
  source: "iem", // mutex with sources
});
// → TypeError("dataset(): sources= and source= are mutually exclusive …")

Unknown options also raise TypeError:

await dataset("KNYC", "...", "...", { inclide_forecast: true });  // typo
// → TypeError("dataset(): unknown option key 'inclide_forecast' …")

Passing a source the vertical does not recognize also raises a loud typed error immediately, before any network call — never a silent fallback to a different provider.

Fused (default) vs source-pinned

Fused (default) — source=None means best available: a deterministic client-side dedup tiebreak runs in your process — AWC > IEM > GHCNh — so output rows carry the rawest source per (station, observed_at). This is the right choice for reproducing a historical settlement. The fused observation frame carries the distinct frame tag merged.live_v1.

df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12")
# obs_high_f / obs_low_f come from the AWC > IEM > GHCNh fused merge

Source-pinned (source=) — pin the provenance to one authority. source= always means WHO produced the row — never delivery, transport, or freshness. Pinning is what keeps a feature backtesting the same way it trades; the validator raises SourceMismatchError if training data and inference data disagree on source.

df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", source="iem")
# Every observation row carries source = "iem"

Retrain event

Pinning source= returns a different row composition than the fused default — the source-isolated exact-window path returns full single-source coverage before the merge, not the post-merge subset. Migrating a pinned feature is a retrain event, not a find-replace. See Source identity and the v1.13 migration guide.

Forecasts attached

df = mostlyright.dataset(
    "KNYC", "2026-05-01", "2026-05-07",
    include_forecast=True,
)
print(df[["date", "obs_high_f", "fcst_high_f", "fcst_model"]])

fcst_* columns ship populated from IEM MOS (the default). For per-model NWP fan-out (Python only):

df = mostlyright.dataset(
    "KNYC", "2026-05-01", "2026-05-07",
    include_forecast=True,
    forecast_models=["hrrr", "gfs"],
)
# Adds fcst_high_f_nwp_hrrr, fcst_high_f_nwp_gfs alongside fcst_high_f (MOS)

See Forecasts.

Covariate columns (labels-only)

include_satellite= / include_cwop= LEFT-join per-settlement-day covariate columns onto the composed frame. The label columns are byte-identical whether the flags are on or off — obs_*, cli_*, date, station never change, row count and order are unchanged, and missing covariate days are NaN (rows are never dropped or added). Defaults are off, so parity fixtures are untouched.

df = mostlyright.dataset(
    "KNYC", "2025-01-06", "2025-01-12",
    include_satellite=True,   # adds sat_* columns; needs the [satellite] extra
    include_cwop=True,        # adds cwop_* columns; needs the [cwop] extra
)

A missing [satellite] / [cwop] extra fails loud (typed error), never a silent empty column. Covariates require a DataFrame return path — combining a covariate flag with return_type="list" raises ValueError. Adding a covariate to a trained model is itself a retrain event.

TypeScript-specific kwargs

backend and return_type are accepted in TS for surface-equivalence with Python:

// Accepted — no-op (TS arrays don't need a wrapper)
await dataset("KNYC", "...", "...", { return_type: "wrapper" });

// REJECTED at runtime — no Polars in browser/Node
await dataset("KNYC", "...", "...", { backend: "polars" });
// → DataAvailabilityError(reason="model_unavailable", hint="polars backend not available in TypeScript SDK …")

This means a cross-language wrapper can pass the same options dict to either SDK; only Python actually routes to Polars.

Output columns

Default fused output (without include_forecast):

Column	Type	Notes
date	str	Settlement date in station LST
station	str	NWS code (e.g. "NYC") — not the ICAO
cli_high_f	float	Climate (CLI) max temp °F
cli_low_f	float	Climate (CLI) min temp °F
cli_report_type	str	"PRELIM" or "FINAL"
obs_high_f	int	Observation max temp °F (integer-precision)
obs_low_f	int	Observation min temp °F (integer-precision)
obs_high_at	str	ISO-8601 UTC timestamp of the obs_high
obs_low_at	str	ISO-8601 UTC timestamp of the obs_low
obs_mean_f	float	Daily mean temp °F
obs_max_wind_kt	float	Max sustained wind (kt)
obs_max_gust_kt	float	Max gust (kt)
obs_total_precip_in	float	Daily precip total (in)
obs_count	int	Number of underlying observations for the day
market_close_utc	str	UTC timestamp of the next Kalshi market close

Adding include_forecast=True appends fcst_high_f, fcst_low_f, fcst_model, fcst_issued_at, fcst_pop_6hr_pct, fcst_qpf_6hr_in. Covariate flags append sat_* / cwop_* columns.

Parity guarantee

dataset(station, from_date, to_date) (and its research() alias) is byte-equivalent to mostlyright==0.14.1’s client.pairs() on the canonical parity fixtures. Default-path changes go through a hard gate before each release. See Source identity and the Parity table.

See also

• Choosing the label — label="cli" vs "daily_extremes" vs None vs BYO, and the D-22 default-label bridge
• Extending dataset() — station panels + the experimental contributor registry
• Markets — the thin kalshi.dataset / polymarket.dataset delegators + outcome=True
• Source identity — the source= / delivery= / grain contract + SourceMismatchError
• Temporal safety — KnowledgeView and LeakageDetector
• Forecasts — IEM MOS + NWP attached via features=["forecasts"]
• v1.15 migration guide — the label axis, markets delegators, panels, and the deprecation trains
• Ingest strategies — how obs() picks exact_window vs warm_cache
• API reference: mostlyright.research (Python) · research (TypeScript)