Train an ML model on observations

You want a model that predicts tomorrow’s high at KNYC from today’s observations + the morning’s forecast. This is the full ML loop: data pull, feature engineering, train/val split, model fit, backtest.

1. Pull 5 years of training data

import mostlyright

df = mostlyright.dataset(
    "KNYC", "2020-01-01", "2024-12-31",
    include_forecast=True,
    qc=True,  # carry QC flags
)

# Filter to QC-clean rows for model training
df = df[df["qc_status"] == "clean"].copy()
print(f"n_clean = {len(df)}")  # ~1750

This is one dataset() call — under a minute on a warm cache, ~5 min cold for 5 years. (research() is a fully working alias of dataset() if you have it in existing code.)

2. Engineer features

import pandas as pd
from mostlyright.transforms import relative_humidity

# 1-day lag features (yesterday's signal predicts today)
for col in ["obs_high_f", "obs_low_f", "obs_mean_dewpoint_f"]:
    df[f"{col}_lag1"] = df[col].shift(1)
    df[f"{col}_lag7"] = df[col].shift(7)  # week-ago analog

# Forecast-derived
df["fcst_vs_climo"] = df["fcst_high_f"] - df["cli_high_f"]
df["fcst_vs_yesterday"] = df["fcst_high_f"] - df["obs_high_f_lag1"]

# Calendar
df["date"] = pd.to_datetime(df["date"])
df["doy"] = df["date"].dt.dayofyear
df["doy_sin"] = (2 * 3.14159 * df["doy"] / 365.25).apply(__import__("math").sin)
df["doy_cos"] = (2 * 3.14159 * df["doy"] / 365.25).apply(__import__("math").cos)

# Drop initial NaNs from lag construction
df = df.dropna()

3. Split chronologically (no random shuffle)

df = df.sort_values("date").reset_index(drop=True)

# Last 90 days = test; previous 90 = val; rest = train
test = df.iloc[-90:]
val = df.iloc[-180:-90]
train = df.iloc[:-180]
print(f"train={len(train)} val={len(val)} test={len(test)}")

Never random-shuffle a time series — it leaks future into past via the seasonal cycle.

4. Train a baseline

import xgboost as xgb

features = [
    "fcst_high_f", "fcst_vs_climo", "fcst_vs_yesterday",
    "obs_high_f_lag1", "obs_high_f_lag7", "obs_low_f_lag1",
    "obs_mean_dewpoint_f_lag1",
    "cli_high_f", "doy_sin", "doy_cos",
]
target = "obs_high_f"

dtrain = xgb.DMatrix(train[features], label=train[target])
dval = xgb.DMatrix(val[features], label=val[target])

params = {
    "objective": "reg:squarederror",
    "eta": 0.05,
    "max_depth": 5,
    "eval_metric": "mae",
}

model = xgb.train(
    params, dtrain,
    num_boost_round=400,
    evals=[(dtrain, "train"), (dval, "val")],
    early_stopping_rounds=30,
)

Typical output:

[0]     train-mae:3.4   val-mae:3.6
[50]    train-mae:1.7   val-mae:2.0
[120]   train-mae:1.4   val-mae:1.9   ← early stop

A 1.9°F val MAE — better than the raw fcst_high_f (~2.6°F MOS baseline).

5. Score on holdout

dtest = xgb.DMatrix(test[features], label=test[target])
preds = model.predict(dtest)

mae = ((preds - test[target]).abs()).mean()
print(f"holdout MAE = {mae:.2f}°F")

# Beats the raw MOS baseline?
mos_mae = (test["fcst_high_f"] - test[target]).abs().mean()
print(f"raw MOS baseline MAE = {mos_mae:.2f}°F")

If your model’s holdout MAE is meaningfully below the MOS baseline, the engineered features add signal. If not, simplify — you’re overfitting.

6. Verify temporal safety

After training, audit the entire pipeline for leakage:

from mostlyright.core.temporal import LeakageDetector

detector = LeakageDetector(as_of=test["date"].iloc[0])
inputs_for_test_row_0 = train[features].iloc[-1:]  # last train row
detector.audit(inputs_for_test_row_0)
# Raises LeakageError if any column reflects data from after as_of.

LeakageDetector is the post-hoc check: if you joined any feature that was computed from a future observation (e.g. a forward-rolling mean), it raises here.

See Temporal safety.

7. Source-identity discipline

If you trained on source="iem" and deploy against source="awc", the integer-precision difference (IEM may emit non-integer °F; AWC is integer) silently corrupts your prediction by ~0.5°F.

# Train + deploy with an explicit source pin
df = mostlyright.dataset(
    "KNYC", "2020-01-01", "2024-12-31",
    include_forecast=True,
    source="iem",  # provenance pin — every observation row's source = "iem"
)

In production, the loader for source="iem" raises SourceMismatchError if it sees an AWC row. No silent drift.

Note that a source pin changes the row composition versus the fused default — treat switching a feature from fused to pinned as a retrain event, not a find-replace. See Source identity and the v1.13 migration guide.

8. Multi-station joint model

You can train one model across N stations by including station as a feature:

import pandas as pd
import mostlyright

dfs = []
for station in ["KNYC", "KORD", "KLAX", "KMIA", "KDEN", "KSEA", "KATL"]:
    df = mostlyright.dataset(
        station, "2020-01-01", "2024-12-31",
        include_forecast=True,
    )
    df["station"] = station
    dfs.append(df)

joint = pd.concat(dfs, ignore_index=True)
joint = pd.get_dummies(joint, columns=["station"], prefix="st")
# Add station-onehot columns to `features`.

A joint model can borrow strength across stations (regularization helps the rarer ones).

9. TypeScript path

The TS SDK gives you dataset() output ready for tfjs / onnxruntime / a remote model:

import { dataset } from "mostlyright";

const rows = await dataset("KNYC", "2020-01-01", "2024-12-31", {
  include_forecast: true,
});

// Hand-engineer features in TS, or POST to a Python inference service.
const features = rows.map((r) => [
  r.fcst_high_f, r.cli_high_f, r.obs_high_f_lag1,
  // …
]);

For training, the Python path is almost always more ergonomic. Run training offline; export the model; serve via TS in the browser.

What this recipe doesn’t do

• Cross-validate — you should run K time-based folds, not just a single holdout.
• Calibrate uncertainty — XGBoost gives point estimates. For probabilistic forecasts (exceedance, intervals) you need a calibrated distribution, not a point.
• Validate seasonally — a 90-day holdout doesn’t see the full year.

See also

• Build a training table — assemble sub-daily features first
• Walk-forward model evaluation — score this model without leakage
• Transforms — feature engineering helpers
• Quality control — qc=True and how clean filtering changes the training set
• Temporal safety — leakage primitives