Skip to content
Mostly Right

Use case

Walk-forward model evaluation without leakage

You have a model that predicts tomorrow’s high. Evaluating it on a random shuffle of history flatters it; the seasonal cycle leaks the future into the past. This recipe scores the model the honest way — walking forward in time, with KnowledgeView enforcing that every prediction uses only what was knowable at decision time.

1. Pull history with temporal safety

Section titled “1. Pull history with temporal safety”
from mostlyright import research
from mostlyright.core.temporal import KnowledgeView
import pandas as pd


df = research("KNYC", "2023-01-01", "2024-12-31", include_forecast=True)

Each row carries date (LST), cli_high_f (climate normal), fcst_high_f (MOS forecast), and obs_high_f (realized). At decision time t you only know the forecast and the normal — not the realized high:

def decision_view(t: pd.Timestamp) -> pd.DataFrame:
    return KnowledgeView(df, as_of=t).filter_known_at()

KnowledgeView is the leakage-detection primitive — see Temporal safety.

2. A point model + its error

Section titled “2. A point model + its error”

Start with a transparent baseline: tomorrow’s high is the MOS forecast, bias-corrected on history.

def fit_bias(train: pd.DataFrame) -> float:
    return (train["obs_high_f"] - train["fcst_high_f"]).mean()


def predict_high(row: dict, bias: float) -> float:
    return row["fcst_high_f"] + bias

Point error is mean absolute error against the realized high. The raw MOS forecast (bias = 0) is the baseline you have to beat.

3. A probabilistic forecast + its score

Section titled “3. A probabilistic forecast + its score”

Many weather questions are about exceeding a threshold — “will tomorrow’s high clear 90 °F?” (a heat-advisory cutoff, a crop-stress line, a load-forecast trigger). Turn a point prediction into an exceedance probability with the Gaussian CDF, then score it with the Brier score, the canonical proper scoring rule for probabilistic forecasts:

from scipy.stats import norm


def exceedance_prob(prediction: float, threshold: float, sigma: float = 4.0) -> float:
    # P(high > threshold) under N(prediction, sigma)
    return 1 - norm.cdf((threshold - prediction) / sigma)


def brier(prob: float, realized_high: float, threshold: float) -> float:
    outcome = 1.0 if realized_high > threshold else 0.0
    return (prob - outcome) ** 2

σ=4 °F is roughly the RMSE of a 24-hour MOS high; calibrate per station and season for production.

4. Compare against climatology

Section titled “4. Compare against climatology”

The trivial baseline is “always predict the normal.” If your forecast-driven model doesn’t beat it, the forecast added no signal:

def climo_exceedance(row: dict, threshold: float, sigma: float = 6.5) -> float:
    return 1 - norm.cdf((threshold - row["cli_high_f"]) / sigma)

Score both the same way and compare mean Brier across a grid of thresholds.

5. Test for leakage

Section titled “5. Test for leakage”

Three common leakage patterns and how the SDK catches them:

LeakageWhat the SDK does
Joining obs_high_f into the predictorKnowledgeView(as_of=t) drops obs rows where observed_at > t
Using a future NWP cycleforecast_nwp(cycle=t) raises if the cycle is in the future
Sweeping a hyperparameter on the test foldHold out a final unseen tail before any sweep — your discipline

Audit the inputs to any prediction:

from mostlyright.core.temporal import LeakageDetector


detector = LeakageDetector(as_of=decision_time)
detector.audit(prediction_inputs)   # raises LeakageError on any future column

6. Walk forward

Section titled “6. Walk forward”

Every prediction refits on history up to as_of, then predicts the next day. Naive whole-history regression contaminates.

records = []
for as_of in pd.date_range("2023-07-01", "2024-12-31", freq="D"):
    train = df[df["date"] < as_of.strftime("%Y-%m-%d")]
    bias = fit_bias(train)


    target_date = (as_of + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
    target = df[df["date"] == target_date]
    if target.empty:
        continue


    row = target.iloc[0].to_dict()
    pred = predict_high(row, bias)
    records.append({
        "date": target_date,
        "abs_err": abs(pred - row["obs_high_f"]),
        "brier_90f": brier(exceedance_prob(pred, 90.0), row["obs_high_f"], 90.0),
    })


scores = pd.DataFrame(records)
print(f"walk-forward MAE   = {scores['abs_err'].mean():.2f} °F")
print(f"walk-forward Brier = {scores['brier_90f'].mean():.4f}  (>90 °F)")

TypeScript

Section titled “TypeScript”

The TS research() returns the same row array. Reproduce the loop with a Math-based normal CDF (scipy.stats.norm.cdf ≈ 0.5 * (1 + erf(z / √2))) — bring your own erf or a small math library.

import { research } from "mostlyright";


const rows = await research("KNYC", "2023-01-01", "2024-12-31", {
  include_forecast: true,
});
// same walk-forward shape: fit bias on the past slice, score the next day

What this recipe doesn’t do

Section titled “What this recipe doesn’t do”
  • Refit a real model each step. A constant bias is a stand-in for whatever you actually train; the walk-forward harness is the point.
  • Calibrate σ per threshold. Exceedance sharpness varies by season and by how far the threshold sits from the normal.
  • K-fold in time. A single forward pass is one path; blocked time-series CV gives a distribution.

See also

Section titled “See also”