Skip to content
Mostly Right

Use case

Compare forecast accuracy across models

You want to know which NWP model best predicts the next-day high for KNYC. Pull each model’s forecast, join against the realized observation, score by lead-time bucket. Python-only — NWP is deferred in TS v1.x.

The question

Section titled “The question”

“For 24-hour-ahead high-temperature forecasts at KNYC over the last 18 months, which of HRRR / GFS / NBM has the lowest MAE?“

1. Pull the joined frame

Section titled “1. Pull the joined frame”
from mostlyright import research


df = research(
    "KNYC", "2024-01-01", "2025-06-30",
    include_forecast=True,
    forecast_models=["hrrr", "gfs", "nbm"],
)
# Columns: obs_high_f, fcst_high_f (IEM MOS),
#          fcst_high_f_nwp_hrrr, fcst_high_f_nwp_gfs, fcst_high_f_nwp_nbm

Each fcst_high_f_nwp_<model> column is the model’s forecast for that settlement date, run from the canonical morning cycle (12Z), with a fixed 24h lead.

2. Per-model error metrics

Section titled “2. Per-model error metrics”
import pandas as pd


models = ["hrrr", "gfs", "nbm"]
metrics = {}


for m in models:
    col = f"fcst_high_f_nwp_{m}"
    rows = df.dropna(subset=[col, "obs_high_f"])
    err = rows[col] - rows["obs_high_f"]
    metrics[m] = {
        "n": len(rows),
        "mae": err.abs().mean(),
        "rmse": (err**2).mean()**0.5,
        "bias": err.mean(),
    }


# IEM MOS baseline
mos = df.dropna(subset=["fcst_high_f", "obs_high_f"])
mos_err = mos["fcst_high_f"] - mos["obs_high_f"]
metrics["mos"] = {
    "n": len(mos), "mae": mos_err.abs().mean(),
    "rmse": (mos_err**2).mean()**0.5, "bias": mos_err.mean(),
}


print(pd.DataFrame(metrics).T)
#         n   mae   rmse   bias
# hrrr  548  2.31   3.05   0.12
# gfs   548  3.18   4.22   0.41
# nbm   548  2.05   2.78  -0.08
# mos   548  1.94   2.61  -0.03

NBM beats GFS handily, HRRR beats GFS too, but IEM MOS is best — unsurprising for a single-station prediction problem (MOS is station-tuned).

3. Score by season

Section titled “3. Score by season”

NWP performance varies seasonally — winter convection vs summer ridge. Slice the frame:

df["month"] = pd.to_datetime(df["date"]).dt.month
df["season"] = df["month"].map(
    lambda m: "winter" if m in (12, 1, 2) else
              "spring" if m in (3, 4, 5) else
              "summer" if m in (6, 7, 8) else
              "fall"
)


for s in ["winter", "spring", "summer", "fall"]:
    sub = df[df["season"] == s].dropna(subset=["fcst_high_f_nwp_hrrr", "obs_high_f"])
    err = (sub["fcst_high_f_nwp_hrrr"] - sub["obs_high_f"]).abs().mean()
    print(f"{s}: HRRR 24h MAE = {err:.2f}°F (n={len(sub)})")

4. Lead-time grid (multi-hour fxx)

Section titled “4. Lead-time grid (multi-hour fxx)”

For lead-time comparison, call the standalone forecast_nwp() per fxx:

from mostlyright.weather import forecast_nwp
from datetime import datetime, timedelta, UTC
import pandas as pd


def score_at_lead(model: str, fxx_hours: int) -> dict:
    cycles = pd.date_range("2024-06-01", "2024-08-31", freq="D", tz="UTC")
    errs = []
    for cycle in cycles:
        try:
            row = forecast_nwp("KNYC", model, cycle=cycle.to_pydatetime(), fxx=fxx_hours)
            # ... join against an obs window centered on cycle + fxx
            ...
        except Exception:
            continue
    return {"model": model, "fxx": fxx_hours, "mae": ...}


# Sweep fxx=12, 24, 48, 72
for fxx in [12, 24, 48, 72]:
    for m in ["hrrr", "gfs", "nbm"]:
        print(score_at_lead(m, fxx))

Typical result shape: HRRR dominates at fxx≤18h; GFS catches up at 48h+; NBM is consistently strong because it blends.

5. Picking the right model per lead

Section titled “5. Picking the right model per lead”

Once you have a per-lead score grid, you can build a blend that picks per (lead, station):

def best_model_for(lead_hours: int, station: str) -> str:
    # Lookup in the score grid you built in step 4.
    ...


forecast = forecast_nwp(station, best_model_for(24, station), cycle=..., fxx=24)

For production, refit this monthly — model performance drifts as NWP centers ship retraining cycles.

TypeScript note

Section titled “TypeScript note”

NWP forecasts are deferred in the TS SDK — see NWP forecasts in TypeScript. Your TS app can either:

  • Run this Python pipeline offline and serialize the score grid to JSON
  • Use IEM MOS via iemMosForecasts() (which is what the fcst_high_f column carries — strong baseline)

See also

Section titled “See also”