# Forecast against observation Source: https://mostlyright.md/docs/recipes/forecast-vs-observation/ Compare three lead hours from one GFS cycle with the airport observations for the same hours. ## The question Forecasts and observations come from different publishers. They also use different formats, timestamps and temperature units. This recipe puts them into one table so the error can be measured. How far off was the 12 September 2026 00Z GFS two-metre temperature forecast for Washington National at six, twelve and twenty-four hours ahead? ## The sources | | GFS forecast | Station observation | | --- | --- | --- | | Publisher | NOAA NCEP, through the NOMADS GRIB filter | Iowa Environmental Mesonet | | Address | `https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f006&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos` | `https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DCA&data=tmpf&...&format=onlycomma&tz=Etc/UTC&report_type=3` | | Format | GRIB2, served as `application/octet-stream`, 508,159 bytes for the six-hour file | comma-separated text with the header `station,valid,tmpf,dwpf,relh` | | Cadence | four model cycles a day, and NOMADS keeps roughly the last ten days | a routine report near the end of every hour | | Rights | `public_domain_asserted`, output of a United States federal model under 17 U.S.C. 105 | `public_domain_asserted`, ASOS and METAR observations under the same section, redistributed by IEM, which asks for attribution | | Unit | kelvin | degrees Fahrenheit | The recipe uses three forecast sources from the same 00Z cycle, one for each lead hour. A fourth source provides station observations for the year. NOMADS only keeps recent GFS cycles, so these example addresses will stop working after NOAA removes this cycle. Replace them with addresses from a current cycle. ## The recipe ```json title="recipe.json" { "dataset": { "name": "Washington National weather forecast accuracy" }, "question": { "text": "How far off was the 12 September 2026 00Z GFS two-metre temperature forecast for Washington National at six, twelve and twenty-four hours ahead?" }, "table": { "name": "forecast_error_kdca", "description": "One row per station, model cycle and forecast valid time.", "grain": [ "station", "run_time", "valid" ], "columns": [ { "name": "station", "type": "string", "nullable": false, "description": "The four-letter station identifier, upper case on both sides of the join.", "presentation": { "chart": "top_values" } }, { "name": "run_time", "type": "timestamp", "nullable": false, "description": "The model cycle the forecast came from.", "presentation": { "chart": "timeline", "bucket": "day" } }, { "name": "valid", "type": "timestamp", "nullable": false, "description": "The hour the forecast value is for, which is the hour the observation is read for. The observation relation names its own time column valid, and the table declares this one under that name so a refresh can partition by it.", "presentation": { "chart": "timeline", "bucket": "hour" } }, { "name": "lead_hours", "type": "integer", "nullable": false, "description": "Hours from the model cycle to the valid time.", "presentation": { "chart": "top_values" } }, { "name": "forecast_temp_c", "type": "decimal", "nullable": true, "description": "The two-metre temperature the model published for the grid point nearest the station, converted from kelvin.", "presentation": { "chart": "histogram", "bins": 4 } }, { "name": "observed_temp_c", "type": "decimal", "nullable": true, "description": "The station's own temperature for the same hour, converted from Fahrenheit.", "presentation": { "chart": "histogram", "bins": 4 } }, { "name": "observed_at", "type": "timestamp", "nullable": true, "description": "When the station actually reported, which is a few minutes off the whole hour." }, { "name": "error_k", "type": "decimal", "nullable": true, "description": "Forecast minus observation. Positive means the model ran warm.", "presentation": { "chart": "histogram", "bins": 4, "story": "The model runs a degree or two warm at this station and the error grows with lead time." } }, { "name": "abs_error_k", "type": "decimal", "nullable": true, "description": "The size of the error, whichever way it went." }, { "name": "matched_grid_lat", "type": "decimal", "nullable": true, "description": "Latitude of the model grid point that answered, so the attribution can be checked." }, { "name": "matched_grid_lon", "type": "decimal", "nullable": true, "description": "Longitude of the model grid point that answered." }, { "name": "message_sha256", "type": "string", "nullable": false, "description": "Digest of the exact GRIB2 message the forecast value was decoded from.", "presentation": { "chart": "none" } } ] }, "sources": [ { "name": "gfs_f006", "description": "The two-metre temperature field of the 12 September 2026 00Z GFS cycle at lead hour 6, as the NOMADS filter serves it.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f006&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:3401506acf726e84ac86627b00857286bb036f3d7c62af3dbec7c0596e758603", "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright under 17 U.S.C. 105." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://nomads.ncep.noaa.gov", "parameters": [ { "name": "reader.family_id", "value": "weather.grib2" }, { "name": "reader.family_version", "value": "3.0.0" }, { "name": "reader.decode_options", "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"dca\",\"latitude\":\"38.8512\",\"longitude\":\"-77.0402\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-12T06:00:00Z\"},\"variable_name\":\"temperature_2m\"}" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 1000, "max_requests": 2 }, "closed": true }, { "name": "gfs_f012", "description": "The two-metre temperature field of the 12 September 2026 00Z GFS cycle at lead hour 12, as the NOMADS filter serves it.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f012&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:3401506acf726e84ac86627b00857286bb036f3d7c62af3dbec7c0596e758603", "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright under 17 U.S.C. 105." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://nomads.ncep.noaa.gov", "parameters": [ { "name": "reader.family_id", "value": "weather.grib2" }, { "name": "reader.family_version", "value": "3.0.0" }, { "name": "reader.decode_options", "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"dca\",\"latitude\":\"38.8512\",\"longitude\":\"-77.0402\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-12T12:00:00Z\"},\"variable_name\":\"temperature_2m\"}" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 1000, "max_requests": 2 }, "closed": true }, { "name": "gfs_f024", "description": "The two-metre temperature field of the 12 September 2026 00Z GFS cycle at lead hour 24, as the NOMADS filter serves it.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f024&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:3401506acf726e84ac86627b00857286bb036f3d7c62af3dbec7c0596e758603", "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright under 17 U.S.C. 105." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://nomads.ncep.noaa.gov", "parameters": [ { "name": "reader.family_id", "value": "weather.grib2" }, { "name": "reader.family_version", "value": "3.0.0" }, { "name": "reader.decode_options", "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"dca\",\"latitude\":\"38.8512\",\"longitude\":\"-77.0402\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-13T00:00:00Z\"},\"variable_name\":\"temperature_2m\"}" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 1000, "max_requests": 2 }, "closed": true }, { "name": "asos_kdca", "description": "Every routine METAR observation Washington National reported in 2026, as the Iowa Environmental Mesonet serves it in comma-separated text.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DCA&data=tmpf&data=dwpf&data=relh&year1=2026&month1=1&day1=1&year2=2026&month2=9&day2=13&format=onlycomma&tz=Etc/UTC&report_type=3&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:d5c5baec85de18b0262cc71cf6646fd60337f16954878f4d8e2ed4a9ba36a213", "claim_note": "ASOS and METAR observations are United States Government works under 17 U.S.C. 105, redistributed by the Iowa Environmental Mesonet, which asks for attribution. The evidence digest is over IEM's ASOS download page." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu" }, "limits": { "max_source_bytes": 33554432, "max_rows": 200000, "max_requests": 1 }, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 31536000, "request": { "start": { "encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": { "year": "year1", "month": "month1", "day": "day1" } }, "end": { "encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": { "year": "year2", "month": "month2", "day": "day2" } } }, "merge": { "materialization": "partition_replace", "partition": { "column": "valid", "key": "iso_date_prefix" }, "row_identity": [ "station", "valid" ] } } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "forecasts", "sql": "select upper(point_id) as station, cast(run_time as timestamp with time zone) as run_time, cast(valid_time as timestamp with time zone) as valid, cast(lead_text as integer) as lead_hours, cast(try_cast(value as decimal(12,4)) - 273.15 as decimal(6,2)) as forecast_temp_c, try_cast(matched_grid_lat as decimal(12,6)) as matched_grid_lat, try_cast(matched_grid_lon as decimal(12,6)) as matched_grid_lon, message_sha256 from (select '006' as lead_text, point_id, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256 from gfs_f006 union all select '012' as lead_text, point_id, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256 from gfs_f012 union all select '024' as lead_text, point_id, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256 from gfs_f024)", "description": "Stack the three lead hours of one model cycle and convert the model's kelvin into Celsius. The Reader emits one row per named place, so each source contributes one row." }, { "step_id": "observations", "sql": "select station, cast(concat(valid, ':00+00') as timestamp with time zone) as observed_at, date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute) as observed_hour, cast((try_cast(tmpf as decimal(6,2)) - 32) * 5 / 9 as decimal(6,2)) as observed_temp_c from asos_kdca where valid is not null and valid <> '' and tmpf <> '' qualify row_number() over (partition by station, date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute) order by abs(date_diff('minute', date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute), cast(concat(valid, ':00+00') as timestamp with time zone))), valid) = 1", "description": "Convert the station's Fahrenheit readings to Celsius, round each report to the nearest whole hour, and keep the one report nearest that hour." }, { "step_id": "forecast_error_kdca", "sql": "select f.station, f.run_time, f.valid, f.lead_hours, f.forecast_temp_c, o.observed_temp_c, o.observed_at, cast(f.forecast_temp_c - o.observed_temp_c as decimal(6,2)) as error_k, cast(abs(f.forecast_temp_c - o.observed_temp_c) as decimal(6,2)) as abs_error_k, f.matched_grid_lat, f.matched_grid_lon, f.message_sha256 from forecasts f left join observations o on o.station = f.station and o.observed_hour = f.valid", "description": "Attach the observation for the hour the forecast is valid for. A left join keeps a lead hour the station reported nothing for." } ] }, "checks": [ { "check_id": "one_row_per_lead", "kind": "key_uniqueness", "enforcement": "required", "columns": [ "station", "run_time", "valid" ], "description": "A duplicate means the observation side held more than one report for a valid hour, which would multiply the forecast rows." }, { "check_id": "three_lead_hours", "kind": "row_expectation", "enforcement": "required", "min_rows": 3, "max_rows": 3, "description": "Three GRIB2 sources, one named place each, so the table is three rows." }, { "check_id": "forecast_present", "kind": "null_ceiling", "enforcement": "required", "columns": [ "forecast_temp_c" ], "max_null_ppm": 0 }, { "check_id": "observation_matched", "kind": "null_ceiling", "enforcement": "advisory", "columns": [ "observed_temp_c" ], "max_null_ppm": 0, "description": "Every lead hour should find an observation. An advisory failure records an hour the station missed without stopping the build." }, { "check_id": "error_plausible", "kind": "value_range", "enforcement": "required", "columns": [ "error_k" ], "min_value": "-30", "max_value": "30", "description": "A difference this large is a join on the wrong hour or a unit that was never converted, not a forecast miss." } ], "units": [ { "column": "forecast_temp_c", "unit": "Cel" }, { "column": "observed_temp_c", "unit": "Cel" }, { "column": "error_k", "unit": "K" }, { "column": "abs_error_k", "unit": "K" }, { "column": "lead_hours", "unit": "h" } ], "timezone": "UTC" } ``` ## Part by part ### The forecast Reader pin Each GRIB2 source uses `weather.grib2@3.0.0`. Versions `1.0.0` and `2.0.0` only accept a single extracted message. Version `3.0.0` indexes a bounded collection and selects one message from it. The run fails if the selector finds no match or more than one match. The selector identifies the message by its coordinates. Only `valid_time` changes between the three sources: | Setting | Value | Meaning | | --- | --- | --- | | `discipline` | `0` | meteorological products | | `parameter_category` | `0` | temperature | | `parameter_number` | `0` | temperature, the `TMP` field | | `surface_type` | `103` | a specified height above ground | | `surface_value` | `{"numerator": 2, "denominator": 1}` | two metres | | `run_time` | `2026-09-12T00:00:00Z` | the model cycle, the same in all three | | `valid_time` | `2026-09-12T06:00:00Z`, `T12:00:00Z`, `2026-09-13T00:00:00Z` | the hour each forecast is for | Recipe documents cannot contain fractional JSON numbers. For that reason, `surface_value` uses a numerator and denominator, while the point coordinates use decimal text such as `"latitude": "38.8512"`. This preserves the same digits on every host. The Reader returns one row for each named place, so each source contributes one row. The columns are `point_id`, `variable`, `level`, `run_time`, `valid_time`, `value`, `matched_grid_lat`, `matched_grid_lon` and `message_sha256`. The `matched_grid_lat` and `matched_grid_lon` values show which grid cell supplied the forecast. You can use them to check its distance from the station. The forecast sources set `closed: true` because the published cycle will not change. Later refreshes reuse the rows from the previous version instead of downloading them from NOAA again. Their source reports show zero bytes fetched. ### The observation window The observation source does not pin a Reader. IEM returns comma-separated text, so the relation keeps the original header and every column arrives as text. The address expresses its date range through six query parameters. The `window.request` block names those parameters: ```json "request": { "start": { "encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": { "year": "year1", "month": "month1", "day": "day1" } }, "end": { "encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": { "year": "year2", "month": "month2", "day": "day2" } } } ``` The locator remains a literal, fetchable address. The window changes its named parameters rather than treating the whole locator as a template. The host allowlist, origin check and Sources card all use `display_locator`. IEM can correct observations after publication, so each refresh looks back two days. `lookback_seconds` sets that period to 172800. The `merge.partition.key` value `iso_date_prefix` takes the first ten characters of `valid`. The `merge.row_identity` fields are `station` and `valid`, which allow the merge to detect duplicate reports. The table declares a column named `valid` as well, and that is what makes this four-source plan refreshable. A refresh plan naming more than one source needs the window's partition column on the table, under the name the acquired relation uses. Here it holds the hour each row is about, which is the forecast's valid time on the model side and the report's own time on the observation side. The table is partitioned by the day read off it. Do not combine `closed: true` with `window.snapshot`. The schema rejects that combination. An ordinary `window` can register beside `closed: true`, but the flag has no effect because every refresh already requests a bounded range. This recipe only declares the window. ### Two units, one column each The model publishes kelvin, while the station publishes Fahrenheit. The transform converts both values to Celsius. The `units` block describes the result but does not perform the conversion. ```sql cast(try_cast(value as decimal(12,4)) - 273.15 as decimal(6,2)) as forecast_temp_c cast((try_cast(tmpf as decimal(6,2)) - 32) * 5 / 9 as decimal(6,2)) as observed_temp_c ``` Keep the outer `cast` in both expressions. Without it, the division returns `DOUBLE`, which does not match a column declared as `decimal`. The `units` block declares `error_k` and `abs_error_k` in `K` rather than `Cel`. Celsius and kelvin intervals have the same size, but `Cel` is an affine unit and cannot describe a difference. `lead_hours` uses `h`. The two grid coordinates are omitted from `units`. ### Rounding the observation to the hour The model forecast is valid on the hour, while Washington National usually reports at 52 minutes past. Round the observation time before joining the two sources: ```sql date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute) as observed_hour ``` Adding thirty minutes before truncation rounds each report to the nearest hour, so 00:52 becomes 01:00. The following `qualify` keeps the report closest to that hour. It orders by `abs(date_diff('minute', observed_hour, observed_at))` and uses `valid` to break ties consistently. The table keeps `observed_at` so each row shows when the station actually reported. Rounding the join key does not imply that the observation happened on the hour. IEM writes `valid` as `2026-09-11 00:52`, without seconds or a UTC offset. The expression `concat(valid, ':00+00')` supplies both. A plain `cast` to `TIMESTAMP` would return a value without a timezone and cause `TRANSFORM_COLUMN_TYPE_MISMATCH` for a column declared as `timestamp`. ### The join ```sql from forecasts f left join observations o on o.station = f.station and o.observed_hour = f.valid ``` The Reader calls its point `dca`, while IEM calls the station `DCA`. The `forecasts` step applies `upper(point_id)` so the values match. Different capitalization is a common reason for an otherwise correct join to return no rows. The left join keeps a forecast even when the station has no report for that hour. An inner join would drop the unmatched forecast and make three requested hours look like two scored hours. ### Checks The `key_uniqueness` check covers `station`, `run_time` and `valid`. It catches cases where two station reports round to the same hour and duplicate a forecast row. Those duplicates would make every mean error calculated from the table wrong. The `row_expectation` check sets `min_rows: 3` and `max_rows: 3` because each of the three forecast sources returns one named place. Any other row count indicates a problem with selection or decoding. The `value_range` check keeps `error_k` between `"-30"` and `"30"`. A difference of 40 degrees is more likely to mean that the join used the wrong hour or a kelvin value was not converted. The check stops the run before that result becomes a table version. The `observation_matched` check is `advisory`. If the station missed an hour, the receipt records it without failing the run. ## Register, run, inspect Use the real table for one full build. For a bounded test first, follow the [scratch-table workflow](/docs/build/run-and-inspect/#test-before-the-full-build). A successful sample on the real table can publish truncated data. Stop if the forecast refuses; size the recipe before running. ```sh mr-data dataset create --name "Washington National weather forecast accuracy" --json mr-data recipe recipe.json --json mr-data recipe explain RECIPE_ID --json mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json mr-data watch RUN_ID --json mr-data peek RUN_ID --json mr-data checks RUN_ID --json mr-data receipt RUN_ID --json mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --refresh --json ``` ## What the run reports The receipt includes one `coverage` entry for each source. A GRIB2 entry records the Reader name and version, validated options digest, raw source digest and row digest. The observation entry also records the `window` covered by the run. A later refresh continues from that window. Every forecast row includes `message_sha256`, which identifies the GRIB2 message that supplied the value. This lets you trace the row back to its source bytes. If the model marks a point as missing, the Reader still returns a row under the document's place name and leaves the value blank. It does not substitute a number like 9999, which could be mistaken for a temperature. ## Traps Reader-pinned sources require the batch environment because the Reader runs in a child process with its own memory limit. The warm container cannot provide that limit. If the run uses the wrong environment, the worker stops before fetching the source and explains why. The weather Reader is an optional dependency loaded on first use. If no compatible binding is installed, the run returns `READER_DEPENDENCY_UNAVAILABLE` instead of crashing. `asos.py` limits concurrent requests by IP address. When two requests overlap, it can return `429 Too many requests from your IP address`. One source covering a year makes one request. Twenty separate yearly sources make twenty requests that may collide. If the model changes its grid, the next refresh stops before reading a value. The error describes the change and names the sample file needed to allow the new combination. Two temperature fields from the same model can both look like ordinary numbers on the same grid and at the same time. If the table includes the wrong field, downstream code cannot detect the mistake. The selector therefore names the field, and the build stops when the file contains a different one. The observation source uses a window, while the forecast sources set `closed: true`. That is the shape a refresh admits: the window asks IEM for the previous two days and the closed sources reuse their sealed bytes without contacting NOAA. The table declares `valid`, so the plan meets the multi-source condition as well and Studio admits it. What a refresh buys here is a corrected observation, not a new row. The cycle is fixed and the three lead hours never move, so a refresh that finds nothing revised ends `unchanged`. Once the scored hours fall outside the two-day lookback, every refresh ends that way. To follow later model cycles, register new sources for each cycle. You can also use one source with the cycle date in its path template, then configure `window.request` with `location: "path"`. See [Source kinds and connectors](/docs/reference/sources/) for the window fields.