Forecast against observation
Compare three lead hours from one GFS cycle with the airport observations for the same hours.
The question
Section titled “The question”Forecasts and observations come from different publishers. They also use different formats, timestamps and temperature units. This recipe puts them into one table so the error can be measured.
How far off was the 12 September 2026 00Z GFS two-metre temperature forecast for Washington National at six, twelve and twenty-four hours ahead?
The sources
Section titled “The sources”| GFS forecast | Station observation | |
|---|---|---|
| Publisher | NOAA NCEP, through the NOMADS GRIB filter | Iowa Environmental Mesonet |
| Address | https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f006&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos |
https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DCA&data=tmpf&...&format=onlycomma&tz=Etc/UTC&report_type=3 |
| Format | GRIB2, served as application/octet-stream, 508,159 bytes for the six-hour file |
comma-separated text with the header station,valid,tmpf,dwpf,relh |
| Cadence | four model cycles a day, and NOMADS keeps roughly the last ten days | a routine report near the end of every hour |
| Rights | public_domain_asserted, output of a United States federal model under 17 U.S.C. 105 |
public_domain_asserted, ASOS and METAR observations under the same section, redistributed by IEM, which asks for attribution |
| Unit | kelvin | degrees Fahrenheit |
The recipe uses three forecast sources from the same 00Z cycle, one for each lead hour. A fourth source provides station observations for the year. NOMADS only keeps recent GFS cycles, so these example addresses will stop working after NOAA removes this cycle. Replace them with addresses from a current cycle.
The recipe
Section titled “The recipe”{ "dataset": { "name": "Washington National weather forecast accuracy", "description": "Each row is one lead hour of the 12 September 2026 00Z GFS cycle at Washington National, carrying the forecast two-metre temperature, the station's own observation for the same hour, and the difference between them." }, "question": { "text": "How far off was the 12 September 2026 00Z GFS two-metre temperature forecast for Washington National at six, twelve and twenty-four hours ahead?" }, "table": { "name": "forecast_error_kdca", "description": "One row per station, model cycle and forecast valid time.", "grain": [ "station", "run_time", "valid" ], "columns": [ { "name": "station", "type": "string", "nullable": false, "description": "The four-letter station identifier, upper case on both sides of the join.", "presentation": { "chart": "top_values" } }, { "name": "run_time", "type": "timestamp", "nullable": false, "description": "The model cycle the forecast came from.", "presentation": { "chart": "timeline", "bucket": "day" } }, { "name": "valid", "type": "timestamp", "nullable": false, "description": "The hour the forecast value is for, which is the hour the observation is read for. The observation relation names its own time column valid, and the table declares this one under that name so a refresh can partition by it.", "presentation": { "chart": "timeline", "bucket": "hour" } }, { "name": "lead_hours", "type": "integer", "nullable": false, "description": "Hours from the model cycle to the valid time.", "presentation": { "chart": "top_values" } }, { "name": "forecast_temp_c", "type": "decimal", "nullable": true, "description": "The two-metre temperature the model published for the grid point nearest the station, converted from kelvin.", "presentation": { "chart": "histogram", "bins": 4 } }, { "name": "observed_temp_c", "type": "decimal", "nullable": true, "description": "The station's own temperature for the same hour, converted from Fahrenheit.", "presentation": { "chart": "histogram", "bins": 4 } }, { "name": "observed_at", "type": "timestamp", "nullable": true, "description": "When the station actually reported, which is a few minutes off the whole hour." }, { "name": "error_k", "type": "decimal", "nullable": true, "description": "Forecast minus observation. Positive means the model ran warm.", "presentation": { "chart": "histogram", "bins": 4, "story": "The model runs a degree or two warm at this station and the error grows with lead time." } }, { "name": "abs_error_k", "type": "decimal", "nullable": true, "description": "The size of the error, whichever way it went." }, { "name": "matched_grid_lat", "type": "decimal", "nullable": true, "description": "Latitude of the model grid point that answered, so the attribution can be checked." }, { "name": "matched_grid_lon", "type": "decimal", "nullable": true, "description": "Longitude of the model grid point that answered." }, { "name": "message_sha256", "type": "string", "nullable": false, "description": "Digest of the exact GRIB2 message the forecast value was decoded from.", "presentation": { "chart": "none" } } ] }, "sources": [ { "name": "gfs_f006", "description": "The two-metre temperature field of the 12 September 2026 00Z GFS cycle at lead hour 6, as the NOMADS filter serves it.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f006&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:3401506acf726e84ac86627b00857286bb036f3d7c62af3dbec7c0596e758603", "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright under 17 U.S.C. 105." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://nomads.ncep.noaa.gov", "parameters": [ { "name": "reader.family_id", "value": "weather.grib2" }, { "name": "reader.family_version", "value": "3.0.0" }, { "name": "reader.decode_options", "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"dca\",\"latitude\":\"38.8512\",\"longitude\":\"-77.0402\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-12T06:00:00Z\"},\"variable_name\":\"temperature_2m\"}" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 1000, "max_requests": 2 }, "closed": true }, { "name": "gfs_f012", "description": "The two-metre temperature field of the 12 September 2026 00Z GFS cycle at lead hour 12, as the NOMADS filter serves it.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f012&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:3401506acf726e84ac86627b00857286bb036f3d7c62af3dbec7c0596e758603", "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright under 17 U.S.C. 105." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://nomads.ncep.noaa.gov", "parameters": [ { "name": "reader.family_id", "value": "weather.grib2" }, { "name": "reader.family_version", "value": "3.0.0" }, { "name": "reader.decode_options", "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"dca\",\"latitude\":\"38.8512\",\"longitude\":\"-77.0402\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-12T12:00:00Z\"},\"variable_name\":\"temperature_2m\"}" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 1000, "max_requests": 2 }, "closed": true }, { "name": "gfs_f024", "description": "The two-metre temperature field of the 12 September 2026 00Z GFS cycle at lead hour 24, as the NOMADS filter serves it.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f024&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:3401506acf726e84ac86627b00857286bb036f3d7c62af3dbec7c0596e758603", "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright under 17 U.S.C. 105." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://nomads.ncep.noaa.gov", "parameters": [ { "name": "reader.family_id", "value": "weather.grib2" }, { "name": "reader.family_version", "value": "3.0.0" }, { "name": "reader.decode_options", "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"dca\",\"latitude\":\"38.8512\",\"longitude\":\"-77.0402\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-13T00:00:00Z\"},\"variable_name\":\"temperature_2m\"}" } ] }, "limits": { "max_source_bytes": 3145728, "max_rows": 1000, "max_requests": 2 }, "closed": true }, { "name": "asos_kdca", "description": "Every routine METAR observation Washington National reported in 2026, as the Iowa Environmental Mesonet serves it in comma-separated text.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DCA&data=tmpf&data=dwpf&data=relh&year1=2026&month1=1&day1=1&year2=2026&month2=9&day2=13&format=onlycomma&tz=Etc/UTC&report_type=3&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:d5c5baec85de18b0262cc71cf6646fd60337f16954878f4d8e2ed4a9ba36a213", "claim_note": "ASOS and METAR observations are United States Government works under 17 U.S.C. 105, redistributed by the Iowa Environmental Mesonet, which asks for attribution. The evidence digest is over IEM's ASOS download page." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu" }, "limits": { "max_source_bytes": 33554432, "max_rows": 200000, "max_requests": 1 }, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 31536000, "request": { "start": { "encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": { "year": "year1", "month": "month1", "day": "day1" } }, "end": { "encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": { "year": "year2", "month": "month2", "day": "day2" } } }, "merge": { "materialization": "partition_replace", "partition": { "column": "valid", "key": "iso_date_prefix" }, "row_identity": [ "station", "valid" ] } } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "forecasts", "sql": "select upper(point_id) as station, cast(run_time as timestamp with time zone) as run_time, cast(valid_time as timestamp with time zone) as valid, cast(lead_text as integer) as lead_hours, cast(try_cast(value as decimal(12,4)) - 273.15 as decimal(6,2)) as forecast_temp_c, try_cast(matched_grid_lat as decimal(12,6)) as matched_grid_lat, try_cast(matched_grid_lon as decimal(12,6)) as matched_grid_lon, message_sha256 from (select '006' as lead_text, point_id, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256 from gfs_f006 union all select '012' as lead_text, point_id, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256 from gfs_f012 union all select '024' as lead_text, point_id, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256 from gfs_f024)", "description": "Stack the three lead hours of one model cycle and convert the model's kelvin into Celsius. The Reader emits one row per named place, so each source contributes one row." }, { "step_id": "observations", "sql": "select station, cast(concat(valid, ':00+00') as timestamp with time zone) as observed_at, date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute) as observed_hour, cast((try_cast(tmpf as decimal(6,2)) - 32) * 5 / 9 as decimal(6,2)) as observed_temp_c from asos_kdca where valid is not null and valid <> '' and tmpf <> '' qualify row_number() over (partition by station, date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute) order by abs(date_diff('minute', date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute), cast(concat(valid, ':00+00') as timestamp with time zone))), valid) = 1", "description": "Convert the station's Fahrenheit readings to Celsius, round each report to the nearest whole hour, and keep the one report nearest that hour." }, { "step_id": "forecast_error_kdca", "sql": "select f.station, f.run_time, f.valid, f.lead_hours, f.forecast_temp_c, o.observed_temp_c, o.observed_at, cast(f.forecast_temp_c - o.observed_temp_c as decimal(6,2)) as error_k, cast(abs(f.forecast_temp_c - o.observed_temp_c) as decimal(6,2)) as abs_error_k, f.matched_grid_lat, f.matched_grid_lon, f.message_sha256 from forecasts f left join observations o on o.station = f.station and o.observed_hour = f.valid", "description": "Attach the observation for the hour the forecast is valid for. A left join keeps a lead hour the station reported nothing for." } ] }, "checks": [ { "check_id": "one_row_per_lead", "kind": "key_uniqueness", "enforcement": "required", "columns": [ "station", "run_time", "valid" ], "description": "A duplicate means the observation side held more than one report for a valid hour, which would multiply the forecast rows." }, { "check_id": "three_lead_hours", "kind": "row_expectation", "enforcement": "required", "min_rows": 3, "max_rows": 3, "description": "Three GRIB2 sources, one named place each, so the table is three rows." }, { "check_id": "forecast_present", "kind": "null_ceiling", "enforcement": "required", "columns": [ "forecast_temp_c" ], "max_null_ppm": 0 }, { "check_id": "observation_matched", "kind": "null_ceiling", "enforcement": "advisory", "columns": [ "observed_temp_c" ], "max_null_ppm": 0, "description": "Every lead hour should find an observation. An advisory failure records an hour the station missed without stopping the build." }, { "check_id": "error_plausible", "kind": "value_range", "enforcement": "required", "columns": [ "error_k" ], "min_value": "-30", "max_value": "30", "description": "A difference this large is a join on the wrong hour or a unit that was never converted, not a forecast miss." } ], "units": [ { "column": "forecast_temp_c", "unit": "Cel" }, { "column": "observed_temp_c", "unit": "Cel" }, { "column": "error_k", "unit": "K" }, { "column": "abs_error_k", "unit": "K" }, { "column": "lead_hours", "unit": "h" } ], "timezone": "UTC"}Part by part
Section titled “Part by part”The forecast Reader pin
Section titled “The forecast Reader pin”Each GRIB2 source uses weather.grib2@3.0.0. Versions 1.0.0 and 2.0.0 only accept a single
extracted message. Version 3.0.0 indexes a bounded collection and selects one message from it. The
run fails if the selector finds no match or more than one match.
The selector identifies the message by its coordinates. Only valid_time changes between the three
sources:
| Setting | Value | Meaning |
|---|---|---|
discipline |
0 |
meteorological products |
parameter_category |
0 |
temperature |
parameter_number |
0 |
temperature, the TMP field |
surface_type |
103 |
a specified height above ground |
surface_value |
{"numerator": 2, "denominator": 1} |
two metres |
run_time |
2026-09-12T00:00:00Z |
the model cycle, the same in all three |
valid_time |
2026-09-12T06:00:00Z, T12:00:00Z, 2026-09-13T00:00:00Z |
the hour each forecast is for |
Recipe documents cannot contain fractional JSON numbers. For that reason, surface_value uses a
numerator and denominator, while the point coordinates use decimal text such as
"latitude": "38.8512". This preserves the same digits on every host.
The Reader returns one row for each named place, so each source contributes one row. The columns are
point_id, variable, level, run_time, valid_time, value, matched_grid_lat,
matched_grid_lon and message_sha256. The matched_grid_lat and matched_grid_lon values show
which grid cell supplied the forecast. You can use them to check its distance from the station.
The forecast sources set closed: true because the published cycle will not change. Later refreshes
reuse the rows from the previous version instead of downloading them from NOAA again. Their source
reports show zero bytes fetched.
The observation window
Section titled “The observation window”The observation source does not pin a Reader. IEM returns comma-separated text, so the relation keeps the original header and every column arrives as text.
The address expresses its date range through six query parameters. The window.request block names
those parameters:
"request": { "start": { "encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": { "year": "year1", "month": "month1", "day": "day1" } }, "end": { "encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": { "year": "year2", "month": "month2", "day": "day2" } }}The locator remains a literal, fetchable address. The window changes its named parameters rather
than treating the whole locator as a template. The host allowlist, origin check and Sources card all
use display_locator.
IEM can correct observations after publication, so each refresh looks back two days.
lookback_seconds sets that period to 172800. The merge.partition.key value
iso_date_prefix takes the first ten characters of valid. The merge.row_identity fields are
station and valid, which allow the merge to detect duplicate reports.
The table declares a column named valid as well, and that is what makes this four-source plan
refreshable. A refresh plan naming more than one source needs the window’s partition column on the
table, under the name the acquired relation uses. Here it holds the hour each row is about, which
is the forecast’s valid time on the model side and the report’s own time on the observation side.
The table is partitioned by the day read off it.
Do not combine closed: true with window.snapshot. The schema rejects that combination. An
ordinary window can register beside closed: true, but the flag has no effect because every
refresh already requests a bounded range. This recipe only declares the window.
Two units, one column each
Section titled “Two units, one column each”The model publishes kelvin, while the station publishes Fahrenheit. The transform converts both
values to Celsius. The units block describes the result but does not perform the conversion.
cast(try_cast(value as decimal(12,4)) - 273.15 as decimal(6,2)) as forecast_temp_ccast((try_cast(tmpf as decimal(6,2)) - 32) * 5 / 9 as decimal(6,2)) as observed_temp_cKeep the outer cast in both expressions. Without it, the division returns DOUBLE, which does not
match a column declared as decimal.
The units block declares error_k and abs_error_k in K rather than Cel. Celsius and kelvin
intervals have the same size, but Cel is an affine unit and cannot describe a difference.
lead_hours uses h. The two grid coordinates are omitted from units.
Rounding the observation to the hour
Section titled “Rounding the observation to the hour”The model forecast is valid on the hour, while Washington National usually reports at 52 minutes past. Round the observation time before joining the two sources:
date_trunc('hour', cast(concat(valid, ':00+00') as timestamp with time zone) + interval 30 minute) as observed_hourAdding thirty minutes before truncation rounds each report to the nearest hour, so 00:52 becomes
01:00. The following qualify keeps the report closest to that hour. It orders by
abs(date_diff('minute', observed_hour, observed_at)) and uses valid to break ties consistently.
The table keeps observed_at so each row shows when the station actually reported. Rounding the
join key does not imply that the observation happened on the hour.
IEM writes valid as 2026-09-11 00:52, without seconds or a UTC offset. The expression
concat(valid, ':00+00') supplies both. A plain cast to TIMESTAMP would return a value without a
timezone and cause TRANSFORM_COLUMN_TYPE_MISMATCH for a column declared as timestamp.
The join
Section titled “The join”from forecasts fleft join observations o on o.station = f.station and o.observed_hour = f.validThe Reader calls its point dca, while IEM calls the station DCA. The forecasts step applies
upper(point_id) so the values match. Different capitalization is a common reason for an otherwise
correct join to return no rows.
The left join keeps a forecast even when the station has no report for that hour. An inner join would drop the unmatched forecast and make three requested hours look like two scored hours.
Checks
Section titled “Checks”The key_uniqueness check covers station, run_time and valid. It catches cases where two
station reports round to the same hour and duplicate a forecast row. Those duplicates would make
every mean error calculated from the table wrong.
The row_expectation check sets min_rows: 3 and max_rows: 3 because each of the three forecast
sources returns one named place. Any other row count indicates a problem with selection or decoding.
The value_range check keeps error_k between "-30" and "30". A difference of 40 degrees is
more likely to mean that the join used the wrong hour or a kelvin value was not converted. The check
stops the run before that result becomes a table version.
The observation_matched check is advisory. If the station missed an hour, the receipt records it
without failing the run.
Register, run, inspect
Section titled “Register, run, inspect”mr-data dataset create --name "Washington National weather forecast accuracy" --jsonmr-data recipe recipe.json --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --sample --max-rows 20000 --jsonmr-data peek RUN_ID --jsonmr-data checks RUN_ID --jsonmr-data receipt RUN_ID --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --jsonmr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --refresh --jsonWhat the run reports
Section titled “What the run reports”The receipt includes one coverage entry for each source. A GRIB2 entry records the Reader name and
version, validated options digest, raw source digest and row digest. The observation entry also
records the window covered by the run. A later refresh continues from that window.
Every forecast row includes message_sha256, which identifies the GRIB2 message that supplied the
value. This lets you trace the row back to its source bytes.
If the model marks a point as missing, the Reader still returns a row under the document’s place name and leaves the value blank. It does not substitute a number like 9999, which could be mistaken for a temperature.
Reader-pinned sources require the batch environment because the Reader runs in a child process with its own memory limit. The warm container cannot provide that limit. If the run uses the wrong environment, the worker stops before fetching the source and explains why.
The weather Reader is an optional dependency loaded on first use. If no compatible binding is
installed, the run returns READER_DEPENDENCY_UNAVAILABLE instead of crashing.
asos.py limits concurrent requests by IP address. When two requests overlap, it can return
429 Too many requests from your IP address. One source covering a year makes one request. Twenty
separate yearly sources make twenty requests that may collide.
If the model changes its grid, the next refresh stops before reading a value. The error describes the change and names the sample file needed to allow the new combination.
Two temperature fields from the same model can both look like ordinary numbers on the same grid and at the same time. If the table includes the wrong field, downstream code cannot detect the mistake. The selector therefore names the field, and the build stops when the file contains a different one.
The observation source uses a window, while the forecast sources set closed: true. That is the
shape a refresh admits: the window asks IEM for the previous two days and the closed sources reuse
their sealed bytes without contacting NOAA. The table declares valid, so the plan meets the
multi-source condition as well and Studio admits it.
What a refresh buys here is a corrected observation, not a new row. The cycle is fixed and the
three lead hours never move, so a refresh that finds nothing revised ends unchanged. Once the
scored hours fall outside the two-day lookback, every refresh ends that way. To follow later model
cycles, register new sources for each cycle. You can also use one source with the cycle date in its
path template, then configure window.request with location: "path". See Source kinds and
connectors for the window fields.