Weather GRIB2

Select one field out of a weather model file and read its values at the places a recipe names.

The question

What two-metre air temperature did the 12 September 2026 00Z GFS run forecast for Denver and New York, six hours ahead?

The source

Publisher	NOAA NCEP, through the NOMADS GRIB filter
Address	https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f006&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos
Format	GRIB2, served as application/octet-stream
Cadence	four model cycles a day, and NOMADS keeps roughly the last ten days
Rights	output of a United States federal model, published without copyright, recorded as public_domain_asserted

The filter endpoint makes this buildable. A whole GFS 0.25 degree file runs to hundreds of megabytes and holds hundreds of messages. weather.grib2 takes a container bounded at 16 MiB and returns a table bounded at 16 MiB. Ask whether a source’s file and its decoded table fit, not which family opens it. Asking NOMADS for one level and one variable returns a small multi-message file that fits.

The dated address in this example stops resolving. NOMADS keeps about ten days. Substitute a current cycle. A forecast cycle’s file never changes once published, which is what closed: true says.

The recipe

recipe.json

{
  "dataset": {
    "name": "Surface temperature at named places, GFS forecast",
    "description": "Each row is one named place and one forecast valid time, carrying the two-metre air temperature the GFS model published for the grid point nearest that place."
  },
  "question": {
    "text": "What two-metre temperature did the 12 September 2026 00Z GFS run forecast for Denver and New York six hours ahead?"
  },
  "table": {
    "name": "gfs_temperature_2m",
    "description": "One row per named place, model run and valid time.",
    "grain": [
      "point_id",
      "run_time",
      "valid_time"
    ],
    "columns": [
      {
        "name": "point_id",
        "type": "string",
        "nullable": false,
        "description": "The name this recipe gave the place.",
        "presentation": {
          "chart": "top_values"
        }
      },
      {
        "name": "run_time",
        "type": "timestamp",
        "nullable": false,
        "description": "The model cycle the forecast came from.",
        "presentation": {
          "chart": "timeline",
          "bucket": "day"
        }
      },
      {
        "name": "valid_time",
        "type": "timestamp",
        "nullable": false,
        "description": "The moment the forecast value is for.",
        "presentation": {
          "chart": "timeline",
          "bucket": "day"
        }
      },
      {
        "name": "temperature_k",
        "type": "decimal",
        "nullable": true,
        "description": "Two-metre air temperature at the nearest grid point, as the model published it.",
        "presentation": {
          "chart": "histogram",
          "bins": 8
        }
      },
      {
        "name": "matched_grid_lat",
        "type": "decimal",
        "nullable": false,
        "description": "Latitude of the grid point that answered, so the attribution can be checked."
      },
      {
        "name": "matched_grid_lon",
        "type": "decimal",
        "nullable": false,
        "description": "Longitude of the grid point that answered."
      },
      {
        "name": "message_sha256",
        "type": "string",
        "nullable": false,
        "description": "Digest of the exact GRIB2 message the value was decoded from.",
        "presentation": {
          "chart": "none"
        }
      }
    ]
  },
  "sources": [
    {
      "name": "gfs_cycle_2026091200",
      "description": "The two-metre temperature field of one GFS 0.25 degree model cycle, as the NOMADS filter serves it.",
      "source_class": "user_url",
      "data_classification": "public",
      "locator": {
        "kind": "https_url",
        "display_locator": "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t00z.pgrb2.0p25.f006&lev_2_m_above_ground=on&var_TMP=on&dir=%2Fgfs.20260912%2F00%2Fatmos"
      },
      "rights_claim": {
        "claimed_basis": "public_domain_asserted",
        "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
        "claim_note": "Output of a United States federal weather model, published by NOAA NCEP without copyright."
      },
      "connector": {
        "adapter_id": "public.https",
        "credential_mode": "none",
        "origin": "https://nomads.ncep.noaa.gov",
        "parameters": [
          {
            "name": "reader.family_id",
            "value": "weather.grib2"
          },
          {
            "name": "reader.family_version",
            "value": "3.0.0"
          },
          {
            "name": "reader.decode_options",
            "value": "{\"admission_allowlist_version\":1,\"collection_profile_version\":1,\"points\":[{\"id\":\"denver\",\"latitude\":\"39.7392\",\"longitude\":\"-104.9903\"},{\"id\":\"new_york\",\"latitude\":\"40.7128\",\"longitude\":\"-74.0060\"}],\"selector\":{\"discipline\":0,\"parameter_category\":0,\"parameter_number\":0,\"run_time\":\"2026-09-12T00:00:00Z\",\"surface_type\":103,\"surface_value\":{\"denominator\":1,\"numerator\":2},\"valid_time\":\"2026-09-12T06:00:00Z\"},\"variable_name\":\"temperature_2m\"}"
          }
        ]
      },
      "limits": {
        "max_source_bytes": 3145728,
        "max_rows": 1000,
        "max_requests": 2
      },
      "closed": true
    }
  ],
  "transform": {
    "engine": "duckdb_sql",
    "steps": [
      {
        "step_id": "gfs_temperature_2m",
        "sql": "select point_id, cast(run_time as timestamptz) as run_time, cast(valid_time as timestamptz) as valid_time, try_cast(value as decimal(12, 4)) as temperature_k, try_cast(matched_grid_lat as decimal(12, 6)) as matched_grid_lat, try_cast(matched_grid_lon as decimal(12, 6)) as matched_grid_lon, message_sha256 from gfs_cycle_2026091200",
        "description": "Rename the Reader's value column and cast the all-text relation. The Reader emits one row per named place."
      }
    ]
  },
  "checks": [
    {
      "check_id": "one_row_per_place_and_time",
      "kind": "key_uniqueness",
      "enforcement": "required",
      "columns": [
        "point_id",
        "run_time",
        "valid_time"
      ]
    },
    {
      "check_id": "temperature_plausible",
      "kind": "value_range",
      "enforcement": "required",
      "columns": [
        "temperature_k"
      ],
      "min_value": "150",
      "max_value": "350"
    },
    {
      "check_id": "every_place_answered",
      "kind": "row_expectation",
      "enforcement": "required",
      "min_rows": 2,
      "max_rows": 2,
      "description": "Two places were named and the Reader emits one row per place."
    }
  ],
  "units": [
    {
      "column": "temperature_k",
      "unit": "K"
    }
  ],
  "timezone": "UTC"
}

Part by part

The Reader pin

weather.grib2@3.0.0 reads a bounded multi-message GRIB2 collection and selects one message. Versions 1.0.0 and 2.0.0 require one already-extracted message and refuse a multi-message GFS or HRRR file. 3.0.0 indexes the collection and selects, then reuses the version-2 decoder. Zero matches and multiple matches both refuse. An unselected record never crosses the native decode boundary.

The selector names the message by its own coordinates:

Setting	Value	What it means
discipline	0	meteorological products
parameter_category	0	temperature
parameter_number	0	temperature (TMP)
surface_type	103	a specified height above ground
surface_value	{"numerator": 2, "denominator": 1}	two metres
run_time	2026-09-12T00:00:00Z	the model cycle
valid_time	2026-09-12T06:00:00Z	the moment the value is for

surface_value is a numerator and a denominator because the document carries no fractional JSON number. The 3.0.0 selector must resolve to a whole level, because the version-2 decoder it reuses takes one.

points name the places. Their coordinates are decimal text rather than numbers, for the same reason: "latitude": "39.7392". The record carries the digits written there, identically on every host. Each point needs an id, and two points may not share one.

admission_allowlist_version and collection_profile_version pin the approved lists this recipe was written against. A recipe frozen before a list grew never starts opening files it used to refuse.

What the Reader returns

One row per named place, in the order the document names them. Version 2 and 3 emit nine columns:

point_id, variable, level, run_time, valid_time, value, matched_grid_lat, matched_grid_lon, message_sha256.

matched_grid_lat and matched_grid_lon are the coordinates of the cell that answered. A reader can measure how far the answering point sits from the place asked for, without re-running anything.

Where a value comes from

A model holds values on a mesh a few kilometres apart. The value returned is the one at the mesh point nearest the named place.

A place exactly between two points takes the more southerly one, and the more westerly. The rule uses compass directions because one model’s files sometimes start at the north edge and sometimes at the south. The answer then depends on where the place is, not on which way round the publisher wrote the file.

The Reader refuses a place outside the area a file covers, naming the place and the four corners of the area. It does not fall back to the closest point on the edge. A reading from four hundred kilometres away is still a number, and it would arrive with nothing to say it came from somewhere else.

Every read recomputes the file’s own stated corners and compares them with the publisher’s. A disagreement beyond the file’s own precision halts the refresh.

Transform

One step: rename value to the column this table declares, and cast the all-text relation.

select point_id,
       cast(run_time as timestamptz) as run_time,
       cast(valid_time as timestamptz) as valid_time,
       try_cast(value as decimal(12, 4)) as temperature_k,
       …
from gfs_cycle_2026091200

The Reader writes times as YYYY-MM-DDTHH:MM:SSZ, which casts to TIMESTAMP WITH TIME ZONE directly.

Checks

row_expectation with min_rows: 2 and max_rows: 2 matches the two named places, because the Reader emits one row per place. Any other count means the selection or the decode did not do what the document says. value_range between "150" and "350" bounds a kelvin temperature and catches a field that turned out to be something other than temperature.

Cadence

closed: true says the bytes behind this address never change. It authorizes predecessor reuse in a refresh plan, and the per-source report then shows zero bytes fetched and names the rule that closed it. It does not by itself make this table refresh. Every source here is closed, so the plan has nothing to acquire and Studio returns RESYNC_REQUIRED before acquisition. One finished cycle is a finished table, and this recipe is complete once its full run has sealed a version.

A table that follows the model forward is many sources, one per cycle, unioned. The other shape is one source whose address carries the cycle date in a path template and declares a window with location: "path". That shape refreshes. closed and window are exclusive, and a source states at most one.

Register, run, inspect

mr-data dataset create --name "Surface temperature at named places, GFS forecast" --json
mr-data recipe recipe.json --json

mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --sample --max-rows 2000 --json
mr-data peek RUN_ID --json
mr-data checks RUN_ID --json
mr-data receipt RUN_ID --json

mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json

What the run reports

The source receipt records the Reader name and version, the validated options digest, the raw source digest and the row digest. message_sha256 on every row names the GRIB2 message the value came out of, so a reader can trace a row back to its bytes.

A point the model marks as having no value is still a row, under the document’s name for the place, with the value left blank. The Reader never writes a stand-in number like 9999, which would read as a temperature. The build records that it met one, so a blank cell means the publisher marked the value absent.

Traps

The gate is the grid and the packing together. Placing values correctly and unpacking them are separate abilities, and one alone is not enough. A model that changes its grid halts the next refresh before a single value is read. Models do change grids, usually at a version upgrade. The refusal names the edit and the sample file that would accept the new combination. The check costs a few hundred bytes.

The Reader refuses a lossy codec, and approving one more method does not change that. A stored dataset says its numbers are the publisher’s numbers, and a codec free to invent them cannot support the claim. The file has to be republished losslessly.

The Reader refuses a file that claims to be enormous before it allocates anything. A message states how many grid points it holds before it holds any of them, and a two-hundred-byte file can claim two million points. The limit is on the claim.

The document names the field as well as the places. A file that holds a different field stops the build, and the message says what it found and what the document asked for. Two fields from one model look identical from the outside, both real numbers on the same mesh at the same moment. A table labelled with one field name and carrying the other is wrong in a way nothing downstream can see.

A Reader-pinned source runs on the batch shape, not the warm one. A Reader needs a memory ceiling of its own, and the warm container gives a child none. The worker refuses a Reader-pinned source there before fetching anything, and names that reason.

The weather family is a lazy optional dependency. First use without a compatible binding answers READER_DEPENDENCY_UNAVAILABLE, a typed refusal rather than a crash.