# Twenty stations, one hourly table Source: https://mostlyright.md/docs/recipes/many-station-weather/ Union twenty weather stations into one hourly table, and give each source its own byte ceiling. ## The question What did each of twenty US airport stations report for temperature and dew point in each hour? One row per station per hour. The stations file a routine report every hour and a special report whenever conditions change, so an hour holds one row or several. The table keeps the last reading of each hour and counts the reports behind it. One address serves one station, so twenty stations are twenty sources. A recipe holds up to 256 of them. They share a publisher, a rights claim, a window and a shape, and the transform puts them together. ## The sources | | | | --- | --- | | Publisher | Iowa Environmental Mesonet, Iowa State University | | Addresses | `…/request/asos.py?station=ATL&data=tmpf&data=dwpf&…`, and the same for nineteen more station codes | | Format | comma-separated text, served as `text/plain; charset=UTF-8` | | Cadence | one routine report per station per hour, plus specials | | Rights | United States federal weather observations, redistributed by IEM, recorded as `public_domain_asserted` | | Why | one open address per station, one identifier the rows already carry, and no credential | The stations are ATL, AUS, BOS, DCA, DEN, DFW, HOU, LAS, LAX, MDW, MIA, MSP, MSY, NYC, OKC, PHL, PHX, SAT, SEA and SFO. `report_type=3` asks for the routine reports and `report_type=4` for the specials. Each address names `data=tmpf&data=dwpf` alone, so the decoded relation is `station,valid,tmpf,dwpf` for every one of the twenty. ## The recipe ```json title="recipe.json" { "dataset": { "name": "US airport temperature and dew point history" }, "question": {"text": "What did each of twenty US airport stations report for temperature and dew point in each hour?"}, "table": { "name": "hourly_station_weather", "description": "One row per station and UTC hour, from January 2026 onward.", "grain": ["station", "valid"], "columns": [ { "name": "city", "type": "string", "nullable": false, "description": "The city the reporting station serves.", "presentation": {"chart": "top_values"} }, { "name": "station", "type": "string", "nullable": false, "description": "The ASOS station identifier the reports came from.", "presentation": {"chart": "top_values"} }, { "name": "valid", "type": "timestamp", "nullable": false, "description": "The UTC hour the reports fall in. The acquired relation names its report-time column valid, and the table declares the hour under that name so a refresh can partition by it.", "presentation": {"chart": "timeline", "bucket": "hour", "story": "Every station reports in every hour, so the twenty series run level."} }, { "name": "temp_f", "type": "decimal", "nullable": true, "description": "The last air temperature the station reported inside the hour.", "presentation": {"chart": "histogram", "bins": 10, "story": "Twenty cities at once spread the histogram wider than any single station does."} }, { "name": "dewpoint_f", "type": "decimal", "nullable": true, "description": "The last dew point the station reported inside the hour.", "presentation": {"chart": "histogram", "bins": 10} }, { "name": "reports", "type": "integer", "nullable": false, "description": "How many reports the station filed inside the hour, routine and special together.", "presentation": {"chart": "histogram", "bins": 6, "story": "A quiet hour holds one report and a changing one holds several."} } ] }, "sources": [ { "name": "asos_atl", "description": "Temperature and dew point reports from Hartsfield-Jackson Atlanta International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=ATL&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_aus", "description": "Temperature and dew point reports from Austin-Bergstrom International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=AUS&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_bos", "description": "Temperature and dew point reports from Boston Logan International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=BOS&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_dca", "description": "Temperature and dew point reports from Ronald Reagan Washington National.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DCA&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_den", "description": "Temperature and dew point reports from Denver International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DEN&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_dfw", "description": "Temperature and dew point reports from Dallas-Fort Worth International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DFW&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_hou", "description": "Temperature and dew point reports from Houston William P. Hobby.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=HOU&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_las", "description": "Temperature and dew point reports from Harry Reid International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=LAS&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_lax", "description": "Temperature and dew point reports from Los Angeles International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=LAX&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_mdw", "description": "Temperature and dew point reports from Chicago Midway International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MDW&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_mia", "description": "Temperature and dew point reports from Miami International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MIA&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_msp", "description": "Temperature and dew point reports from Minneapolis-Saint Paul International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MSP&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_msy", "description": "Temperature and dew point reports from Louis Armstrong New Orleans International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MSY&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_nyc", "description": "Temperature and dew point reports from New York Central Park.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=NYC&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_okc", "description": "Temperature and dew point reports from Will Rogers World.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=OKC&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_phl", "description": "Temperature and dew point reports from Philadelphia International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=PHL&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_phx", "description": "Temperature and dew point reports from Phoenix Sky Harbor International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=PHX&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_sat", "description": "Temperature and dew point reports from San Antonio International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=SAT&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_sea", "description": "Temperature and dew point reports from Seattle-Tacoma International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=SEA&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_sfo", "description": "Temperature and dew point reports from San Francisco International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=SFO&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "reports", "sql": "select 'Atlanta' as city, station, valid, tmpf, dwpf from asos_atl union all select 'Austin' as city, station, valid, tmpf, dwpf from asos_aus union all select 'Boston' as city, station, valid, tmpf, dwpf from asos_bos union all select 'Washington' as city, station, valid, tmpf, dwpf from asos_dca union all select 'Denver' as city, station, valid, tmpf, dwpf from asos_den union all select 'Dallas-Fort Worth' as city, station, valid, tmpf, dwpf from asos_dfw union all select 'Houston' as city, station, valid, tmpf, dwpf from asos_hou union all select 'Las Vegas' as city, station, valid, tmpf, dwpf from asos_las union all select 'Los Angeles' as city, station, valid, tmpf, dwpf from asos_lax union all select 'Chicago' as city, station, valid, tmpf, dwpf from asos_mdw union all select 'Miami' as city, station, valid, tmpf, dwpf from asos_mia union all select 'Minneapolis-Saint Paul' as city, station, valid, tmpf, dwpf from asos_msp union all select 'New Orleans' as city, station, valid, tmpf, dwpf from asos_msy union all select 'New York City' as city, station, valid, tmpf, dwpf from asos_nyc union all select 'Oklahoma City' as city, station, valid, tmpf, dwpf from asos_okc union all select 'Philadelphia' as city, station, valid, tmpf, dwpf from asos_phl union all select 'Phoenix' as city, station, valid, tmpf, dwpf from asos_phx union all select 'San Antonio' as city, station, valid, tmpf, dwpf from asos_sat union all select 'Seattle' as city, station, valid, tmpf, dwpf from asos_sea union all select 'San Francisco' as city, station, valid, tmpf, dwpf from asos_sfo", "description": "Put the twenty stations into one relation and name the city each one serves." }, { "step_id": "hourly", "sql": "select city, station, substr(valid, 1, 13) as hour_text, cast(arg_max(try_cast(tmpf as decimal(6, 2)), valid) as decimal(6, 2)) as temp_f, cast(arg_max(try_cast(dwpf as decimal(6, 2)), valid) as decimal(6, 2)) as dewpoint_f, cast(count(*) as integer) as reports from reports where valid is not null and valid <> '' group by city, station, substr(valid, 1, 13)", "description": "Reduce the reports to one row per station and hour, keeping the last reading of the hour." }, { "step_id": "hourly_station_weather", "sql": "select city, station, cast(concat(hour_text, ':00:00+00') as timestamp with time zone) as valid, temp_f, dewpoint_f, reports from hourly", "description": "Read the hour as a UTC instant and return it under the name the window partitions on." } ] }, "checks": [ { "check_id": "one_row_per_station_hour", "kind": "key_uniqueness", "enforcement": "required", "columns": ["station", "valid"], "description": "One row per station per UTC hour." }, { "check_id": "temperature_present", "kind": "null_ceiling", "enforcement": "required", "columns": ["temp_f"], "max_null_ppm": 20000 }, { "check_id": "temperature_plausible", "kind": "value_range", "enforcement": "required", "columns": ["temp_f"], "min_value": "-60", "max_value": "140" }, { "check_id": "hours_present", "kind": "row_expectation", "enforcement": "required", "min_rows": 1 }, { "check_id": "no_hour_without_any_station", "kind": "continuity", "enforcement": "advisory", "columns": ["valid"], "max_gap_seconds": 7200, "description": "A gap here means every one of the twenty stations went quiet, which is an outage at the publisher rather than at a station." } ], "units": [{"column": "temp_f", "unit": "[degF]"}, {"column": "dewpoint_f", "unit": "[degF]"}], "timezone": "UTC" } ``` ## Part by part ### Twenty sources of one shape Each source differs in three places: its `name`, the `station=` pair in its address, and the sentence in its `description`. The rest repeats. Registration returns one `source_id` for each, and the receipt reports each one separately. A recipe holds at most 256 sources. What twenty costs is requests. Every run opens twenty fetches, and one refused source ends the run. ### Per-source byte ceilings `limits.max_source_bytes` is required on every source. Exceeding it fails every mode, including `sample`, because bytes already across the wire cannot be truncated. A run over it fails `CLAMP_EXCEEDED` and names the clamp and the value that crossed it. Size it from a measurement rather than a habit. One station-day of this projection is 25 lines and 816 bytes, so a station-year is under a megabyte. `4194304` leaves room for a year of special reports and for a station that files more of them than the rest. `max_rows` clamps the merged relation. `max_requests` bounds the requests one source may spend. `mr-data run --max-source-bytes N` narrows every source of the run at once, and the document's own ceiling still applies to each. The rest of the bounds are on [Limits and ceilings](/docs/reference/limits/). ### The window ```json "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } ``` `granularity` is what the address can spell, not what the table's grain is. This address carries a year, a month and a day, so a refresh asks for whole days. The transform buckets those days into hours afterwards. All twenty sources carry the same window, because all twenty addresses spell time the same way. `start_at` is the first UTC midnight of the corpus, and the registered address needs query literals that render the same instant, so it says `year1=2026&month1=1&day1=1`. `pad: "none"` renders `month1=1` rather than `month1=01`, which is how this publisher writes it. `lookback_seconds: 172800` re-requests the two days before the last committed boundary, so a corrected report lands. `max_span_seconds: 34560000` is 400 days, and the engine clips a run that fell further behind rather than refusing it. `merge.partition.column` is `valid`, a column of the acquired relation, and `iso_date_prefix` takes the first ten ISO characters of that cell. `row_identity` is what the merged relation must be unique on, one station and one report time. The table declares a column named `valid` too, and that is what makes twenty windows refreshable together. A refresh plan naming more than one source needs every window's partition column on the table, under the name the acquired relation uses. The transform carries the name through to the last step rather than handing the table an `observed_hour` the window cannot find. ### Transform The first step puts the twenty relations together and names the city each station serves: ```sql select 'Atlanta' as city, station, valid, tmpf, dwpf from asos_atl union all select 'Austin' as city, station, valid, tmpf, dwpf from asos_aus union all -- eighteen more branches, one per station select 'San Francisco' as city, station, valid, tmpf, dwpf from asos_sfo ``` A per-source constant carries a source's own identity into the table without a lookup file. The second step reduces the reports to one row per station and hour: ```sql select city, station, substr(valid, 1, 13) as hour_text, cast(arg_max(try_cast(tmpf as decimal(6, 2)), valid) as decimal(6, 2)) as temp_f, cast(arg_max(try_cast(dwpf as decimal(6, 2)), valid) as decimal(6, 2)) as dewpoint_f, cast(count(*) as integer) as reports from reports where valid is not null and valid <> '' group by city, station, substr(valid, 1, 13) ``` `valid` arrives as `2026-09-01 00:52`. Its first thirteen characters are the hour. This step keeps them as text under another name, because it still reads the full `valid` in four places. An output alias of the same name would make a bare `valid` mean two things at once. `arg_max(tmpf, valid)` takes the reading with the greatest report time in the group, which is the last reading of the hour. IEM writes `valid` zero padded, so text order is time order. `count(*)` counts every report in the hour, including one whose temperature did not parse. The third step gives the hour its table name and its declared type: ```sql select city, station, cast(concat(hour_text, ':00:00+00') as timestamp with time zone) as valid, temp_f, dewpoint_f, reports from hourly ``` `concat(…, ':00:00+00')` makes the text a UTC instant the cast accepts. Nothing else in this step reads a column called `valid`, so the name is free to take. ### Checks `key_uniqueness` on `["station", "valid"]` is the grain, and it catches a merge that duplicated an hour. `value_range` bounds `temp_f` between `"-60"` and `"140"` as decimal strings, because the document carries no fractional JSON number. `null_ceiling` tolerates two per cent missing. `row_expectation` guards against an empty answer. The `continuity` check on `valid` is `advisory`, and it sees the union rather than a station. Nineteen stations reporting normally fill the gap one silent station leaves. Ask for a station's own coverage with a query instead: ```sh mr-data query RUN_ID "select station, count(*) as hours, min(valid) as first_hour, \ max(valid) as last_hour from run_table group by station order by hours" --json ``` ### Cadence The first succeeded run goes live. `mr-data promote TABLE_ID` records how often the table refreshes and why, and it does not make the table readable. ```sh mr-data promote TABLE_ID --cadence "5 * * * *" \ --why "The stations file a routine report every hour, and IEM has it a few minutes later." --json ``` Twenty windowed sources are one more condition. Where a refresh plan names more than one source, every window's `merge.partition.column` has to be a column the table declares, and all of them have to agree on the materialization and the partition key. All twenty name `valid` with `partition_replace` and `iso_date_prefix`, and the table declares `valid`. So the twenty windows refresh together, and each refresh replaces the day partitions it re-requested and keeps the rest. One window that disagreed with the other nineteen would refuse the whole plan, and so would a table that stopped declaring `valid`. Studio answers `RESYNC_REQUIRED` before acquisition in both cases, so the recipe would advance only by explicit resync. The cadence is a seed. Studio watches what each source does on every refresh and moves the schedule to match. `--cadence source` hands it the choice outright. The worker compares the sha256 of each source's acquired bytes against the predecessor receipt's digest for that source. It reports one observation per source on `run:succeeded`, and it stops before the transform when nothing moved. Studio turns those observations into a schedule and applies it without an agent. `mr-data table TABLE_ID --json` reports the state in plain words, the schedule in force, the next refresh, and the interval Studio measured once there is one. Do not call a schedule learned while that command says otherwise. ```sh mr-data reschedule TABLE_ID --cadence "5 * * * *" --why "…" --lock --json ``` `--lock` keeps exactly that schedule and stops the loop adjusting it. It overrides the evidence, so it is a person's decision rather than an agent's. [Publish and keep it fresh](/docs/build/publish-and-refresh/) has the rest of the lifecycle. ### When a source is closed `closed: true` says the bytes behind an address cannot change again. In a strict refresh plan it makes `reuse_predecessor` this source's action, and a closed source rides beside a request window. Adding the closed archive branches below keeps this recipe refreshable. The twenty windows still agree on one partition layout, and each archive contributes its sealed bytes without a fetch. A source that declares a `window` is never closed, because its end parameters are a placeholder the engine rewrites on every run. The two members answer different questions and a source states at most one of them. Splitting the ended years off their own address is what makes the member writable: ```json { "name": "asos_atl_archive", "description": "Temperature and dew point reports from Atlanta for 2020 through 2025.", "source_class": "user_api", "data_classification": "public", "closed": true, "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=ATL&data=tmpf&data=dwpf&year1=2020&month1=1&day1=1&year2=2026&month2=1&day2=1&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 33554432, "max_rows": 2000000, "max_requests": 2} } ``` Twenty of those take the recipe to forty sources and the table back to 2020. Each one adds a branch to the union step. In a strict refresh plan, only a source explicitly declared `closed: true` reuses the predecessor's rows. A publisher's conditional `304` is an unchanged reply after acquisition, and the legacy settled-source heuristic is not a reuse authorization. A source without an incremental strategy is refused before Studio acquires it. `closed` states something about the source, not about the run. It narrows no fetch and it never stands in for a window. Write `closed` only where it is true whenever anybody asks. Historical weather is corrected, delayed and republished under the same dates, and the address is identical before and after. ## Register, run, inspect Use the real table for one full build. For a bounded test first, follow the [scratch-table workflow](/docs/build/run-and-inspect/#test-before-the-full-build). A successful sample on the real table can publish truncated data. Stop if the forecast refuses; size the recipe before running. ```sh mr-data dataset create --name "US airport temperature and dew point history" --json mr-data dataset set DATASET_ID --category climate-environment \ --topics "weather,temperature,asos,hourly,united states" --license "CC0-1.0" --json mr-data recipe recipe.json --json mr-data recipe explain RECIPE_ID --json mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json mr-data watch RUN_ID --json mr-data peek RUN_ID --json mr-data checks RUN_ID --json mr-data query RUN_ID "select city, count(*) from run_table group by city" --json mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --refresh --json ``` ## What the run reports `mr-data receipt RUN_ID --json` carries one entry per source, so twenty here. Each entry names the rendered address, fetched bytes, byte digest and coverage disposition. On a refresh the per-source report names the rule that closed each source. Zero bytes fetched under a named rule is a refresh working. A refresh that gained nothing produces bytes identical to the predecessor's, seals nothing, uploads nothing, and ends `unchanged`. `mr-data checks RUN_ID --json` prints one result per declared check, with the recipe's own `check_id` upper-cased, whether it passed, and what the worker measured in words. A refresh replaces the partitions the window asked for, so each check states which rows its answer covers. `row_expectation` counts the retained parts plus this run's rows. `null_ceiling` sums the null count each retained part already records. `value_range` holds over the replacement rows, and the predecessor's own evaluated pass covers the rows this run did not touch. `key_uniqueness` runs over the successor manifest's part bounds. `continuity` reads a column in order across the whole table and has no such decomposition, so a refresh reports it `not_evaluated`. A `not_evaluated` result never ends a run, including a required one. ## Traps `--max-rows` applies per source. Twenty sources cut at 500 rows each deliver up to 10,000 rows. The engine stitches those partial spans into one table with seams that do not exist in the data. A continuity check that passes over the whole table fails over that sample. Measure the sources, then set the ceiling above the largest. Twenty sources are twenty chances of a transient refusal. A source refused at the acquisition boundary fails the whole run with `ACQUISITION_FAILED`, and the detail names the source and the reason. The next scheduled refresh starts from where the last successful one reached, so a single failure costs a cycle rather than a day. [Fix a failed run](/docs/build/troubleshooting/) walks the common refusals. The uniqueness check does not catch a mislabelled city. `city` is a constant in the union step and it is not in the grain. A typo in one branch produces a table that passes every check and names the wrong place. Read `select distinct city, station from run_table` once after the first run. A grouped column must appear in `group by` in the same spelling. `substr(valid, 1, 13)` appears in both the select list and the grouping of the second step, not the alias `hour_text` it is written under. Each declared window parameter name must occur once in the query. The engine refuses an address where `year1` appears twice or not at all. The address can repeat pairs that the window does not declare. This address repeats both `data` and `report_type`.