Twenty stations, one hourly table
Union twenty weather stations into one hourly table, and give each source its own byte ceiling.
The question
Section titled “The question”What did each of twenty US airport stations report for temperature and dew point in each hour?
One row per station per hour. The stations file a routine report every hour and a special report whenever conditions change, so an hour holds one row or several. The table keeps the last reading of each hour and counts the reports behind it.
One address serves one station, so twenty stations are twenty sources. A recipe holds up to 256 of them. They share a publisher, a rights claim, a window and a shape, and the transform puts them together.
The sources
Section titled “The sources”| Publisher | Iowa Environmental Mesonet, Iowa State University |
| Addresses | …/request/asos.py?station=ATL&data=tmpf&data=dwpf&…, and the same for nineteen more station codes |
| Format | comma-separated text, served as text/plain; charset=UTF-8 |
| Cadence | one routine report per station per hour, plus specials |
| Rights | United States federal weather observations, redistributed by IEM, recorded as public_domain_asserted |
| Why | one open address per station, one identifier the rows already carry, and no credential |
The stations are ATL, AUS, BOS, DCA, DEN, DFW, HOU, LAS, LAX, MDW, MIA, MSP, MSY, NYC, OKC, PHL,
PHX, SAT, SEA and SFO. report_type=3 asks for the routine reports and report_type=4 for the
specials. Each address names data=tmpf&data=dwpf alone, so the decoded relation is
station,valid,tmpf,dwpf for every one of the twenty.
The recipe
Section titled “The recipe”{ "dataset": { "name": "US airport temperature and dew point history", "description": "Each row is one station hour at one of twenty US airport weather stations, carrying the last temperature and dew point that station reported inside the hour and how many reports the hour held, from the Iowa Environmental Mesonet ASOS archive." }, "question": {"text": "What did each of twenty US airport stations report for temperature and dew point in each hour?"}, "table": { "name": "hourly_station_weather", "description": "One row per station and UTC hour, from January 2026 onward.", "grain": ["station", "valid"], "columns": [ { "name": "city", "type": "string", "nullable": false, "description": "The city the reporting station serves.", "presentation": {"chart": "top_values"} }, { "name": "station", "type": "string", "nullable": false, "description": "The ASOS station identifier the reports came from.", "presentation": {"chart": "top_values"} }, { "name": "valid", "type": "timestamp", "nullable": false, "description": "The UTC hour the reports fall in. The acquired relation names its report-time column valid, and the table declares the hour under that name so a refresh can partition by it.", "presentation": {"chart": "timeline", "bucket": "hour", "story": "Every station reports in every hour, so the twenty series run level."} }, { "name": "temp_f", "type": "decimal", "nullable": true, "description": "The last air temperature the station reported inside the hour.", "presentation": {"chart": "histogram", "bins": 10, "story": "Twenty cities at once spread the histogram wider than any single station does."} }, { "name": "dewpoint_f", "type": "decimal", "nullable": true, "description": "The last dew point the station reported inside the hour.", "presentation": {"chart": "histogram", "bins": 10} }, { "name": "reports", "type": "integer", "nullable": false, "description": "How many reports the station filed inside the hour, routine and special together.", "presentation": {"chart": "histogram", "bins": 6, "story": "A quiet hour holds one report and a changing one holds several."} } ] }, "sources": [ { "name": "asos_atl", "description": "Temperature and dew point reports from Hartsfield-Jackson Atlanta International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=ATL&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_aus", "description": "Temperature and dew point reports from Austin-Bergstrom International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=AUS&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_bos", "description": "Temperature and dew point reports from Boston Logan International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=BOS&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_dca", "description": "Temperature and dew point reports from Ronald Reagan Washington National.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DCA&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_den", "description": "Temperature and dew point reports from Denver International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DEN&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_dfw", "description": "Temperature and dew point reports from Dallas-Fort Worth International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=DFW&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_hou", "description": "Temperature and dew point reports from Houston William P. Hobby.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=HOU&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_las", "description": "Temperature and dew point reports from Harry Reid International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=LAS&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_lax", "description": "Temperature and dew point reports from Los Angeles International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=LAX&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_mdw", "description": "Temperature and dew point reports from Chicago Midway International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MDW&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_mia", "description": "Temperature and dew point reports from Miami International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MIA&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_msp", "description": "Temperature and dew point reports from Minneapolis-Saint Paul International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MSP&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_msy", "description": "Temperature and dew point reports from Louis Armstrong New Orleans International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=MSY&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_nyc", "description": "Temperature and dew point reports from New York Central Park.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=NYC&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_okc", "description": "Temperature and dew point reports from Will Rogers World.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=OKC&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_phl", "description": "Temperature and dew point reports from Philadelphia International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=PHL&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_phx", "description": "Temperature and dew point reports from Phoenix Sky Harbor International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=PHX&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_sat", "description": "Temperature and dew point reports from San Antonio International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=SAT&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_sea", "description": "Temperature and dew point reports from Seattle-Tacoma International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=SEA&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } }, { "name": "asos_sfo", "description": "Temperature and dew point reports from San Francisco International.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=SFO&data=tmpf&data=dwpf&year1=2026&month1=1&day1=1&year2=2026&month2=1&day2=2&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 4194304, "max_rows": 400000, "max_requests": 2}, "window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]} } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "reports", "sql": "select 'Atlanta' as city, station, valid, tmpf, dwpf from asos_atl union all select 'Austin' as city, station, valid, tmpf, dwpf from asos_aus union all select 'Boston' as city, station, valid, tmpf, dwpf from asos_bos union all select 'Washington' as city, station, valid, tmpf, dwpf from asos_dca union all select 'Denver' as city, station, valid, tmpf, dwpf from asos_den union all select 'Dallas-Fort Worth' as city, station, valid, tmpf, dwpf from asos_dfw union all select 'Houston' as city, station, valid, tmpf, dwpf from asos_hou union all select 'Las Vegas' as city, station, valid, tmpf, dwpf from asos_las union all select 'Los Angeles' as city, station, valid, tmpf, dwpf from asos_lax union all select 'Chicago' as city, station, valid, tmpf, dwpf from asos_mdw union all select 'Miami' as city, station, valid, tmpf, dwpf from asos_mia union all select 'Minneapolis-Saint Paul' as city, station, valid, tmpf, dwpf from asos_msp union all select 'New Orleans' as city, station, valid, tmpf, dwpf from asos_msy union all select 'New York City' as city, station, valid, tmpf, dwpf from asos_nyc union all select 'Oklahoma City' as city, station, valid, tmpf, dwpf from asos_okc union all select 'Philadelphia' as city, station, valid, tmpf, dwpf from asos_phl union all select 'Phoenix' as city, station, valid, tmpf, dwpf from asos_phx union all select 'San Antonio' as city, station, valid, tmpf, dwpf from asos_sat union all select 'Seattle' as city, station, valid, tmpf, dwpf from asos_sea union all select 'San Francisco' as city, station, valid, tmpf, dwpf from asos_sfo", "description": "Put the twenty stations into one relation and name the city each one serves." }, { "step_id": "hourly", "sql": "select city, station, substr(valid, 1, 13) as hour_text, cast(arg_max(try_cast(tmpf as decimal(6, 2)), valid) as decimal(6, 2)) as temp_f, cast(arg_max(try_cast(dwpf as decimal(6, 2)), valid) as decimal(6, 2)) as dewpoint_f, cast(count(*) as integer) as reports from reports where valid is not null and valid <> '' group by city, station, substr(valid, 1, 13)", "description": "Reduce the reports to one row per station and hour, keeping the last reading of the hour." }, { "step_id": "hourly_station_weather", "sql": "select city, station, cast(concat(hour_text, ':00:00+00') as timestamp with time zone) as valid, temp_f, dewpoint_f, reports from hourly", "description": "Read the hour as a UTC instant and return it under the name the window partitions on." } ] }, "checks": [ { "check_id": "one_row_per_station_hour", "kind": "key_uniqueness", "enforcement": "required", "columns": ["station", "valid"], "description": "One row per station per UTC hour." }, { "check_id": "temperature_present", "kind": "null_ceiling", "enforcement": "required", "columns": ["temp_f"], "max_null_ppm": 20000 }, { "check_id": "temperature_plausible", "kind": "value_range", "enforcement": "required", "columns": ["temp_f"], "min_value": "-60", "max_value": "140" }, { "check_id": "hours_present", "kind": "row_expectation", "enforcement": "required", "min_rows": 1 }, { "check_id": "no_hour_without_any_station", "kind": "continuity", "enforcement": "advisory", "columns": ["valid"], "max_gap_seconds": 7200, "description": "A gap here means every one of the twenty stations went quiet, which is an outage at the publisher rather than at a station." } ], "units": [{"column": "temp_f", "unit": "[degF]"}, {"column": "dewpoint_f", "unit": "[degF]"}], "timezone": "UTC"}Part by part
Section titled “Part by part”Twenty sources of one shape
Section titled “Twenty sources of one shape”Each source differs in three places: its name, the station= pair in its address, and the
sentence in its description. The rest repeats. Registration returns one source_id for
each, and the receipt reports each one separately.
A recipe holds at most 256 sources. What twenty costs is requests. Every run opens twenty fetches, and one refused source ends the run.
Per-source byte ceilings
Section titled “Per-source byte ceilings”limits.max_source_bytes is required on every source. Exceeding it fails every mode, including
sample, because bytes already across the wire cannot be truncated. A run over
it fails CLAMP_EXCEEDED and names the clamp and the value that crossed it.
Size it from a measurement rather than a habit. One station-day of this projection is 25 lines and
816 bytes, so a station-year is under a megabyte. 4194304 leaves room for a year of special
reports and for a station that files more of them than the rest.
max_rows clamps the merged relation. max_requests bounds the requests one source may spend.
mr-data run --max-source-bytes N narrows every source of the run at once, and the document’s own
ceiling still applies to each. The rest of the bounds are on
Limits and ceilings.
The window
Section titled “The window”"window": { "start_at": "2026-01-01T00:00:00Z", "granularity": "day", "timezone": "UTC", "lookback_seconds": 172800, "max_span_seconds": 34560000, "request": { "start": {"encoding": "date_parts", "bound": "inclusive", "pad": "none", "parameters": {"year": "year1", "month": "month1", "day": "day1"}}, "end": {"encoding": "date_parts", "bound": "exclusive", "pad": "none", "parameters": {"year": "year2", "month": "month2", "day": "day2"}} }, "merge": {"materialization": "partition_replace", "partition": {"column": "valid", "key": "iso_date_prefix"}, "row_identity": ["station", "valid"]}}granularity is what the address can spell, not what the table’s grain is. This address carries a
year, a month and a day, so a refresh asks for whole days. The transform buckets those days into
hours afterwards.
All twenty sources carry the same window, because all twenty addresses spell time the same way.
start_at is the first UTC midnight of the corpus, and the registered address needs query literals
that render the same instant, so it says year1=2026&month1=1&day1=1. pad: "none" renders
month1=1 rather than month1=01, which is how this publisher writes it.
lookback_seconds: 172800 re-requests the two days before the last committed boundary, so a
corrected report lands. max_span_seconds: 34560000 is 400 days, and the engine clips a run that
fell further behind rather than refusing it.
merge.partition.column is valid, a column of the acquired relation, and iso_date_prefix takes
the first ten ISO characters of that cell. row_identity is what the merged relation must be
unique on, one station and one report time.
The table declares a column named valid too, and that is what makes twenty windows refreshable
together. A refresh plan naming more than one source needs every window’s partition column on the
table, under the name the acquired relation uses. The transform carries the name through to the
last step rather than handing the table an observed_hour the window cannot find.
Transform
Section titled “Transform”The first step puts the twenty relations together and names the city each station serves:
select 'Atlanta' as city, station, valid, tmpf, dwpf from asos_atlunion allselect 'Austin' as city, station, valid, tmpf, dwpf from asos_ausunion all-- eighteen more branches, one per stationselect 'San Francisco' as city, station, valid, tmpf, dwpf from asos_sfoA per-source constant carries a source’s own identity into the table without a lookup file.
The second step reduces the reports to one row per station and hour:
select city, station, substr(valid, 1, 13) as hour_text, cast(arg_max(try_cast(tmpf as decimal(6, 2)), valid) as decimal(6, 2)) as temp_f, cast(arg_max(try_cast(dwpf as decimal(6, 2)), valid) as decimal(6, 2)) as dewpoint_f, cast(count(*) as integer) as reportsfrom reportswhere valid is not null and valid <> ''group by city, station, substr(valid, 1, 13)valid arrives as 2026-09-01 00:52. Its first thirteen characters are the hour. This step keeps
them as text under another name, because it still reads the full valid in four places. An output
alias of the same name would make a bare valid mean two things at once.
arg_max(tmpf, valid) takes the reading with the greatest report time in the group, which is the
last reading of the hour. IEM writes valid zero padded, so text order is time order.
count(*) counts every report in the hour, including one whose temperature did not parse.
The third step gives the hour its table name and its declared type:
select city, station, cast(concat(hour_text, ':00:00+00') as timestamp with time zone) as valid, temp_f, dewpoint_f, reportsfrom hourlyconcat(…, ':00:00+00') makes the text a UTC instant the cast accepts. Nothing else in this step
reads a column called valid, so the name is free to take.
Checks
Section titled “Checks”key_uniqueness on ["station", "valid"] is the grain, and it catches a merge that duplicated an
hour. value_range bounds temp_f between "-60" and "140" as decimal strings, because the
document carries no fractional JSON number. null_ceiling tolerates two per cent missing.
row_expectation guards against an empty answer.
The continuity check on valid is advisory, and it sees the union rather than a station.
Nineteen stations reporting normally fill the gap one silent station leaves. Ask for a station’s
own coverage with a query instead:
mr-data query RUN_ID "select station, count(*) as hours, min(valid) as first_hour, \ max(valid) as last_hour from run_table group by station order by hours" --jsonCadence
Section titled “Cadence”The first succeeded run goes live. mr-data promote TABLE_ID records how often the table refreshes
and why, and it does not make the table readable.
mr-data promote TABLE_ID --cadence "5 * * * *" \ --why "The stations file a routine report every hour, and IEM has it a few minutes later." --jsonTwenty windowed sources are one more condition. Where a refresh plan names more than one source,
every window’s merge.partition.column has to be a column the table declares, and all of them have
to agree on the materialization and the partition key. All twenty name valid with
partition_replace and iso_date_prefix, and the table declares valid. So the twenty windows
refresh together, and each refresh replaces the day partitions it re-requested and keeps the rest.
One window that disagreed with the other nineteen would refuse the whole plan, and so would a table
that stopped declaring valid. Studio answers RESYNC_REQUIRED before acquisition in both cases,
so the recipe would advance only by explicit resync.
The cadence is a seed. Studio watches what each source does on every refresh and moves the schedule
to match. --cadence source hands it the choice outright.
The worker compares the sha256 of each source’s acquired bytes against the predecessor receipt’s
digest for that source. It reports one observation per source on run:succeeded, and it stops
before the transform when nothing moved. Studio turns those observations into a schedule and
applies it without an agent.
mr-data table TABLE_ID --json reports the state in plain words, the schedule in force, the next
refresh, and the interval Studio measured once there is one. Do not call a schedule learned while
that command says otherwise.
mr-data reschedule TABLE_ID --cadence "5 * * * *" --why "…" --lock --json--lock keeps exactly that schedule and stops the loop adjusting it. It overrides the evidence, so
it is a person’s decision rather than an agent’s.
Publish and keep it fresh has the rest of the lifecycle.
When a source is closed
Section titled “When a source is closed”closed: true says the bytes behind an address cannot change again. In a strict refresh plan it
makes reuse_predecessor this source’s action, and a closed source rides beside a request window.
Adding the closed archive branches below keeps this recipe refreshable. The twenty windows still
agree on one partition layout, and each archive contributes its sealed bytes without a fetch. A
source that declares a window is never closed, because its end parameters are a placeholder the
engine rewrites on every run. The two members answer different questions and a source states at
most one of them.
Splitting the ended years off their own address is what makes the member writable:
{ "name": "asos_atl_archive", "description": "Temperature and dew point reports from Atlanta for 2020 through 2025.", "source_class": "user_api", "data_classification": "public", "closed": true, "locator": { "kind": "https_url", "display_locator": "https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=ATL&data=tmpf&data=dwpf&year1=2020&month1=1&day1=1&year2=2026&month2=1&day2=1&format=onlycomma&tz=Etc/UTC&report_type=3&report_type=4&latlon=no&missing=empty&trace=empty" }, "rights_claim": { "claimed_basis": "public_domain_asserted", "claim_evidence_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "claim_note": "United States federal weather observations, redistributed by the Iowa Environmental Mesonet." }, "connector": {"adapter_id": "public.https", "credential_mode": "none", "origin": "https://mesonet.agron.iastate.edu"}, "limits": {"max_source_bytes": 33554432, "max_rows": 2000000, "max_requests": 2}}Twenty of those take the recipe to forty sources and the table back to 2020. Each one adds a branch to the union step.
In a strict refresh plan, only a source explicitly declared closed: true reuses the predecessor’s
rows. A publisher’s conditional 304 is an unchanged reply after acquisition, and the legacy
settled-source heuristic is not a reuse authorization. A source without an incremental strategy is
refused before Studio acquires it.
closed states something about the source, not about the run. It narrows no fetch and it never
stands in for a window.
Write closed only where it is true whenever anybody asks. Historical weather is corrected,
delayed and republished under the same dates, and the address is identical before and after.
Register, run, inspect
Section titled “Register, run, inspect”mr-data dataset create --name "US airport temperature and dew point history" --jsonmr-data dataset set DATASET_ID --category climate-environment \ --topics "weather,temperature,asos,hourly,united states" --license "CC0-1.0" --json
mr-data recipe recipe.json --jsonmr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --sample --max-rows 500 --jsonmr-data peek RUN_ID --jsonmr-data checks RUN_ID --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --jsonmr-data query RUN_ID "select city, count(*) from run_table group by city" --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --refresh --jsonWhat the run reports
Section titled “What the run reports”mr-data receipt RUN_ID --json carries one entry per source, so twenty here. Each entry names the
rendered address, fetched bytes, byte digest and coverage disposition.
On a refresh the per-source report names the rule that closed each source. Zero bytes fetched under a named rule is a refresh working.
A refresh that gained nothing produces bytes identical to the predecessor’s, seals nothing, uploads
nothing, and ends unchanged.
mr-data checks RUN_ID --json prints one result per declared check, with the recipe’s own
check_id upper-cased, whether it passed, and what the worker measured in words.
A refresh replaces the partitions the window asked for, so each check states which rows its answer
covers. row_expectation counts the retained parts plus this run’s rows. null_ceiling sums the
null count each retained part already records. value_range holds over the replacement rows, and
the predecessor’s own evaluated pass covers the rows this run did not touch. key_uniqueness runs
over the successor manifest’s part bounds. continuity reads a column in order across the
whole table and has no such decomposition, so a refresh reports it not_evaluated. A
not_evaluated result never ends a run, including a required one.
--max-rows applies per source. Twenty sources cut at 500 rows each deliver up to 10,000 rows.
The engine stitches those partial spans into one table with seams that do not exist in the data.
A continuity check that passes over the whole table fails over that sample. Measure the sources,
then set the ceiling above the largest.
Twenty sources are twenty chances of a transient refusal. A source refused at the acquisition
boundary fails the whole run with ACQUISITION_FAILED, and the detail names the source and the
reason. The next scheduled refresh starts from where the last successful one reached, so a single
failure costs a cycle rather than a day. Fix a failed run walks the
common refusals.
The uniqueness check does not catch a mislabelled city. city is a constant in the union step and
it is not in the grain. A typo in one branch produces a table that passes every check and names the
wrong place. Read select distinct city, station from run_table once after the first run.
A grouped column must appear in group by in the same spelling. substr(valid, 1, 13) appears in
both the select list and the grouping of the second step, not the alias hour_text it is written
under.
Each declared window parameter name must occur once in the query. The engine refuses an address
where year1 appears twice or not at all. The address can repeat pairs that the window does not
declare. This address repeats both data and report_type.