mostlyright.weather.cache

Local parquet cache for Mostly Right weather observations and climate.

Repeated research() calls over the same stable date range use local parquet files instead of refetching data from IEM, AWC, or GHCNh.

Path layout:

$HOME/.mostlyright/cache/v1/observations/<STATION>/<YYYY>/<MM>.parquet
$HOME/.mostlyright/cache/v1/climate/<STATION>/<YYYY>.parquet

Override the root via the MOSTLYRIGHT_CACHE_DIR environment variable.

Safety guarantees: : - Atomic write: write to a sibling .tmp file inside a FileLock, then os.replace to the final path. A crash mid-write never leaves a truncated parquet at the read path.

• FileLock-guarded: two concurrent write_cache workers serialize on a .lock sidecar file. Verified by the multiprocess test.
• LST-current-month-skip: the current calendar month in the station’s Local Standard Time is mutable (observations still arriving). Writes to that (year, month) are no-ops, and reads return None even if a stale file exists. Climate cache applies the same rule at year granularity. This prevents serving incomplete data on re-runs.

Parquet options: : Every write uses version="2.6" and coerce_timestamps="us" so the byte output is stable across pyarrow versions and microsecond-resolution timestamps survive roundtrip without nanosecond inflation.

This module is parser-agnostic — it operates on already-parsed list[dict] rows. Importing a parser (_iem, _awc, etc.) would couple cache failures to fetcher failures; keeping the cache pure lets research() swap fetchers without touching the cache layer.

Functions

Function	Description
cache_path(station, year, month)	Return the parquet cache path for the (station, year, month) tuple.
climate_cache_path(station, year)	Return the parquet cache path for annual climate data.
forecast_cache_path(station, source, model, …)	Return the parquet cache path for the (station, source, model, year, month) tuple.
invalidate(station, year, month)	Remove the cache entry for (station, year, month).
invalidate_climate(station, year)	Remove the climate cache entry for (station, year).
invalidate_forecast(station, source, model, …)	Remove a forecast cache partition if it exists; return whether removed.
invalidate_satellite(satellite, product, …)	Remove a satellite cache partition if it exists; return whether removed.
read_cache(station, year, month, *[, …])	Return cached observation rows for (station, year, month) or None.
read_climate_cache(station, year, *[, …])	Return cached climate rows for (station, year) or None.
read_forecast_cache(station, source, model, …)	Return cached forecast rows for the partition key or None on miss.
read_satellite_cache(satellite, product, …)	Return cached satellite rows for the partition key or None on miss.
satellite_cache_path(satellite, product, …)	Return the parquet cache path for a satellite partition (mirror-invariant).
write_cache(station, year, month, rows, *[, …])	Atomically write rows to the observation cache.
write_climate_cache(station, year, rows, *)	Atomically write rows to the annual climate cache.
write_forecast_cache(station, source, model, …)	Atomically write rows to the forecast cache partition.
write_satellite_cache(satellite, product, …)	Atomically write rows to the satellite cache partition.

mostlyright.weather.cache.cache_path(station, year, month)

Return the parquet cache path for the (station, year, month) tuple.

Example:

cache_path("KNYC", 2025, 1)
# -> Path("$HOME/.mostlyright/cache/v1/observations/KNYC/2025/01.parquet")

The month is zero-padded to two digits so a lexicographic directory listing matches chronological order.

Validates station against STATION_CODE_RE and asserts the resolved path stays under the cache root (path-traversal backstop).

• Return type: Path
• Parameters:
  • station (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.climate_cache_path(station, year)

Return the parquet cache path for annual climate data.

Example:

climate_cache_path("KNYC", 2025)
# -> Path("$HOME/.mostlyright/cache/v1/climate/KNYC/2025.parquet")

Same validation contract as cache_path().

• Return type: Path
• Parameters:
  • station (str)
  • year (int)

mostlyright.weather.cache.forecast_cache_path(station, source, model, year, month)

Return the parquet cache path for the (station, source, model, year, month) tuple.

Layout:

~/.mostlyright/cache/v1/forecasts/{source}/{model}/{station}/{YYYY}/{MM}.parquet

Partition by issued_at cycle month (immutable once published).

• Return type: Path
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.invalidate(station, year, month)

Remove the cache entry for (station, year, month).

Returns True if a file was removed, False if the file did not exist. Acquires the same FileLock as write_cache so an invalidation racing a write either runs strictly before or strictly after — never mid-rename.

• Return type: bool
• Parameters:
  • station (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.invalidate_climate(station, year)

Remove the climate cache entry for (station, year).

Returns True if removed, False if absent.

• Return type: bool
• Parameters:
  • station (str)
  • year (int)

mostlyright.weather.cache.invalidate_forecast(station, source, model, year, month)

Remove a forecast cache partition if it exists; return whether removed.

• Return type: bool
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.invalidate_satellite(satellite, product, station, year, month, , cache_root=None)

Remove a satellite cache partition if it exists; return whether removed.

cache_root matches the read/write override so a partition written under a --out directory can be invalidated from there; None preserves the default resolution.

• Return type: bool
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • cache_root (Path | None)

mostlyright.weather.cache.read_cache(station, year, month, , tz_fallback=None)

Return cached observation rows for (station, year, month) or None.

• Return type: list[dict] | None
• Parameters:
  • station (str)
  • year (int)
  • month (int)
  • tz_fallback (str | None)

Returns None when: : - the cache file does not exist

• (year, month) is the station’s current LST month (file may be stale)
• a concurrent invalidate() removes the file between the exists() check and the read (treated as cache-miss; the caller re-fetches transparently)

Returns list[dict] otherwise. The list is materialised eagerly from pyarrow - callers can iterate freely without holding a file handle.

tz_fallback: the caller’s resolved StationInfo.timezone, used only when neither the built-in tz map / fetchable registry nor the bundled catalog places station. Orchestrator call sites pass info.timezone so a station reachable only through the documented timezone= override still gets a correct current-LST-month gate instead of raising.

mostlyright.weather.cache.read_climate_cache(station, year, , tz_fallback=None)

Return cached climate rows for (station, year) or None.

• Return type: list[dict] | None
• Parameters:
  • station (str)
  • year (int)
  • tz_fallback (str | None)

Returns None when: : - the cache file does not exist

• year is the station’s current LST year (file may be stale)
• a concurrent invalidate_climate() removes the file between the exists() check and the read (cache-miss semantics; caller re-fetches transparently)

tz_fallback: see read_cache().

mostlyright.weather.cache.read_forecast_cache(station, source, model, year, month)

Return cached forecast rows for the partition key or None on miss.

• Return type: list[dict] | None
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)

Returns None when: : - the partition file does not exist

• source is live or seamless (never cached)
• (year, month) is the current UTC month (cycles may still publish)

mostlyright.weather.cache.read_satellite_cache(satellite, product, station, year, month, , cache_root=None)

Return cached satellite rows for the partition key or None on miss.

Returns None when the partition does not exist or (year, month) is the current UTC month (the same gate the forecast tier uses; the current month may still receive scans). cache_root optionally overrides the root so a partition written under a --out directory reads back from there; when None the default resolution is byte-for-byte unchanged.

• Return type: list[dict] | None
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • cache_root (Path | None)

mostlyright.weather.cache.satellite_cache_path(satellite, product, station, year, month, , cache_root=None)

Return the parquet cache path for a satellite partition (mirror-invariant).

Validates every user-controlled path segment (the forecast equivalent validates only station):

1. station via validate_site_id_for_path() (a 4-12 character site id: a station ICAO, or a coordinate-derived identifier). ICAO is a strict subset of that alphabet, so every existing partition path is byte-unchanged. This applies to the satellite tier only — the observation and forecast tiers keep the strict ICAO validator.
2. satellite must be a registered native-ring satellite (the whole ring goes16/17/18/19 | himawari8/9 | viirs-* | meteosat-*, per the per-source registry) and contain no path separator.
3. product must be a registered product for satellite’s owning source (per-source registry — e.g. an ABI product is invalid on a Himawari satellite) and contain no path separator.

Then assert_path_under() is the final path-traversal backstop. The path carries no mirror segment; the mirror is transport-only.

cache_root optionally overrides the resolved cache root. The backfill CLI threads its --out directory here so the parquet partition lands under --out rather than the home/env cache root. When None the behavior is byte-for-byte unchanged: _cache_root() resolves the root. The path-traversal backstop validates against whichever root is in effect.

• Return type: Path
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • cache_root (Path | None)

mostlyright.weather.cache.write_cache(station, year, month, rows, , source=None, tz_fallback=None)

Atomically write rows to the observation cache.

• Return type: None
• Parameters:
  • station (str)
  • year (int)
  • month (int)
  • rows (list [dict ])
  • source (str | None)
  • tz_fallback (str | None)

No-op (does NOT raise) when: : - (year, month) is the station’s current LST month

• source ends with .live — live endpoints are never cached

The source kwarg is optional. The standard call site (research()) passes the fetcher’s endpoint identifier (e.g. "iem.asos", "awc.live") so the cache can gate writes uniformly without each fetcher having its own no-cache branch.

tz_fallback: see read_cache(). The written bytes are unaffected — the fallback only feeds the current-LST-month gate.

mostlyright.weather.cache.write_climate_cache(station, year, rows, , source=None, tz_fallback=None)

Atomically write rows to the annual climate cache.

• Return type: None
• Parameters:
  • station (str)
  • year (int)
  • rows (list [dict ])
  • source (str | None)
  • tz_fallback (str | None)

No-op (does NOT raise) when: : - year is the station’s current LST year

• source ends with .live

tz_fallback: see read_cache(). The written bytes are unaffected — the fallback only feeds the current-LST-year gate.

mostlyright.weather.cache.write_forecast_cache(station, source, model, year, month, rows)

Atomically write rows to the forecast cache partition.

• Return type: None
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)
  • rows (list [dict ])

No-op (does NOT raise) when: : - source is live or seamless (never cached)

• (year, month) is the current UTC month (cycles may still publish)
• rows is empty

mostlyright.weather.cache.write_satellite_cache(satellite, product, station, year, month, rows, , cache_root=None)

Atomically write rows to the satellite cache partition.

No-op (does not raise) when rows is empty or (year, month) is the current UTC month. On merge into an existing partition, the entire read→concat→“_dedup_satellite_rows“ (first-seen-wins, mirror-invariant) →write sequence runs under a single FileLock acquisition, so two writers targeting the same (satellite, product, station, year, month) partition cannot lost-update each other: the second writer reads the first writer’s already-committed rows and merges on top of them rather than clobbering them. The inner write is the usual .tmp + os.replace atomic swap, with no staging directory.

Unlike the overwrite-only forecast tier (write_forecast_cache does not read-modify-write), this tier is a true read-modify-write merge, so the lock must span the read as well as the write — see _write_table_unlocked.

cache_root optionally overrides the cache root so the partition lands under a caller-supplied directory (the backfill CLI threads its --out here). When None the default _cache_root() resolution is byte-for-byte unchanged — the forecast/observation/climate tiers and all existing satellite-cache callers are unaffected.

• Return type: None
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • rows (list [dict ])
  • cache_root (Path | None)