mostlyright.weather.cache

mostlyright.weather.cache

Local parquet cache for mostlyright weather observations and climate.

NEW module (no v0.14.1 reference). v0.14.1 is a hosted-client SDK — merging and caching happen server-side there. The mostlyright local-first SDK ships caching client-side so re-runs of research() against the same date range hit a local parquet file instead of re-fetching from IEM/AWC/GHCNh.

Path layout (CACHE-01):

$HOME/.mostlyright/cache/v1/observations/<STATION>/<YYYY>/<MM>.parquet
$HOME/.mostlyright/cache/v1/climate/<STATION>/<YYYY>.parquet

Override the root via the MOSTLYRIGHT_CACHE_DIR environment variable.

Safety guarantees (CACHE-07): : - Atomic write: write to a sibling .tmp file inside a FileLock, then os.rename to the final path. A crash mid-write never leaves a truncated parquet at the read path.

• FileLock-guarded: two concurrent write_cache workers serialize on a .lock sidecar file. Verified by the multiprocess test.
• LST-current-month-skip: the current calendar month in the station’s Local Standard Time is mutable (observations still arriving). Writes to that (year, month) are no-ops, and reads return None even if a stale file exists. Climate cache applies the same rule at year granularity. This prevents serving incomplete data on re-runs.

Parquet options (CACHE-01): : Every write uses version="2.6" and coerce_timestamps="us" so the byte output is stable across pyarrow versions and microsecond-resolution timestamps survive roundtrip without nanosecond inflation.

This module is parser-agnostic — it operates on already-parsed list[dict] rows. Importing a parser (_iem, _awc, etc.) would couple cache failures to fetcher failures; keeping the cache pure lets research() swap fetchers without touching the cache layer.

Functions

cache_path(station, year, month)	Return the parquet cache path for the (station, year, month) tuple.
climate_cache_path(station, year)	Return the parquet cache path for annual climate data.
forecast_cache_path(station, source, model, …)	Return the parquet cache path for the (station, source, model, year, month) tuple per Phase 20 D-09.
invalidate(station, year, month)	Remove the cache entry for (station, year, month).
invalidate_climate(station, year)	Remove the climate cache entry for (station, year).
invalidate_forecast(station, source, model, …)	Remove a forecast cache partition if it exists; return whether removed.
invalidate_satellite(satellite, product, …)	Remove a satellite cache partition if it exists; return whether removed.
read_cache(station, year, month)	Return cached observation rows for (station, year, month) or None.
read_climate_cache(station, year)	Return cached climate rows for (station, year) or None.
read_forecast_cache(station, source, model, …)	Return cached forecast rows for the partition key or None on miss.
read_satellite_cache(satellite, product, …)	Return cached satellite rows for the partition key or None on miss.
satellite_cache_path(satellite, product, …)	Return the parquet cache path for a satellite partition (SAT-25-04, D8).
write_cache(station, year, month, rows, *[, …])	Atomically write rows to the observation cache.
write_climate_cache(station, year, rows, *)	Atomically write rows to the annual climate cache.
write_forecast_cache(station, source, model, …)	Atomically write rows to the forecast cache partition.
write_satellite_cache(satellite, product, …)	Atomically write rows to the satellite cache partition (D8).

mostlyright.weather.cache.cache_path(station, year, month)

Return the parquet cache path for the (station, year, month) tuple.

Example:

cache_path("KNYC", 2025, 1)
# -> Path("$HOME/.mostlyright/cache/v1/observations/KNYC/2025/01.parquet")

The month is zero-padded to two digits so a lexicographic directory listing matches chronological order.

Validates station against STATION_CODE_RE (Rob C1) and asserts the resolved path stays under the cache root (defense-in-depth backstop - Rob C1).

• Return type: Path
• Parameters:
  • station (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.climate_cache_path(station, year)

Return the parquet cache path for annual climate data.

Example:

climate_cache_path("KNYC", 2025)
# -> Path("$HOME/.mostlyright/cache/v1/climate/KNYC/2025.parquet")

Same validation contract as cache_path() (Rob C1).

• Return type: Path
• Parameters:
  • station (str)
  • year (int)

mostlyright.weather.cache.forecast_cache_path(station, source, model, year, month)

Return the parquet cache path for the (station, source, model, year, month) tuple per Phase 20 D-09.

Layout:

~/.mostlyright/cache/v1/forecasts/{source}/{model}/{station}/{YYYY}/{MM}.parquet

Partition by issued_at cycle month (immutable once published).

• Return type: Path
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.invalidate(station, year, month)

Remove the cache entry for (station, year, month).

Returns True if a file was removed, False if the file did not exist. Acquires the same FileLock as write_cache so an invalidation racing a write either runs strictly before or strictly after — never mid-rename.

• Return type: bool
• Parameters:
  • station (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.invalidate_climate(station, year)

Remove the climate cache entry for (station, year).

Returns True if removed, False if absent.

• Return type: bool
• Parameters:
  • station (str)
  • year (int)

mostlyright.weather.cache.invalidate_forecast(station, source, model, year, month)

Remove a forecast cache partition if it exists; return whether removed.

• Return type: bool
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)

mostlyright.weather.cache.invalidate_satellite(satellite, product, station, year, month, , cache_root=None)

Remove a satellite cache partition if it exists; return whether removed.

cache_root (P2-1) mirrors the read/write override so a partition written under a --out directory can be invalidated from there; None preserves the default resolution.

• Return type: bool
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • cache_root (Path | None)

mostlyright.weather.cache.read_cache(station, year, month)

Return cached observation rows for (station, year, month) or None.

• Return type: list[dict] | None
• Parameters:
  • station (str)
  • year (int)
  • month (int)

Returns None when: : - the cache file does not exist

• (year, month) is the station’s current LST month (file may be stale)
• a concurrent invalidate() removes the file between the exists() check and the read (treated as cache-miss; the caller re-fetches transparently)

Returns list[dict] otherwise. The list is materialised eagerly from pyarrow - callers can iterate freely without holding a file handle.

mostlyright.weather.cache.read_climate_cache(station, year)

Return cached climate rows for (station, year) or None.

• Return type: list[dict] | None
• Parameters:
  • station (str)
  • year (int)

Returns None when: : - the cache file does not exist

• year is the station’s current LST year (file may be stale)
• a concurrent invalidate_climate() removes the file between the exists() check and the read (cache-miss semantics; caller re-fetches transparently)

mostlyright.weather.cache.read_forecast_cache(station, source, model, year, month)

Return cached forecast rows for the partition key or None on miss.

• Return type: list[dict] | None
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)

Returns None when: : - the partition file does not exist

• source is live or seamless (never cached)
• (year, month) is the current UTC month (cycles may still publish)

mostlyright.weather.cache.read_satellite_cache(satellite, product, station, year, month, , cache_root=None)

Return cached satellite rows for the partition key or None on miss.

Returns None when the partition does not exist or (year, month) is the current UTC month (A6 — mirrors the forecast tier; the current month may still receive scans). cache_root (P2-1) optionally overrides the root so a partition written under a --out directory reads back from there; when None the default resolution is byte-for-byte unchanged.

• Return type: list[dict] | None
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • cache_root (Path | None)

mostlyright.weather.cache.satellite_cache_path(satellite, product, station, year, month, , cache_root=None)

Return the parquet cache path for a satellite partition (SAT-25-04, D8).

Hardens EVERY user-controlled path segment (P2-e — the forecast analog validates only station):

1. station via validate_icao_for_path() (4-letter ICAO).
2. satellite must be a registered native-ring satellite (Phase 26 W1: the whole ring goes16/17/18/19 | himawari8/9 | viirs-* | meteosat-*, per the per-source registry) and contain no path separator.
3. product must be a registered product FOR satellite’s OWNING SOURCE (per-source registry — e.g. an ABI product is invalid on a Himawari satellite) and contain no path separator.

Then assert_path_under() is the final path-traversal backstop. The path has NO mirror segment (D9 — mirror is transport-only).

cache_root (P2-1) optionally overrides the resolved cache root. The backfill CLI threads its --out directory here so the parquet partition lands under --out rather than the home/env cache root. When None (every existing caller) the behavior is byte-for-byte unchanged: _cache_root() resolves the root exactly as before. The path-traversal backstop validates against whichever root is in effect.

• Return type: Path
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • cache_root (Path | None)

mostlyright.weather.cache.write_cache(station, year, month, rows, , source=None)

Atomically write rows to the observation cache.

• Return type: None
• Parameters:
  • station (str)
  • year (int)
  • month (int)
  • rows (list [dict ])
  • source (str | None)

No-op (does NOT raise) when: : - (year, month) is the station’s current LST month

• source ends with .live — live endpoints are never cached

The source kwarg is optional. The standard call site (research()) passes the fetcher’s endpoint identifier (e.g. "iem.asos", "awc.live") so the cache can gate writes uniformly without each fetcher having its own no-cache branch.

mostlyright.weather.cache.write_climate_cache(station, year, rows, , source=None)

Atomically write rows to the annual climate cache.

• Return type: None
• Parameters:
  • station (str)
  • year (int)
  • rows (list [dict ])
  • source (str | None)

No-op (does NOT raise) when: : - year is the station’s current LST year

• source ends with .live

mostlyright.weather.cache.write_forecast_cache(station, source, model, year, month, rows)

Atomically write rows to the forecast cache partition.

• Return type: None
• Parameters:
  • station (str)
  • source (str)
  • model (str)
  • year (int)
  • month (int)
  • rows (list [dict ])

No-op (does NOT raise) when: : - source is live or seamless (never cached)

• (year, month) is the current UTC month (cycles may still publish)
• rows is empty

mostlyright.weather.cache.write_satellite_cache(satellite, product, station, year, month, rows, , cache_root=None)

Atomically write rows to the satellite cache partition (D8).

No-op (does NOT raise) when rows is empty or (year, month) is the current UTC month (A6). On merge into an existing partition, the entire read→concat→“_dedup_satellite_rows“ (first-seen-wins, mirror-invariant) →write sequence runs under a single FileLock acquisition, so two writers targeting the same (satellite, product, station, year, month) partition cannot lost-update each other: the second writer reads the first writer’s already-committed rows and merges on top of them rather than clobbering them. The inner write is the usual .tmp + os.replace atomic swap. No staging dir, no R2.

Unlike the overwrite-only forecast tier (write_forecast_cache does not read-modify-write), this tier is a true read-modify-write merge, so the lock must span the read as well as the write — see _write_table_unlocked.

cache_root (P2-1) optionally overrides the cache root so the partition lands under a caller-supplied directory (the backfill CLI threads its --out here). When None the default _cache_root() resolution is byte-for-byte unchanged — the forecast/observation/climate tiers and all existing satellite-cache callers are unaffected.

• Return type: None
• Parameters:
  • satellite (str)
  • product (str)
  • station (str)
  • year (int)
  • month (int)
  • rows (list [dict ])
  • cache_root (Path | None)