About our crawler
Mostly Right fetches public datasets that our users explicitly register — government statistics, published CSV and SDMX feeds, open data portals. The crawler is not a spider: it follows no links, discovers nothing on its own, and requests exactly the documents a person asked us to fetch.
How it identifies itself
Section titled “How it identifies itself”Every request carries this User-Agent:
MostlyRightDataHarness-PublicHttps/1.0 (+https://mostlyright.md/crawler; crawler@mostlyright.md)Where requests come from
Section titled “Where requests come from”Requests come from these static Google Cloud addresses (us-central1):
34.123.121.17535.254.54.16234.58.239.334.72.34.10335.226.221.934.57.110.3735.225.49.3735.238.158.9134.123.136.435.202.152.22235.253.120.125
How it behaves
Section titled “How it behaves”- At most two concurrent connections per origin, with at least one second between requests.
- It never authenticates, never submits forms, and never executes page scripts.
- Each fetch is bounded, one-shot, and tied to a specific user request — there is no recurring background scraping of any origin a user has not scheduled.
- Large objects are fetched as byte ranges when the origin advertises them, to keep single connections short.
To stop us, or reach us
Section titled “To stop us, or reach us”To stop us fetching from your site, write to crawler@mostlyright.md: a person reads it, and we stop on a written notice. The same address is for allowlisting, questions, or reporting a problem with our crawler’s behavior.