About our crawler

Mostly Right fetches public datasets that our users explicitly register — government statistics, published CSV and SDMX feeds, open data portals. The crawler is not a spider: it follows no links, discovers nothing on its own, and requests exactly the documents a person asked us to fetch.

How it identifies itself

Every request carries this User-Agent:

MostlyRightDataHarness-PublicHttps/1.0 (+https://mostlyright.md/crawler)

Requests come from these static Google Cloud addresses (us-central1):

• 34.123.121.175
• 35.254.54.162
• 34.58.239.3
• 34.72.34.103
• 35.226.221.9
• 34.57.110.37
• 35.225.49.37
• 35.238.158.91
• 34.123.136.4
• 35.202.152.222
• 35.253.120.125

How it behaves

• At most two concurrent connections per origin, with at least one second between requests.
• It respects robots.txt and never authenticates, never submits forms, and never executes page scripts.
• Each fetch is bounded, one-shot, and tied to a specific user request — there is no recurring background scraping of any origin a user has not scheduled.
• Large objects are fetched as byte ranges when the origin advertises them, to keep single connections short.

To block or reach us

Disallowing MostlyRightDataHarness in robots.txt is honored. For allowlisting, questions, or to report a problem with our crawler's behavior, write tocontact@mostlyright.md — a person reads it.