Many-page sources

Use one collection source to read an index and its detail pages into a single relation.

Treat a corpus with an index and many detail pages as one source. A corpus of 2,819 pages still uses one source entry and appears once on the dataset’s Sources card.

The adapter is public.https.collection@2.0.0. The recipe declares how to list the corpus and how much of it one run may fetch. The run lists the corpus and fetches pages under those budgets. It decodes every page with one pinned Reader and writes one relation whose rows carry the page they came from.

Source count and publisher count are not page counts and are not request counts. A collection with thousands of pages behind it leaves the 256-source ceiling alone.

What it is not

It is not a general discovery system. It follows only the pagination the document declares, and never a link found inside a detail page. Every request is a credential-free HTTPS GET. The worker treats each listing as inert data and does not execute page scripts. It reads the body as JSON, through a closed CSS-like selector, or as a sitemap with no doctype and no entity.

The source, in full

{
  "name": "council_decisions",
  "description": "Published decisions of the council, one page per decision.",
  "source_class": "user_url",
  "data_classification": "public",
  "locator": {
    "kind": "https_url",
    "display_locator": "https://www.example.com/decisions"
  },
  "rights_claim": {
    "claimed_basis": "public_domain_asserted",
    "claim_evidence_digest": "sha256:…"
  },
  "connector": {
    "adapter_id": "public.https.collection@2.0.0",
    "credential_mode": "none",
    "origin": "https://www.example.com",
    "parameters": [
      {"name": "reader.family_id", "value": "html.web_extract"},
      {"name": "reader.family_version", "value": "1.1.0"},
      {"name": "reader.decode_options", "value": "{…canonical JSON…}"}
    ]
  },
  "limits": {
    "max_source_bytes": 1073741824,
    "max_rows": 200000,
    "max_requests": 1064
  },
  "collection": {
    "discovery": {
      "kind": "json_api",
      "url": "https://www.example.com/wp-json/wp/v2/decisions?per_page=100",
      "pagination": {"kind": "page_parameter", "parameter": "page", "first": 1},
      "records": {"pointer": ""},
      "link": {"pointer": "/link"},
      "identity": {"pointer": "/id"},
      "revision": {"pointer": "/modified"},
      "max_requests": 40
    },
    "pages": {
      "allowed_origins": ["https://www.example.com"],
      "max_pages": 4000,
      "max_fetches_per_run": 500,
      "concurrency": 2,
      "min_interval_seconds": 1,
      "request_timeout_seconds": 30,
      "retries": 2,
      "revisit": {"kind": "revision_hint"}
    }
  }
}

collection is required exactly when the connector is public.https.collection@2.0.0, and refused beside every other adapter. It has exactly two members, discovery and pages.

A collection source also states limits.max_requests, pins a Reader of the html.tabular, html.web_extract, json.tabular, or xml.tabular family, and states credential_mode: "none". It may not declare a window, because a window narrows one address and a collection lists many. It may not declare closed either, because a collection continues across refreshes. Registration forbids the member itself and refuses closed: false written out in full.

The Reader pin is the same settings for every page.

Discovery kinds

kind	Records	Link	Page identity	Revision hint	Pagination kinds
json_api	records.pointer, an RFC 6901 pointer. "" means the whole body is the array	link.pointer	identity.pointer, else the resolved URL	revision.pointer	none, page_parameter, offset_parameter, link_header, cursor
html_index	records.selector, the reader’s closed CSS-like selector	the match’s href, or link.attribute	the resolved URL	none	none, page_parameter, next_selector
sitemap	every <url><loc> of a urlset	loc	the resolved URL	lastmod	none
explicit	members[] of {page_id, url}	given	given	none	none

The run follows a <sitemapindex> exactly one level. It contributes its children and nothing deeper.

Pagination grammars

kind	Members	How it composes the next request	Stops on
none	none	there is no next request	one request
page_parameter	parameter, first (default 1)	?<parameter>=<n>, counting up from first	an empty page, a repeated address, discovery.max_requests
offset_parameter	parameter, first (default 0), step (default 1)	?<parameter>=<n>, advancing by step	an empty page, a repeated address, discovery.max_requests
link_header	none	the address in Link: <…>; rel="next"	no rel=next, an empty page, a repeated address, the ceiling
cursor	parameter, pointer	?<parameter>=<token> from the body at pointer	an absent or repeated token, an empty page, the ceiling
next_selector	selector	the href of the first match	no match, an empty page, a repeated address, the ceiling

A paged address may not already state its own pagination parameter. The document states the parameter once, and the grammar writes it.

The bounds

Registration refuses all of them when it reads the recipe.

Member	Bound	Default when absent
discovery.max_requests	1 to 1024	64
discovery.pagination	optional	none
pages.allowed_origins	1 to 8 canonical https origins, and they must include the discovery address’s own origin	required
pages.max_pages	1 to 50,000	50,000
pages.max_fetches_per_run	1 to 10,000	10,000
pages.concurrency	1 to 10,000 requested operations. The worker admits at most 25 concurrent fetches and still obeys provider and egress budgets	admitted capacity
pages.min_interval_seconds	1 to 3600, a whole number of seconds	1
pages.request_timeout_seconds	5 to 120	30
pages.retries	0 to 3	2
pages.revisit	never, revision_hint, or interval with seconds ≥ 3600	never

min_interval_seconds is whole because the registered document carries no fractional JSON number anywhere. One second is both the floor here and the limiter’s own gap between two requests to one host.

Sizing limits.max_requests

The budget counts HTTP requests, redirect hops and retries included, not pages. A page that redirects once costs two. One that redirects and retries once costs more again.

Registration enforces discovery.max_requests + pages.max_fetches_per_run <= limits.max_requests. A budget set to exactly that sum still runs out before the last page whenever anything redirects. The run then stops with budget_exhausted: "max_requests" short of the pages the recipe allowed.

Where redirects are likely, which covers any publisher that canonicalises trailing slashes, http→https or www, write:

limits.max_requests >= discovery.max_requests + 2 * pages.max_fetches_per_run

Which media types a listing may answer with

The pinned Reader decodes a detail page, so that fetch accepts the family’s media types, text/html for html.web_extract. The run decodes a listing itself and reaches no Reader, so a listing accepts what its discovery kind accepts:

kind	Accepted
json_api	application/json, application/ld+json, application/vnd.api+json, text/json
sitemap	application/xml, text/xml, application/rss+xml
html_index	text/html, application/xhtml+xml
explicit	the run makes no request

A listing that answers something outside its kind’s set is refused before its body is read, and reported as discovery_failure.

Addresses and origins

The run resolves a relative link against the page it appeared on, then canonicalises it: scheme and host lowercased, fragment dropped, an explicit :443 dropped. It leaves the trailing slash, the case of the path and the order of query parameters alone, because publishers may distinguish them.

The ledger records a link that is not https, or whose origin is not in allowed_origins, as skipped_origin, and the run never fetches it. The origin comparison alone is not the check. https://host:8443/x and https://user:pw@host/x carry the allowed host and are neither the allowed origin nor an address this source may reach. A page whose redirect lands outside the allowlist goes the same way. The run does not decode the bytes, the ledger records where it landed, and the page’s previous rows survive.

allowed_origins also folds into the run’s own egress allowlist. If the driver cannot fetch an address, the run cannot reach it either. Duplicate links inside one discovery pass collapse to the first occurrence, and duplicates_dropped counts them.

revisit

kind	What it refetches
never	only pages never fetched, and failures
revision_hint	additionally, pages whose discovery revision hint moved since the fetch that wrote them. Needs a revision extractor.
interval	additionally, pages whose last fetch is older than seconds, oldest first

What the transform is handed

The pinned Reader’s own columns, plus seven provenance columns appended in this order:

page_id, page_url, page_fetched_at, page_content_sha256, page_revision, page_discovered_at, page_ordinal

Those names are reserved. Registration refuses a pinned Reader that declares one of them. Nobody can read back a relation holding two columns of one name.

page_revision is 1 on a page’s first fetch and increments each time that page’s content digest changes. page_ordinal is the 1-based row index inside its page, so record identity is (page_id, page_ordinal). The relation sorts by page_id then page_ordinal, so the bytes written do not depend on the order the run fetched pages in.

All seven arrive as text and need a cast. The run writes the merged relation as CSV, so every column reaches the transform as VARCHAR. Selecting page_ordinal bare while declaring the column integer fails the build with TRANSFORM_COLUMN_TYPE_MISMATCH.

select page_id,
       cast(page_ordinal as integer) as page_ordinal,
       cast(page_revision as integer) as page_revision,
       cast(page_fetched_at as timestamp with time zone) as page_fetched_at,
       cast(page_discovered_at as timestamp with time zone) as page_discovered_at,
       title, story
from council_decisions

A declared timestamp column takes TIMESTAMP WITH TIME ZONE and nothing else. The recipe declares page_url, page_id and page_content_sha256 as string, and they need no cast.

Select them in the transform so every row on the table says which page and which revision produced it.

The merge

Every run starts from the predecessor’s relation and its ledger, both empty on a full run.

1. Discover under discovery.max_requests. New page ids become pending, and known pages get a fresh last_seen_at and a refreshed revision hint.
2. Choose work until a budget runs out: pending pages in discovery order, then failed pages oldest failure first, then the revisits revisit names.
3. Fetch and decode each chosen page with the one pinned Reader.
4. Write the relation and the ledger, and emit the coverage block.

Step three ends three ways. A page carrying the ledger’s content digest stays fetched unchanged. It keeps its predecessor rows verbatim, provenance included, so an idle refresh writes the predecessor’s bytes. A different digest replaces exactly that page’s rows, increments revisions, and records the old digest in previous_content_sha256. That is a correction, and it is worth saying on the run record. A refused fetch, a decode that raised, or a page whose columns are not the pin’s marks the page failed. The ledger row carries a code, and that page keeps its previous rows.

Absence never deletes anything. A page discovery no longer lists keeps its rows and its ledger row. A page whose fetch failed keeps its rows. Only that page answering with different content removes a page’s rows.

A page-level failure is never a run failure.

Expect the first run not to finish

A run that spends its fetch budget still succeeds. Its coverage block reports complete: false and names the budget that ended it.

{
  "discovered": 2819, "discovery_requests": 29, "discovery_complete": true,
  "discovery_failure": null,
  "duplicates_dropped": 0, "known": 2819, "fetched_this_run": 500,
  "unchanged": 0, "changed": 0, "failed_this_run": 3,
  "pending": 2316, "failed": 3, "skipped": 0,
  "budget_exhausted": "max_fetches_per_run",
  "complete": false
}

All fifteen members are always present. budget_exhausted names which ceiling ended the run: max_requests, max_source_bytes, deadline, max_pages, max_fetches_per_run, or null. complete is true only when discovery is complete, no pages are pending or failed, and the merged relation is not truncated.

A collection has no scheduled refresh. Studio refuses a refresh whose action plan carries one, with RESYNC_REQUIRED before acquisition, so a corpus advances by explicit resync rather than on a cadence. A resync starts the ledger empty and discovers the corpus from the beginning. Size discovery.max_requests, pages.max_fetches_per_run and limits.max_rows so that one run can cover the corpus, and read complete to know whether it did. Report a shortfall as an incomplete corpus rather than as a failure and rather than as a complete one.

truncated keeps its own separate meaning: the merged relation hit limits.max_rows. The clamp falls between pages, never through one. A first page larger than the clamp is the exception, and the run cuts through that one. Clipped pages remain pending and coverage is incomplete. Raise max_rows before refreshing to restore their omitted rows, including when the publisher has not changed the page. Restoring identical content does not increment its revision. Refresh also detects and repairs missing rows in ledgers written before this recovery behavior was available. Keeping the same insufficient limit cannot complete the corpus.

When a collection is refused

Refusal	What it means
Registration refusal	Registration checks four things before a run spends anybody’s rate limit. They are the first request’s address, whether that address already carries the pagination parameter, whether the records pointer names an array, and whether discovery.max_requests + pages.max_fetches_per_run fits inside limits.max_requests. It also refuses a window, a closed, a credential mode other than none, or a Reader outside the four supported collection families. It refuses a missing limits.max_requests, an allowed_origins that does not include the discovery origin, or a pinned Reader declaring one of the seven reserved column names.
COLLECTION_DISCOVERY_FAILED	Listing the collection failed and the run has nothing to continue from: no predecessor ledger, no explicit members, and not one address listed before the refusal. Writing it would leave an empty relation and an empty ledger, and the next refresh would merge onto them as if the corpus really were empty.
COLLECTION_MERGE_INVALID	The run and its predecessor contradict each other. The stored relation’s columns are not the ones this run decodes, the ledger sits under a different header, or the discovery body is not the shape the document declared.
TRANSFORM_COLUMN_TYPE_MISMATCH	A provenance column selected bare while the recipe declares its column as something other than string. Cast it.
HOSTED_COLLECTION_UNSUPPORTED	The document named public.https.collection@1.0.0. That is a different adapter kept only so old receipts re-verify. It is not reachable, and 2.0.0 is not a widening of it.

A run that already holds pages succeeds when a discovery refusal lands. It keeps every row and reports discovery_complete: false with a discovery_failure naming the first listing request it could not read. Read that member before concluding that a run reporting zero discovered found an empty publisher index.

What operators of the origin see

The fetcher identifies itself on every request:

MostlyRightDataHarness-PublicHttps/1.0 (+https://mostlyright.md/crawler)

That URL resolves to a page naming the static addresses requests come from. An operator can hold the fetcher to this behaviour:

• At most two concurrent connections per origin, with at least one second between requests.
• It respects robots.txt, never authenticates, never submits forms and never executes page scripts. It honors a MostlyRightDataHarness disallow in robots.txt.
• Every fetch carries a bound, happens once, and ties to a specific request, so no background scraping recurs against an origin nobody scheduled.
• It fetches large objects as byte ranges when the origin advertises them, which keeps single connections short.

The full page is at /crawler.

Next

Documents and files covers PDFs and scientific formats. Readers covers html.web_extract settings.