# Many-page sources

Source: https://mostlyright.md/docs/build/collections/

Use one collection source to read an index and its detail pages into a single relation.

Treat a corpus with an index and many detail pages as one source. A corpus of 2,819 pages still uses
one source entry and appears once on the dataset's Sources card.

The adapter is `public.https.collection@2.0.0`. The recipe declares how to list the corpus and how much of it one run may fetch. The run lists the corpus and fetches pages under those budgets. It decodes every page with one pinned Reader and writes one relation whose rows carry the page they came from.

Source count and publisher count are not page counts and are not request counts. A collection with thousands of pages behind it leaves the 256-source ceiling alone.

## What it is not

It is not a general discovery system. It follows only the pagination the document declares, and never a link found inside a detail page. Every request is a credential-free HTTPS GET. The worker treats each listing as inert data and does not execute page scripts. It reads the body as JSON, through a closed CSS-like selector, or as a sitemap with no doctype and no entity.

## The source, in full

```json
{
  "name": "council_decisions",
  "description": "Published decisions of the council, one page per decision.",
  "source_class": "user_url",
  "data_classification": "public",
  "locator": {
    "kind": "https_url",
    "display_locator": "https://www.example.com/decisions"
  },
  "rights_claim": {
    "claimed_basis": "public_domain_asserted",
    "claim_evidence_digest": "sha256:…"
  },
  "connector": {
    "adapter_id": "public.https.collection@2.0.0",
    "credential_mode": "none",
    "origin": "https://www.example.com",
    "parameters": [
      {"name": "reader.family_id", "value": "html.web_extract"},
      {"name": "reader.family_version", "value": "1.1.0"},
      {"name": "reader.decode_options", "value": "{…canonical JSON…}"}
    ]
  },
  "limits": {
    "max_source_bytes": 1073741824,
    "max_rows": 200000,
    "max_requests": 1064
  },
  "collection": {
    "discovery": {
      "kind": "json_api",
      "url": "https://www.example.com/wp-json/wp/v2/decisions?per_page=100",
      "pagination": {"kind": "page_parameter", "parameter": "page", "first": 1},
      "records": {"pointer": ""},
      "link": {"pointer": "/link"},
      "identity": {"pointer": "/id"},
      "revision": {"pointer": "/modified"},
      "max_requests": 40
    },
    "pages": {
      "allowed_origins": ["https://www.example.com"],
      "max_pages": 4000,
      "max_fetches_per_run": 500,
      "concurrency": 2,
      "min_interval_seconds": 1,
      "request_timeout_seconds": 30,
      "retries": 2,
      "revisit": {"kind": "revision_hint"}
    }
  }
}
```

`collection` is required exactly when the connector is `public.https.collection@2.0.0`, and refused beside every other adapter. It has exactly two members, `discovery` and `pages`.

A collection source also states `limits.max_requests`, pins a Reader of the `html.tabular`, `html.web_extract`, `json.tabular`, or `xml.tabular` family, and states `credential_mode: "none"`. It may not declare a `window`, because a window narrows one address and a collection lists many. It may not declare `closed` either, because a collection continues across refreshes. Registration forbids the member itself and refuses `closed: false` written out in full.

The Reader pin is the same settings for every page.

## Discovery kinds

| `kind` | Records | Link | Page identity | Revision hint | Pagination kinds |
| --- | --- | --- | --- | --- | --- |
| `json_api` | `records.pointer`, an RFC 6901 pointer. `""` means the whole body is the array | `link.pointer` | `identity.pointer`, else the resolved URL | `revision.pointer` | `none`, `page_parameter`, `offset_parameter`, `link_header`, `cursor` |
| `html_index` | `records.selector`, the reader's closed CSS-like selector | the match's `href`, or `link.attribute` | the resolved URL | none | `none`, `page_parameter`, `next_selector` |
| `sitemap` | every `<url><loc>` of a urlset | `loc` | the resolved URL | `lastmod` | none |
| `explicit` | `members[]` of `{page_id, url}` | given | given | none | none |

The run follows a `<sitemapindex>` exactly one level. It contributes its children and nothing deeper.

## Pagination grammars

| `kind` | Members | How it composes the next request | Stops on |
| --- | --- | --- | --- |
| `none` | none | there is no next request | one request |
| `page_parameter` | `parameter`, `first` (default 1) | `?<parameter>=<n>`, counting up from `first` | an empty page, a repeated address, `discovery.max_requests` |
| `offset_parameter` | `parameter`, `first` (default 0), `step` (default 1) | `?<parameter>=<n>`, advancing by `step` | an empty page, a repeated address, `discovery.max_requests` |
| `link_header` | none | the address in `Link: <…>; rel="next"` | no `rel=next`, an empty page, a repeated address, the ceiling |
| `cursor` | `parameter`, `pointer` | `?<parameter>=<token>` from the body at `pointer` | an absent or repeated token, an empty page, the ceiling |
| `next_selector` | `selector` | the `href` of the first match | no match, an empty page, a repeated address, the ceiling |

A paged address may not already state its own pagination parameter. The document states the parameter once, and the grammar writes it.

## The bounds

Registration refuses all of them when it reads the recipe.

| Member | Bound | Default when absent |
| --- | --- | --- |
| `discovery.max_requests` | 1 to 1024 | 64 |
| `discovery.pagination` | optional | `none` |
| `pages.allowed_origins` | 1 to 8 canonical https origins, and they must include the discovery address's own origin | required |
| `pages.max_pages` | 1 to 50,000 | 50,000 |
| `pages.max_fetches_per_run` | 1 to 10,000 | 10,000 |
| `pages.concurrency` | 1 to 10,000 requested operations. The worker admits at most 25 concurrent fetches and still obeys provider and egress budgets | admitted capacity |
| `pages.min_interval_seconds` | 1 to 3600, a whole number of seconds | 1 |
| `pages.request_timeout_seconds` | 5 to 120 | 30 |
| `pages.retries` | 0 to 3 | 2 |
| `pages.revisit` | `never`, `revision_hint`, or `interval` with `seconds` ≥ 3600 | `never` |

`min_interval_seconds` is whole because the registered document carries no fractional JSON number anywhere. One second is both the floor here and the limiter's own gap between two requests to one host.

### Sizing `limits.max_requests`

The budget counts HTTP requests, redirect hops and retries included, not pages. A page that redirects once costs two. One that redirects and retries once costs more again.

Registration enforces `discovery.max_requests + pages.max_fetches_per_run <= limits.max_requests`. A budget set to exactly that sum still runs out before the last page whenever anything redirects. The run then stops with `budget_exhausted: "max_requests"` short of the pages the recipe allowed.

Where redirects are likely, which covers any publisher that canonicalises trailing slashes, `http`→`https` or `www`, write:

```
limits.max_requests >= discovery.max_requests + 2 * pages.max_fetches_per_run
```

### Which media types a listing may answer with

The pinned Reader decodes a detail page, so that fetch accepts the family's media types, `text/html` for `html.web_extract`. The run decodes a listing itself and reaches no Reader, so a listing accepts what its discovery kind accepts:

| `kind` | Accepted |
| --- | --- |
| `json_api` | `application/json`, `application/ld+json`, `application/vnd.api+json`, `text/json` |
| `sitemap` | `application/xml`, `text/xml`, `application/rss+xml` |
| `html_index` | `text/html`, `application/xhtml+xml` |
| `explicit` | the run makes no request |

A listing that answers something outside its kind's set is refused before its body is read, and reported as `discovery_failure`.

## Addresses and origins

The run resolves a relative link against the page it appeared on, then canonicalises it: scheme and host lowercased, fragment dropped, an explicit `:443` dropped. It leaves the trailing slash, the case of the path and the order of query parameters alone, because publishers may distinguish them.

The ledger records a link that is not https, or whose origin is not in `allowed_origins`, as `skipped_origin`, and the run never fetches it. The origin comparison alone is not the check. `https://host:8443/x` and `https://user:pw@host/x` carry the allowed host and are neither the allowed origin nor an address this source may reach. A page whose redirect lands outside the allowlist goes the same way. The run does not decode the bytes, the ledger records where it landed, and the page's previous rows survive.

`allowed_origins` also folds into the run's own egress allowlist. If the driver cannot fetch an address, the run cannot reach it either. Duplicate links inside one discovery pass collapse to the first occurrence, and `duplicates_dropped` counts them.

## `revisit`

| `kind` | What it refetches |
| --- | --- |
| `never` | only pages never fetched, and failures |
| `revision_hint` | additionally, pages whose discovery revision hint moved since the fetch that wrote them. Needs a `revision` extractor. |
| `interval` | additionally, pages whose last fetch is older than `seconds`, oldest first |

## What the transform is handed

The pinned Reader's own columns, plus seven provenance columns appended in this order:

`page_id`, `page_url`, `page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at`, `page_ordinal`

Those names are reserved. Registration refuses a pinned Reader that declares one of them. Nobody can read back a relation holding two columns of one name.

`page_revision` is 1 on a page's first fetch and increments each time that page's content digest changes. `page_ordinal` is the 1-based row index inside its page, so record identity is `(page_id, page_ordinal)`. The relation sorts by `page_id` then `page_ordinal`, so the bytes written do not depend on the order the run fetched pages in.

All seven arrive as text and need a cast. The run writes the merged relation as CSV, so every column reaches the transform as `VARCHAR`. Selecting `page_ordinal` bare while declaring the column `integer` fails the build with `TRANSFORM_COLUMN_TYPE_MISMATCH`.

```sql
select page_id,
       cast(page_ordinal as integer) as page_ordinal,
       cast(page_revision as integer) as page_revision,
       cast(page_fetched_at as timestamp with time zone) as page_fetched_at,
       cast(page_discovered_at as timestamp with time zone) as page_discovered_at,
       title, story
from council_decisions
```

A declared `timestamp` column takes `TIMESTAMP WITH TIME ZONE` and nothing else. The recipe declares `page_url`, `page_id` and `page_content_sha256` as `string`, and they need no cast.

Select them in the transform so every row on the table says which page and which revision produced it.

## The merge

Every run starts from the predecessor's relation and its ledger, both empty on a `full` run.

1. Discover under `discovery.max_requests`. New page ids become `pending`, and known pages get a fresh `last_seen_at` and a refreshed revision hint.
2. Choose work until a budget runs out: pending pages in discovery order, then failed pages oldest failure first, then the revisits `revisit` names.
3. Fetch and decode each chosen page with the one pinned Reader.
4. Write the relation and the ledger, and emit the coverage block.

Step three ends three ways. A page carrying the ledger's content digest stays `fetched` unchanged. It keeps its predecessor rows verbatim, provenance included, so an idle refresh writes the predecessor's bytes. A different digest replaces exactly that page's rows, increments `revisions`, and records the old digest in `previous_content_sha256`. That is a correction, and it is worth saying on the run record. A refused fetch, a decode that raised, or a page whose columns are not the pin's marks the page `failed`. The ledger row carries a code, and that page keeps its previous rows.

Absence never deletes anything. A page discovery no longer lists keeps its rows and its ledger row. A page whose fetch failed keeps its rows. Only that page answering with different content removes a page's rows.

A page-level failure is never a run failure.

## Expect the first run not to finish

A run that spends its fetch budget still succeeds. Its coverage block reports `complete: false` and names the budget that ended it.

```json
{
  "discovered": 2819, "discovery_requests": 29, "discovery_complete": true,
  "discovery_failure": null,
  "duplicates_dropped": 0, "known": 2819, "fetched_this_run": 500,
  "unchanged": 0, "changed": 0, "failed_this_run": 3,
  "pending": 2316, "failed": 3, "skipped": 0,
  "budget_exhausted": "max_fetches_per_run",
  "complete": false
}
```

All fifteen members are always present. `budget_exhausted` names which ceiling ended the run: `max_requests`, `max_source_bytes`, `deadline`, `max_pages`, `max_fetches_per_run`, or `null`. `complete` is true only when discovery is complete, no pages are pending or failed, and the merged relation is not truncated.

A page collection has no incremental refresh. Studio refuses a refresh whose action plan carries one, with `RESYNC_REQUIRED` before acquisition, so a corpus advances by an explicit resync or by a declared `rebuild`, which runs that full read on an interval the recipe states ([Rebuild a table that cannot refresh](/docs/build/publish-and-refresh/#rebuild-a-table-that-cannot-refresh)). A resync or a rebuild starts the ledger empty and discovers the corpus from the beginning. A per-row request collection is the one kind that refreshes, and only beside the request window it reads from ([What a refresh actually does](/docs/build/publish-and-refresh/#what-a-refresh-actually-does)). Size `discovery.max_requests`, `pages.max_fetches_per_run` and `limits.max_rows` so that one run can cover the corpus, and read `complete` to know whether it did. Report a shortfall as an incomplete corpus rather than as a failure and rather than as a complete one.

`truncated` keeps its own separate meaning: the merged relation hit `limits.max_rows`. The clamp falls between pages, never through one. A first page larger than the clamp is the exception, and the run cuts through that one. Clipped pages remain pending and coverage is incomplete. Raise `max_rows` before refreshing to restore their omitted rows, including when the publisher has not changed the page. Restoring identical content does not increment its revision. Refresh also detects and repairs missing rows in ledgers written before this recovery behavior was available. Keeping the same insufficient limit cannot complete the corpus.

## When a collection is refused

| Refusal | What it means |
| --- | --- |
| Registration refusal | Registration checks four things before a run spends anybody's rate limit. They are the first request's address, whether that address already carries the pagination parameter, whether the records pointer names an array, and whether `discovery.max_requests + pages.max_fetches_per_run` fits inside `limits.max_requests`. It also refuses a `window`, a `closed`, a credential mode other than `none`, or a Reader outside the four supported collection families. It refuses a missing `limits.max_requests`, an `allowed_origins` that does not include the discovery origin, or a pinned Reader declaring one of the seven reserved column names. |
| `COLLECTION_DISCOVERY_FAILED` | Listing the collection failed and the run has nothing to continue from: no predecessor ledger, no `explicit` members, and not one address listed before the refusal. Writing it would leave an empty relation and an empty ledger, and the next refresh would merge onto them as if the corpus really were empty. |
| `COLLECTION_MERGE_INVALID` | The run and its predecessor contradict each other. The stored relation's columns are not the ones this run decodes, the ledger sits under a different header, or the discovery body is not the shape the document declared. |
| `TRANSFORM_COLUMN_TYPE_MISMATCH` | A provenance column selected bare while the recipe declares its column as something other than `string`. Cast it. |
| `HOSTED_COLLECTION_UNSUPPORTED` | The document named `public.https.collection@1.0.0`. That is a different adapter kept only so old receipts re-verify. It is not reachable, and 2.0.0 is not a widening of it. |

A run that already holds pages succeeds when a discovery refusal lands. It keeps every row and reports `discovery_complete: false` with a `discovery_failure` naming the first listing request it could not read. Read that member before concluding that a run reporting zero discovered found an empty publisher index.

## What operators of the origin see

The fetcher identifies itself on every request:

```
MostlyRightDataHarness-PublicHttps/1.0 (+https://mostlyright.md/crawler)
```

That URL resolves to a page naming the static addresses requests come from. An operator can hold the fetcher to this behaviour:

- At most two concurrent connections per origin, with at least one second between requests.
- It respects `robots.txt`, never authenticates, never submits forms and never executes page scripts. It honors a `MostlyRightDataHarness` disallow in `robots.txt`.
- Every fetch carries a bound, happens once, and ties to a specific request, so no background scraping recurs against an origin nobody scheduled.
- It fetches large objects as byte ranges when the origin advertises them, which keeps single connections short.

The full page is at [/crawler](/crawler/).

## Next

[Documents and files](/docs/build/documents/) covers PDFs and scientific formats. [Readers](/docs/reference/readers/) covers `html.web_extract` settings.