# Many HTML pages Source: https://mostlyright.md/docs/recipes/html-collection/ Read a corpus published as an index and many detail pages as one source, with per-page provenance. ## The question What does the publisher report about each hand matchup, page by page, and which page and revision does every reported line come from? Behind one index address sit 2,819 detail pages. That is one entry in `sources`, one publisher, one line on the dataset's sources card. It spends one of the 256 source slots. ## Rights, first This publisher's terms restrict automated and database reuse. The example asserts no permission and claims no open licence. `rights_claim.claimed_basis` is `unknown`, the claim note says so in the document, and the dataset page repeats it. The example is here for its shape, and its selectors and coverage numbers come from a real run rather than invention. Anyone who runs it decides about the publisher's terms themselves. For another corpus, pick a publisher whose terms permit the read and say so in `claim_note`. ## The source | | | | --- | --- | | Publisher | Card Player | | Index | `https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified` | | Detail pages | `https://www.cardplayer.com/poker-tools/hand-matchups`, 2,819 of them | | Format | a JSON index, HTML detail pages | | Cadence | the publisher adds and corrects pages, and the index carries a `modified` stamp | | Rights | recorded as `unknown`, with no licence claimed | ## The recipe ```json title="recipe.json" { "dataset": { "name": "Card Player hand matchups" }, "question": { "text": "What does the publisher report about each hand matchup, page by page, and which page and revision does every reported line come from?" }, "table": { "name": "hand_matchups", "description": "One row per extracted block of one matchup page.", "grain": [ "page_id", "page_ordinal" ], "columns": [ { "name": "page_id", "type": "string", "nullable": false, "description": "The publisher's own identifier for the matchup page this row came from." }, { "name": "page_ordinal", "type": "integer", "nullable": false, "description": "The one-based position of this row inside its page." }, { "name": "page_url", "type": "string", "nullable": false, "description": "The address the page was fetched from." }, { "name": "page_fetched_at", "type": "timestamp", "nullable": false, "description": "When the page this row came from was last fetched." }, { "name": "page_revision", "type": "integer", "nullable": false, "description": "How many times this page's content has changed since it was first fetched." }, { "name": "title", "type": "string", "nullable": true, "description": "The matchup headline the page states." }, { "name": "story", "type": "string", "nullable": true, "description": "The reported narrative block, in the publisher's own words." }, { "name": "posted", "type": "string", "nullable": true, "description": "The posting date the page displays, as the publisher writes it." }, { "name": "source_url", "type": "string", "nullable": true, "description": "The canonical address the page declares for itself." } ] }, "sources": [ { "name": "hand_matchups", "description": "Card Player's hand-matchups index and every matchup page it lists.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://www.cardplayer.com/poker-tools/hand-matchups" }, "rights_claim": { "claimed_basis": "unknown", "claim_evidence_digest": "sha256:074f5983b4d0d33292ab8180f8d2ac54d379a9e9bc11b105496858062eb91965", "claim_note": "The publisher's terms restrict automated and database reuse. No permission and no reuse licence are asserted here, and no open licence is claimed; the basis is recorded as unknown so the page says so." }, "connector": { "adapter_id": "public.https.collection@2.0.0", "credential_mode": "none", "origin": "https://www.cardplayer.com", "parameters": [ { "name": "reader.family_id", "value": "html.web_extract" }, { "name": "reader.family_version", "value": "1.1.0" }, { "name": "reader.decode_options", "value": "{\"columns\":[{\"attribute\":null,\"name\":\"title\",\"required\":false,\"scope\":\"document\",\"selector\":\"div.cp-single-matchup h1\",\"value\":\"text\"},{\"attribute\":null,\"name\":\"story\",\"required\":false,\"scope\":\"record\",\"selector\":\"div.cp-handmatchup-content\",\"value\":\"text\"},{\"attribute\":null,\"name\":\"posted\",\"required\":false,\"scope\":\"document\",\"selector\":\"div.cp-single-matchup div.text-end\",\"value\":\"text\"},{\"attribute\":null,\"name\":\"source_url\",\"required\":true,\"scope\":\"document\",\"selector\":\"link[rel=\\\"canonical\\\"]\",\"value\":\"href\"}],\"mode\":\"dom\",\"record_selector\":\"div.cp-handmatchup-content\"}" } ] }, "limits": { "max_source_bytes": 1073741824, "max_rows": 200000, "max_requests": 600 }, "collection": { "discovery": { "kind": "json_api", "url": "https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified", "pagination": { "kind": "page_parameter", "parameter": "page", "first": 1 }, "records": { "pointer": "" }, "link": { "pointer": "/link" }, "identity": { "pointer": "/id" }, "revision": { "pointer": "/modified" }, "max_requests": 40 }, "pages": { "allowed_origins": [ "https://www.cardplayer.com" ], "max_pages": 4000, "max_fetches_per_run": 500, "min_interval_milliseconds": 1000, "request_timeout_seconds": 30, "retries": 2, "revisit": { "kind": "revision_hint" } } } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "hand_matchups_rows", "sql": "select page_id, cast(page_ordinal as integer) as page_ordinal, page_url, cast(page_fetched_at as timestamp with time zone) as page_fetched_at, cast(page_revision as integer) as page_revision, title, story, posted, source_url from hand_matchups", "description": "Carry each page's extracted rows through under the declared names, casting the provenance columns the collection appended: the merged relation is sealed as CSV, so every column reaches the transform as text." } ] }, "checks": [], "units": [], "timezone": "UTC" } ``` ## Part by part ### The connector `public.https.collection@2.0.0` is the only adapter that takes a `collection` member. The member is required there and refused everywhere else. That adapter also requires `credential_mode: "none"`, because a page collection reads public addresses and has no credential to present to a link discovery found. It requires `limits.max_requests` too. A collection source may declare neither `window` nor `closed`. Both say the predecessor's bytes are the whole answer for a range. A collection builds its result from a ledger, which decides page by page what to refetch. ### Discovery ```json "discovery": { "kind": "json_api", "url": "https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified", "pagination": {"kind": "page_parameter", "parameter": "page", "first": 1}, "records": {"pointer": ""}, "link": {"pointer": "/link"}, "identity": {"pointer": "/id"}, "revision": {"pointer": "/modified"}, "max_requests": 40 } ``` The publisher's own JSON index lists the corpus. `records.pointer` is `""` because the whole body is the array. `link` gives the detail page's address, `identity` its stable identifier, and `revision` a hint the revisit rule reads. Four discovery kinds exist: | `discovery.kind` | records | link | page identity | revision hint | | --- | --- | --- | --- | --- | | `json_api` | `records.pointer` in RFC 6901 syntax, where `""` means the whole body | `link.pointer` | `identity.pointer`, else the resolved URL | `revision.pointer` | | `html_index` | `records.selector` | the match's `href`, or `link.attribute` | the resolved URL | none | | `sitemap` | every `` | `loc` | the resolved URL | `lastmod` | | `explicit` | `members[]` of `{page_id, url}` | given | given | none | The collection follows the declared pagination only. It never follows a link found inside a detail page, and it follows a `` one level. ### Pages, and the budget ```json "pages": { "allowed_origins": ["https://www.cardplayer.com"], "max_pages": 4000, "max_fetches_per_run": 500, "min_interval_milliseconds": 1000, "request_timeout_seconds": 30, "retries": 2, "revisit": {"kind": "revision_hint"} } ``` `allowed_origins` is the only required member, and it must include the discovery address's own origin. `min_interval_milliseconds` is the gap between two requests. It is a whole number and it defaults to 1000, one request a second. This publisher documents no rate limit, so the example states 1000 and says so here. At that pace 500 fetches take about eight and a half minutes. The platform accepts a gap as low as 16, which is a bound and never a target. `concurrency` is omitted. `revisit: revision_hint` refetches a page only when the index says it changed. The alternatives are `never` and `interval` with `seconds` of at least 3600. ### The Reader pin The pin is required and must be `html.web_extract`, and the same settings apply to every page. `mode: dom` uses the closed CSS-like grammar of tags, ids, classes, attributes, descendants and children. A `record_selector` says what one row is. Each column names a `selector`, a `value` of `text` or `href`, a `scope` of `document` or `record`, and whether it is `required`. Pin `1.1.0` rather than `1.0.0` for anything new. The two take identical settings and differ in reading order. `1.1.0` reads the words in the order they appear on the page, so a link keeps the comma that follows it and `$19.99` reads `$19.99`. `1.0.0` puts an element's own words before the words of anything inside it, which glues neighbouring cells together. `1.0.0` stays unchanged because datasets built with it keep re-verifying. ### The provenance columns The transform receives the Reader's columns plus seven reserved names: `page_id`, `page_url`, `page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at`, `page_ordinal`. A pinned Reader that declares one of those names fails registration. Select them so every row names the page and the revision that produced it. All seven arrive as text. The merged relation is stored as CSV, so selecting `page_ordinal` bare while declaring the column `integer` fails the build with `TRANSFORM_COLUMN_TYPE_MISMATCH`: ```sql select page_id, cast(page_ordinal as integer) as page_ordinal, page_url, cast(page_fetched_at as timestamp with time zone) as page_fetched_at, cast(page_revision as integer) as page_revision, title, story, posted, source_url from hand_matchups ``` ### Checks This document declares none, which is a gap. A collection's usual checks are `key_uniqueness` on `["page_id", "page_ordinal"]`, the grain, and a `row_expectation` floor. Add both before running against a corpus that matters. ### Cadence A collection has no scheduled refresh. Studio refuses a refresh whose action plan carries one, with `RESYNC_REQUIRED` before acquisition, so recording a cadence on this table changes nothing. Read the corpus again with an explicit resync. Size `pages.max_fetches_per_run` and `limits.max_rows` so that one run can cover the corpus, and read `complete` off the coverage block to know whether it did. ## Register, run, inspect Use the real table for one full build. For a bounded test first, follow the [scratch-table workflow](/docs/build/run-and-inspect/#test-before-the-full-build). A successful sample on the real table can publish truncated data. Stop if the forecast refuses; size the recipe before running. ```sh mr-data dataset create --name "Card Player hand matchups" --json mr-data recipe recipe.json --json mr-data recipe explain RECIPE_ID --json mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json mr-data watch RUN_ID --json mr-data status RUN_ID --json mr-data receipt RUN_ID --json # the pages coverage block # a collection has no incremental refresh. Read the corpus again with an explicit table resync ``` ## What the run reports A run writes a `pages` coverage block. The worked run recorded this one: ```json {"discovered": 2819, "discovery_requests": 29, "discovery_complete": true, "duplicates_dropped": 0, "known": 2819, "fetched_this_run": 500, "unchanged": 0, "changed": 497, "failed_this_run": 3, "pending": 2316, "failed": 3, "skipped": 0, "budget_exhausted": "max_fetches_per_run", "complete": false} ``` All fourteen members are always present. None is derived from another. A reader needs both halves: what this run reached, and what it still owes. `complete: false` is not a failure. The run succeeded. The block says what the run reached and which ceiling stopped it. `pending` is work waiting, `failed` is work to retry, and `discovery_complete` says whether the run walked the index to the end. The next refresh continues from the ledger. New pages come first in discovery order, then failures with the oldest first, then the revisits the rule names. Rows of pages this run did not fetch survive byte for byte. Six refreshes of 500 pages cover this corpus. `discovery_failure` is `null`, or the code and one sentence for the first listing request the publisher refused. Read it before concluding a publisher's index is empty. A refresh that already holds pages keeps every row and succeeds. A first run whose listing is refused fails with `COLLECTION_DISCOVERY_FAILED` rather than storing an empty corpus. ## Traps `limits.max_requests` counts every HTTP request, redirects and retries included. Registration only checks that `discovery.max_requests + pages.max_fetches_per_run` fits inside it. That check is necessary and not enough. This document's 600 covers `40 + 500` plus a little, and it runs out early on any publisher that redirects. Where redirects are likely (trailing slashes, `http` to `https`, `www`, which covers most publishers) write `limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run`. A budget larger than the corpus needs costs nothing. Report the shortfall as a backfill in progress, not as a failure and not as a complete corpus. A first run that fetched 500 of 2,819 pages built a real table, with real rows and a real number of pages still owed. A page that disappears from the index deletes nothing. A page answering with different content replaces its own rows, increments `page_revision` and keeps the previous digest. Write that correction on the run record when it happens. Check the discovery block by hand before registering. Check the address the first request goes to, and whether the pagination parameter is already written into it. Check that the records pointer names an array, and that the two budgets fit inside `limits.max_requests`. Registration refuses all four, and all four are cheap to fix before a run spends somebody's rate limit. `html.web_extract` reads the markup and never runs it. Nothing on the page executes. It follows no link, loads no image or stylesheet, and never consults what the page would look like once styled. A selector that matches more than one value where the column asked for one fails, and so does a required field that matches nothing.