Many HTML pages

Read a corpus published as an index and many detail pages as one source, with per-page provenance.

The question

What does the publisher report about each hand matchup, page by page, and which page and revision does every reported line come from?

Behind one index address sit 2,819 detail pages. That is one entry in sources, one publisher, one line on the dataset’s sources card. It spends one of the 256 source slots.

Rights, first

This publisher’s terms restrict automated and database reuse. The example asserts no permission and claims no open licence. rights_claim.claimed_basis is unknown, the claim note says so in the document, and the dataset page repeats it.

The example is here for its shape, and its selectors and coverage numbers come from a real run rather than invention. Anyone who runs it decides about the publisher’s terms themselves. For another corpus, pick a publisher whose terms permit the read and say so in claim_note.

The source


Publisher	Card Player
Index	`https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified`
Detail pages	`https://www.cardplayer.com/poker-tools/hand-matchups`, 2,819 of them
Format	a JSON index, HTML detail pages
Cadence	the publisher adds and corrects pages, and the index carries a `modified` stamp
Rights	recorded as `unknown`, with no licence claimed

The recipe

{
  "dataset": {
    "name": "Card Player hand matchups",
    "description": "Every reported hand matchup Card Player publishes under its hand-matchups tool, one row per extracted block of a matchup page, with the page each row came from named on the row."
  },
  "question": {
    "text": "What does the publisher report about each hand matchup, page by page, and which page and revision does every reported line come from?"
  },
  "table": {
    "name": "hand_matchups",
    "description": "One row per extracted block of one matchup page.",
    "grain": [
      "page_id",
      "page_ordinal"
    ],
    "columns": [
      {
        "name": "page_id",
        "type": "string",
        "nullable": false,
        "description": "The publisher's own identifier for the matchup page this row came from."
      },
      {
        "name": "page_ordinal",
        "type": "integer",
        "nullable": false,
        "description": "The one-based position of this row inside its page."
      },
      {
        "name": "page_url",
        "type": "string",
        "nullable": false,
        "description": "The address the page was fetched from."
      },
      {
        "name": "page_fetched_at",
        "type": "timestamp",
        "nullable": false,
        "description": "When the page this row came from was last fetched."
      },
      {
        "name": "page_revision",
        "type": "integer",
        "nullable": false,
        "description": "How many times this page's content has changed since it was first fetched."
      },
      {
        "name": "title",
        "type": "string",
        "nullable": true,
        "description": "The matchup headline the page states."
      },
      {
        "name": "story",
        "type": "string",
        "nullable": true,
        "description": "The reported narrative block, in the publisher's own words."
      },
      {
        "name": "posted",
        "type": "string",
        "nullable": true,
        "description": "The posting date the page displays, as the publisher writes it."
      },
      {
        "name": "source_url",
        "type": "string",
        "nullable": true,
        "description": "The canonical address the page declares for itself."
      }
    ]
  },
  "sources": [
    {
      "name": "hand_matchups",
      "description": "Card Player's hand-matchups index and every matchup page it lists.",
      "source_class": "user_url",
      "data_classification": "public",
      "locator": {
        "kind": "https_url",
        "display_locator": "https://www.cardplayer.com/poker-tools/hand-matchups"
      },
      "rights_claim": {
        "claimed_basis": "unknown",
        "claim_evidence_digest": "sha256:074f5983b4d0d33292ab8180f8d2ac54d379a9e9bc11b105496858062eb91965",
        "claim_note": "The publisher's terms restrict automated and database reuse. No permission and no reuse licence are asserted here, and no open licence is claimed; the basis is recorded as unknown so the page says so."
      },
      "connector": {
        "adapter_id": "public.https.collection@2.0.0",
        "credential_mode": "none",
        "origin": "https://www.cardplayer.com",
        "parameters": [
          {
            "name": "reader.family_id",
            "value": "html.web_extract"
          },
          {
            "name": "reader.family_version",
            "value": "1.1.0"
          },
          {
            "name": "reader.decode_options",
            "value": "{\"columns\":[{\"attribute\":null,\"name\":\"title\",\"required\":false,\"scope\":\"document\",\"selector\":\"div.cp-single-matchup h1\",\"value\":\"text\"},{\"attribute\":null,\"name\":\"story\",\"required\":false,\"scope\":\"record\",\"selector\":\"div.cp-handmatchup-content\",\"value\":\"text\"},{\"attribute\":null,\"name\":\"posted\",\"required\":false,\"scope\":\"document\",\"selector\":\"div.cp-single-matchup div.text-end\",\"value\":\"text\"},{\"attribute\":null,\"name\":\"source_url\",\"required\":true,\"scope\":\"document\",\"selector\":\"link[rel=\\\"canonical\\\"]\",\"value\":\"href\"}],\"mode\":\"dom\",\"record_selector\":\"div.cp-handmatchup-content\"}"
          }
        ]
      },
      "limits": {
        "max_source_bytes": 1073741824,
        "max_rows": 200000,
        "max_requests": 600
      },
      "collection": {
        "discovery": {
          "kind": "json_api",
          "url": "https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified",
          "pagination": {
            "kind": "page_parameter",
            "parameter": "page",
            "first": 1
          },
          "records": {
            "pointer": ""
          },
          "link": {
            "pointer": "/link"
          },
          "identity": {
            "pointer": "/id"
          },
          "revision": {
            "pointer": "/modified"
          },
          "max_requests": 40
        },
        "pages": {
          "allowed_origins": [
            "https://www.cardplayer.com"
          ],
          "max_pages": 4000,
          "max_fetches_per_run": 500,
          "concurrency": 2,
          "min_interval_seconds": 1,
          "request_timeout_seconds": 30,
          "retries": 2,
          "revisit": {
            "kind": "revision_hint"
          }
        }
      }
    }
  ],
  "transform": {
    "engine": "duckdb_sql",
    "steps": [
      {
        "step_id": "hand_matchups_rows",
        "sql": "select page_id, cast(page_ordinal as integer) as page_ordinal, page_url, cast(page_fetched_at as timestamp with time zone) as page_fetched_at, cast(page_revision as integer) as page_revision, title, story, posted, source_url from hand_matchups",
        "description": "Carry each page's extracted rows through under the declared names, casting the provenance columns the collection appended: the merged relation is sealed as CSV, so every column reaches the transform as text."
      }
    ]
  },
  "checks": [],
  "units": [],
  "timezone": "UTC"
}

Part by part

The connector

public.https.collection@2.0.0 is the only adapter that takes a collection member. The member is required there and refused everywhere else. That adapter also requires credential_mode: "none", because a page collection reads public addresses and has no credential to present to a link discovery found. It requires limits.max_requests too.

A collection source may declare neither window nor closed. Both say the predecessor’s bytes are the whole answer for a range. A collection builds its result from a ledger, which decides page by page what to refetch.

Discovery

"discovery": {
  "kind": "json_api",
  "url": "https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified",
  "pagination": {"kind": "page_parameter", "parameter": "page", "first": 1},
  "records": {"pointer": ""},
  "link": {"pointer": "/link"},
  "identity": {"pointer": "/id"},
  "revision": {"pointer": "/modified"},
  "max_requests": 40
}

The publisher’s own JSON index lists the corpus. records.pointer is "" because the whole body is the array. link gives the detail page’s address, identity its stable identifier, and revision a hint the revisit rule reads.

Four discovery kinds exist:

`discovery.kind`	records	link	page identity	revision hint
`json_api`	`records.pointer` in RFC 6901 syntax, where `""` means the whole body	`link.pointer`	`identity.pointer`, else the resolved URL	`revision.pointer`
`html_index`	`records.selector`	the match’s `href`, or `link.attribute`	the resolved URL	none
`sitemap`	every `<url><loc>`	`loc`	the resolved URL	`lastmod`
`explicit`	`members[]` of `{page_id, url}`	given	given	none

The collection follows the declared pagination only. It never follows a link found inside a detail page, and it follows a <sitemapindex> one level.

Pages, and the budget

"pages": {
  "allowed_origins": ["https://www.cardplayer.com"],
  "max_pages": 4000,
  "max_fetches_per_run": 500,
  "concurrency": 2,
  "min_interval_seconds": 1,
  "request_timeout_seconds": 30,
  "retries": 2,
  "revisit": {"kind": "revision_hint"}
}

allowed_origins is the only required member, and it must include the discovery address’s own origin. min_interval_seconds is a whole number, because the document carries no fractional JSON number. revisit: revision_hint refetches a page only when the index says it changed. The alternatives are never and interval with seconds of at least 3600.

The Reader pin

The pin is required and must be html.web_extract, and the same settings apply to every page. mode: dom uses the closed CSS-like grammar of tags, ids, classes, attributes, descendants and children. A record_selector says what one row is. Each column names a selector, a value of text or href, a scope of document or record, and whether it is required.

Pin 1.1.0 rather than 1.0.0 for anything new. The two take identical settings and differ in reading order. 1.1.0 reads the words in the order they appear on the page, so a link keeps the comma that follows it and $<b>19</b>.<small>99</small> reads $19.99. 1.0.0 puts an element’s own words before the words of anything inside it, which glues neighbouring cells together. 1.0.0 stays unchanged because datasets built with it keep re-verifying.

The provenance columns

The transform receives the Reader’s columns plus seven reserved names:

page_id, page_url, page_fetched_at, page_content_sha256, page_revision, page_discovered_at, page_ordinal.

A pinned Reader that declares one of those names fails registration. Select them so every row names the page and the revision that produced it.

All seven arrive as text. The merged relation is stored as CSV, so selecting page_ordinal bare while declaring the column integer fails the build with TRANSFORM_COLUMN_TYPE_MISMATCH:

select page_id,
       cast(page_ordinal as integer) as page_ordinal,
       page_url,
       cast(page_fetched_at as timestamp with time zone) as page_fetched_at,
       cast(page_revision as integer) as page_revision,
       title, story, posted, source_url
from hand_matchups

Checks

This document declares none, which is a gap. A collection’s usual checks are key_uniqueness on ["page_id", "page_ordinal"], the grain, and a row_expectation floor. Add both before running against a corpus that matters.

Cadence

A collection has no scheduled refresh. Studio refuses a refresh whose action plan carries one, with RESYNC_REQUIRED before acquisition, so recording a cadence on this table changes nothing. Read the corpus again with an explicit resync. Size pages.max_fetches_per_run and limits.max_rows so that one run can cover the corpus, and read complete off the coverage block to know whether it did.

Register, run, inspect

mr-data dataset create --name "Card Player hand matchups" --json
mr-data recipe recipe.json --json

mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --sample --max-rows 2000 --json
mr-data status RUN_ID --json
mr-data receipt RUN_ID --json          # the pages coverage block

mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json
# a collection has no incremental refresh. Read the corpus again with an explicit table resync

What the run reports

A run writes a pages coverage block. The worked run recorded this one:

{"discovered": 2819, "discovery_requests": 29, "discovery_complete": true,
 "duplicates_dropped": 0, "known": 2819, "fetched_this_run": 500, "unchanged": 0, "changed": 497,
 "failed_this_run": 3, "pending": 2316, "failed": 3, "skipped": 0,
 "budget_exhausted": "max_fetches_per_run", "complete": false}

All fourteen members are always present. None is derived from another. A reader needs both halves: what this run reached, and what it still owes.

complete: false is not a failure. The run succeeded. The block says what the run reached and which ceiling stopped it. pending is work waiting, failed is work to retry, and discovery_complete says whether the run walked the index to the end.

The next refresh continues from the ledger. New pages come first in discovery order, then failures with the oldest first, then the revisits the rule names. Rows of pages this run did not fetch survive byte for byte. Six refreshes of 500 pages cover this corpus.

discovery_failure is null, or the code and one sentence for the first listing request the publisher refused. Read it before concluding a publisher’s index is empty. A refresh that already holds pages keeps every row and succeeds. A first run whose listing is refused fails with COLLECTION_DISCOVERY_FAILED rather than storing an empty corpus.

Traps

limits.max_requests counts every HTTP request, redirects and retries included. Registration only checks that discovery.max_requests + pages.max_fetches_per_run fits inside it. That check is necessary and not enough. This document’s 600 covers 40 + 500 plus a little, and it runs out early on any publisher that redirects. Where redirects are likely (trailing slashes, http to https, www, which covers most publishers) write limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run. A budget larger than the corpus needs costs nothing.

Report the shortfall as a backfill in progress, not as a failure and not as a complete corpus. A first run that fetched 500 of 2,819 pages built a real table, with real rows and a real number of pages still owed.

A page that disappears from the index deletes nothing. A page answering with different content replaces its own rows, increments page_revision and keeps the previous digest. Write that correction on the run record when it happens.

Check the discovery block by hand before registering. Check the address the first request goes to, and whether the pagination parameter is already written into it. Check that the records pointer names an array, and that the two budgets fit inside limits.max_requests. Registration refuses all four, and all four are cheap to fix before a run spends somebody’s rate limit.

html.web_extract reads the markup and never runs it. Nothing on the page executes. It follows no link, loads no image or stylesheet, and never consults what the page would look like once styled. A selector that matches more than one value where the column asked for one fails, and so does a required field that matches nothing.