Source kinds and connectors

Choose the adapter with source.connector.adapter_id, then add the parameters, credentials and limits it accepts.

The shape of one source

Every source in document.sources carries seven required members. credential, binding, closed, window, collection and description are optional, and some adapters forbid them.

one source

{
  "name": "city_air_feed",
  "description": "Daily air quality readings, as the agency publishes them.",
  "source_class": "user_url",
  "data_classification": "public",
  "locator": {
    "kind": "https_url",
    "display_locator": "https://example.invalid/air/2024.csv"
  },
  "rights_claim": {
    "claimed_basis": "public_domain_asserted",
    "claim_evidence_digest": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
    "claim_note": "Published under an open licence by the publisher"
  },
  "connector": {
    "adapter_id": "public.https",
    "credential_mode": "none",
    "origin": "https://example.invalid",
    "parameters": [{ "name": "accept", "value": "text/csv" }]
  },
  "limits": {
    "max_source_bytes": 8388608,
    "max_rows": 5000000,
    "max_requests": 4
  }
}

Member	Required	Values
name	yes	lowercase snake_case identifier, unique in the document
description	no	1 to 240 characters. The dataset page draws it beside the endpoint
source_class	yes	mostlyright_sdk, external_adapter, user_file, user_url, user_api, database_extract, webhook, stream
data_classification	yes	public, internal, confidential, restricted
locator.kind	yes	sdk_connector, https_url, artifact, document_capture, webhook, stream, database
locator.display_locator	yes	1 to 2048 characters. The literal address, never templated
rights_claim.claimed_basis	yes	unknown, prohibited, permission_asserted, public_domain_asserted, contractual_license_asserted, terms_of_service_asserted
rights_claim.claim_evidence_digest	yes	sha256: plus 64 lowercase hex characters
rights_claim.claim_note	no	1 to 2000 characters
connector	yes	see The adapters
limits.max_source_bytes	yes	positive integer. Exceeding it fails every mode
limits.max_rows	no	positive integer. Clamps the merged relation
limits.max_requests	no	positive integer. Required on public.https.collection@2.0.0
credential	no	{"secret_name": …} or {"connection_id": …, "connection_version_digest": …}
binding	no	text (default) or typed
closed	no	boolean, default false
window	no	see Recipe document
collection	no	required on public.https.collection@2.0.0, forbidden elsewhere

limits.max_source_bytes is where the fetch stops. The run stops earlier where the read has a lower ceiling of its own. An unpinned json or parquet source fetches at most 16 MiB, and an unpinned ndjson source at most 37,745,664 bytes. Limits carries the table.

Studio refuses a source whose claimed_basis is prohibited instead of fetching it.

source_class and locator.kind must agree. The pairs are fixed:

source_class	Allowed locator.kind
database_extract	database
external_adapter	https_url
mostlyright_sdk	sdk_connector
stream	stream
user_api	https_url
user_file	artifact, document_capture
user_url	https_url
webhook	webhook

A recipe declares at most 256 sources. A collection counts as one source however many pages sit behind it.

The adapters

connector.adapter_id is a closed enum of ten values. Nothing else registers.

Adapter	What it fetches	Credential	Window	Collection
public.https	one public HTTPS address	none	yes	no
public.https.collection@2.0.0	an index plus many detail pages	none	forbidden	required
authenticated.https.api_key@1.0.0	one keyed HTTPS request	opaque_reference	yes	no
authenticated.https.api_key@2.0.0	a keyed HTTPS API, request pinned inline	opaque_reference	no	no
authenticated.https.connector@3.0.0	the same, plus budgets and async jobs	opaque_reference	no	no
stream.wss@1.0.0	stored batches of a recorded wss venue	none or reference	no	no
stream.wss@2.0.0	the same, keyed venue grammar	none or reference	no	no
snapshot.artifact@1.0.0	a Studio-held snapshot, redeemed	none	no	no
document.projection@1.0.0	a certified projection from the capture store	none	no	no
external.openligadb	the OpenLigaDB football API	none	no	no

public.https and external.openligadb carry no @version in the name. locator.connector_contract_version carries it when one is needed.

Reserved connector parameters

connector.parameters holds at most 64 {name, value} pairs. A name matches ^[A-Za-z][A-Za-z0-9._-]*$ and runs to 128 characters. A value runs to 2048 characters, except reader.decode_options, which runs to 262144.

The grammar accepts any name. These names are reserved:

Parameter	Meaning
reader.family_id	the Reader family this source decodes through
reader.family_version	the exact certified version of that family
reader.decode_options	canonical JSON text of that family’s accepted options
stream.members	the exact members a stream source reads, comma separated
stream.series	the series whose members a stream source follows
document.receipt_digest	the redeemed projection’s receipt
document.raw_document_digest	the source document it was projected from
document.projection_manifest_digest	the projection manifest
document.projection_content_digest	the projected content
document.candidate_manifest_digest	the candidate manifest
document.projection_policy_digest	the policy the projection ran under
document.media_type	the media type the projection produced
document.source_version	the source document version

Registration enforces these:

• All three reader.* names, or none. Registration refuses a fourth name under reader. rather than ignoring it.
• stream.members and stream.series are exclusive. A source states one or neither.
• The eight document.* names make one redemption, and they route the run to the batch worker.
• Any other name registers and digests into the source authority, and nothing acts on it. On the generic HTTPS family, connector.parameters supplies the values of the request parameters generic_api.request.parameters declares.

See Readers for the Reader pin and its 48 names.

Table-driven stream series

stream.series can name a harness-stream-series.v7 document whose selection_rule is table_members. The series reads one live table materialization at its discovery interval, projects the configured member IDs and typed context, and binds the resulting roster to a recorded stream. The source table must already contain the venue’s actual IDs; matching a schedule or external event to those IDs is upstream work. See Table-driven stream discovery for the series document, output-source template, freshness rules and replay identity.

public.https

One public HTTPS address, fetched without a credential.

Member	Required	Notes
adapter_id	yes	public.https
credential_mode	yes	none
origin	no	canonical HTTPS origin, no port, path, query or fragment, at most 255 characters
parameters	no	a Reader pin rides here
generic_api	never	forbidden

It may carry window, a request window or a snapshot one, and closed. It may not carry collection.

public.https with a Reader pin

{
  "name": "batch_winter_2025",
  "source_class": "user_url",
  "data_classification": "public",
  "locator": {
    "kind": "https_url",
    "display_locator": "https://yc-oss.github.io/api/batches/winter-2025.json"
  },
  "rights_claim": {
    "claimed_basis": "unknown",
    "claim_evidence_digest": "sha256:981a1cf0e469ff912bab4e28a52c04a546adfc06e31868ade1785df1cf94d50c"
  },
  "connector": {
    "adapter_id": "public.https",
    "credential_mode": "none",
    "origin": "https://yc-oss.github.io",
    "parameters": [
      { "name": "reader.family_id", "value": "json.tabular" },
      { "name": "reader.family_version", "value": "1.0.0" },
      { "name": "reader.decode_options", "value": "{\"columns\":[{\"name\":\"id\",\"pointer\":\"/id\",\"required\":true},{\"name\":\"name\",\"pointer\":\"/name\",\"required\":true}],\"document_format\":\"json\",\"expand\":[],\"records_pointer\":\"\"}" }
    ]
  },
  "limits": { "max_source_bytes": 4194304, "max_rows": 4096, "max_requests": 1 }
}

It refuses when:

• the address is not https://, or carries a port, userinfo or a fragment
• credential_mode is opaque_reference
• the response media type is not one the pinned Reader family accepts
• the fetched bytes exceed limits.max_source_bytes. The worker cannot truncate bytes that have already crossed the wire, so the run fails in every mode, including sample.

public.https.collection@2.0.0

One index plus many detail pages, as a single source. No script runs, and the worker follows no link it finds inside a detail page.

This adapter requires collection, limits.max_requests and credential_mode: "none". It forbids window and closed. It needs a Reader pin, and every page decodes through the same settings.

collection.discovery

Member	Required	Notes
kind	yes	json_api, html_index, sitemap, explicit
url	on the first three	absolute HTTPS, at most 2048 characters. Its origin must sit in pages.allowed_origins
records	json_api (pointer), html_index (selector)	exactly one of the two members
link	json_api (pointer)	on html_index, attribute names the anchor attribute. Absent means href
identity	no	RFC 6901 pointer to the page’s stable id. Absent means the page’s canonical URL
revision	no	RFC 6901 pointer to the change hint revisit.kind: revision_hint compares
members	explicit only	1 to 50000 {page_id, url} entries
pagination	no	absent means none
max_requests	no	1 to 1024. Absent means 64

pagination.kind follows the discovery kind. json_api takes none, page_parameter, offset_parameter, link_header, cursor. html_index takes none, page_parameter, next_selector. sitemap and explicit take neither. An absent first means 1 for page_parameter and 0 for offset_parameter. An absent step means 1, and only offset_parameter states it.

collection.pages

Only allowed_origins has no default. The worker applies a stated default for every other member.

Member	Range	Absent means
allowed_origins	1 to 8 canonical HTTPS origins	required
max_pages	1 to 50000	50000
max_fetches_per_run	1 to 10000	10000
concurrency	1 to 10,000 requested operations	admitted capacity, capped at 25 concurrent fetches by the worker
min_interval_seconds	1 to 3600	1
request_timeout_seconds	5 to 120	30
retries	0 to 3	2
revisit.kind	never, revision_hint, interval	never
revisit.seconds	3600 to 31536000, required on interval	no default

limits.max_requests charges one unit per HTTP request, redirect hops and retries included. Registration only checks that discovery.max_requests + pages.max_fetches_per_run fits inside it. Where redirects are likely, write limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run.

Provenance columns

The transform reads the Reader’s columns plus seven reserved names: page_id, page_url, page_fetched_at, page_content_sha256, page_revision, page_discovered_at, page_ordinal. Registration refuses a Reader that declares one of them. All seven arrive as VARCHAR. Cast them with cast(page_ordinal as integer) or cast(page_fetched_at as timestamp with time zone), or the build fails with TRANSFORM_COLUMN_TYPE_MISMATCH.

public.https.collection@2.0.0

{
  "name": "hand_matchups",
  "source_class": "user_url",
  "data_classification": "public",
  "locator": {
    "kind": "https_url",
    "display_locator": "https://www.cardplayer.com/poker-tools/hand-matchups"
  },
  "rights_claim": {
    "claimed_basis": "unknown",
    "claim_evidence_digest": "sha256:074f5983b4d0d33292ab8180f8d2ac54d379a9e9bc11b105496858062eb91965"
  },
  "connector": {
    "adapter_id": "public.https.collection@2.0.0",
    "credential_mode": "none",
    "origin": "https://www.cardplayer.com",
    "parameters": [
      { "name": "reader.family_id", "value": "html.web_extract" },
      { "name": "reader.family_version", "value": "1.1.0" },
      { "name": "reader.decode_options", "value": "{\"columns\":[{\"attribute\":null,\"name\":\"story\",\"required\":false,\"scope\":\"record\",\"selector\":\"div.cp-handmatchup-content\",\"value\":\"text\"}],\"mode\":\"dom\",\"record_selector\":\"div.cp-handmatchup-content\"}" }
    ]
  },
  "limits": { "max_source_bytes": 1073741824, "max_rows": 200000, "max_requests": 600 },
  "collection": {
    "discovery": {
      "kind": "json_api",
      "url": "https://www.cardplayer.com/wp-json/wp/v2/hand-matchups?per_page=100&_fields=id,date,link,title,modified",
      "pagination": { "kind": "page_parameter", "parameter": "page", "first": 1 },
      "records": { "pointer": "" },
      "link": { "pointer": "/link" },
      "identity": { "pointer": "/id" },
      "revision": { "pointer": "/modified" },
      "max_requests": 40
    },
    "pages": {
      "allowed_origins": ["https://www.cardplayer.com"],
      "max_pages": 4000,
      "max_fetches_per_run": 500,
      "concurrency": 2,
      "min_interval_seconds": 1,
      "request_timeout_seconds": 30,
      "retries": 2,
      "revisit": { "kind": "revision_hint" }
    }
  }
}

It refuses when:

• the source states window or closed
• limits.max_requests is absent
• credential_mode is not none
• the discovery URL’s origin sits outside allowed_origins
• discovery.max_requests + pages.max_fetches_per_run exceeds limits.max_requests
• an explicit list repeats a page_id or exceeds pages.max_pages

A first run whose listing request fails ends COLLECTION_DISCOVERY_FAILED instead of storing an empty corpus.

Spending the fetch budget is not a failure. The run succeeds with pages.complete: false and names the budget that ended it in budget_exhausted. An explicit resync is a full run from the beginning: it has no incremental predecessor or saved-ledger continuation.

The ledger records what that full run observed, but it is not yet a normal-refresh materialization capability. A collection-only or mixed recipe receives RESYNC_REQUIRED before acquisition on a normal refresh. Use an explicit resync to rebuild the collection from the beginning until a compatible incremental materializer is available.

authenticated.https.api_key@1.0.0

One keyed request against one origin.

Member	Required	Notes
adapter_id	yes	authenticated.https.api_key@1.0.0
credential_mode	yes	must be opaque_reference
origin	yes	canonical HTTPS origin
parameters	yes	must contain exactly one credential_scheme, value bearer or x-api-key
generic_api	never	forbidden

The source carries credential, and this adapter accepts a window.

authenticated.https.api_key@1.0.0

{
  "connector": {
    "adapter_id": "authenticated.https.api_key@1.0.0",
    "credential_mode": "opaque_reference",
    "origin": "https://observations.example",
    "parameters": [{ "name": "credential_scheme", "value": "bearer" }]
  },
  "credential": { "secret_name": "noaa_token" }
}

It refuses a credential_scheme that is missing, stated twice, or valued outside the two. Nothing infers a scheme from a URL.

authenticated.https.api_key@2.0.0

The generic HTTPS API. generic_api pins the request, the credential presentation, the response reader and the pagination inline, and the recipe carries every value.

Member	Required
credential_mode	opaque_reference
origin	yes
generic_api.request	yes. method (GET or POST), path_template, static_query, static_headers, parameters
generic_api.auth	yes. bearer, header (with header_name, value_prefix, optional key_encoding) or query (with query_name)
generic_api.response	yes. format (json, ndjson, csv, parquet), records_pointer, reader
generic_api.pagination	yes. none, page, offset, cursor or link
generic_api.limits	forbidden
generic_api.job	forbidden

auth is the only credential authority here, and a credential_scheme parameter is forbidden. POST requires body_template, and GET forbids it.

generic_api.response.reader takes family_id, family_version, decode_options and options_digest. options_digest is the bare sha256 of the canonical option bytes. family_id is json.tabular, delimited_text or parquet.tabular. family_version is 1.0.0 or 1.1.0, and parquet.tabular takes 1.0.0 only. The family must match the format: json and ndjson take json.tabular, csv takes delimited_text, parquet takes parquet.tabular.

Pagination bounds: page, offset, cursor and link each take max_pages 1 to 100. offset takes page_size 1 to 10000 and offset_start 0 to 1000000000. page takes page_start 0 to 1000000. link takes link_relation: "next" and nothing else.

authenticated.https.api_key@2.0.0

{
  "connector": {
    "adapter_id": "authenticated.https.api_key@2.0.0",
    "credential_mode": "opaque_reference",
    "origin": "https://observations.example",
    "parameters": [{ "name": "category", "value": "7" }],
    "generic_api": {
      "request": {
        "method": "GET",
        "path_template": "/daily/{category}",
        "static_query": [{ "name": "include", "value": "summary" }],
        "static_headers": [{ "name": "Accept", "value": "text/csv" }],
        "parameters": [{ "name": "category", "type": "integer", "required": true }]
      },
      "auth": { "kind": "header", "header_name": "X-Custom-Token", "value_prefix": "Key " },
      "response": {
        "format": "csv",
        "records_pointer": "",
        "reader": {
          "family_id": "delimited_text",
          "family_version": "1.0.0",
          "decode_options": { "delimiter": ",", "encoding": "utf-8", "header": true },
          "options_digest": "21da9722f5d3fc8df5c8a93a4578bbfbe2680f844e38a0b8b062300f0c5d474a"
        }
      },
      "pagination": { "kind": "page", "max_pages": 3, "page_parameter": "page", "page_start": 1 }
    }
  },
  "credential": { "secret_name": "observations_key" }
}

The {category} placeholder in path_template resolves from connector.parameters. A v4 recipe cannot name a mutable dataset setting.

It refuses when:

• origin is absent
• generic_api is absent
• a credential_scheme parameter appears
• generic_api carries limits or job
• a setting_name appears in auth or in a request parameter

authenticated.https.connector@3.0.0

The @2.0.0 grammar plus two members no other adapter may carry.

generic_api.limits is required here: max_pages 1 to 1000, max_records 1 to 10000000, max_bytes 1 to 268435456, max_seconds 1 to 900. This max_pages counts every request the acquisition makes, the submit, each status poll and each chunk. The top-level pagination.max_pages still stops at 100.

generic_api.job is optional and describes an async submit, poll and fetch cycle. It takes status.path_template, succeeded_values (1 to 8), failed_values and pending_values (up to 8 each), and pending_http_statuses (up to 4, each 200 to 299). It also takes poll_interval_seconds 1 to 60, max_wait_seconds 1 to 900, and chunking.max_chunks 1 to 1000.

auth accepts the v2 kinds plus the derived-credential kinds and extra_static_headers (up to 8). A JWT exchange takes issuer_template, subject_template, audience and lifetime_seconds 60 to 3600.

stream.wss@1.0.0 and stream.wss@2.0.0

A recorded wss venue. The build touches no venue and reads the batches mr-data stream already recorded.

Member	Required	Notes
source_class	yes	stream
locator.kind	yes	stream
stream_connector_document_digest	yes	sha256: plus 64 hex. The registered connector document this capture binds to
parameters	no	stream.members (comma separated) or stream.series, never both

A recipe may name either version.

stream.wss@2.0.0

{
  "name": "orderbook",
  "source_class": "stream",
  "data_classification": "public",
  "locator": { "kind": "stream", "display_locator": "wss://venue.example/ws" },
  "rights_claim": {
    "claimed_basis": "terms_of_service_asserted",
    "claim_evidence_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
  },
  "connector": {
    "adapter_id": "stream.wss@2.0.0",
    "credential_mode": "none",
    "stream_connector_document_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
    "parameters": [{ "name": "stream.members", "value": "alpha" }]
  },
  "limits": { "max_source_bytes": 268435456 }
}

It refuses a source that omits stream_connector_document_digest, and a source that states both stream.members and stream.series. That digest names the venue the recorded bytes came off. A build has no duration flag, because the recorded batches bound it. See Live streams.

snapshot.artifact@1.0.0

A snapshot Studio already holds, redeemed rather than fetched. source_class is user_file and locator.kind is artifact. locator.artifact carries snapshot_id, content_digest, size_bytes and media_type (text/html).

The adapter needs a Reader pin and no credential. Its query is exact: snapshot_id, content_digest, size_bytes, media_type, data_format, filename. data_format must be csv.

document.projection@1.0.0

A certified table projection out of a document capture, redeemed from the operator’s capture store. source_class is user_file and locator.kind is document_capture. No fetch happens, and the receipt records this adapter as the route.

This adapter rides on the eight reserved document.* parameters. Declaring any of them routes the run to the batch worker, the only shape that mounts the capture store. The data formats are csv and parquet. See Documents and files.

external.openligadb

The OpenLigaDB football API, over the public HTTPS boundary with the host fixed. source_class is external_adapter, locator.kind is https_url, data format json. It supports snapshot, historical, live and refresh acquisition.

Windows, snapshots and closed sources

Three optional members change what a refresh fetches. Recipe document carries the full grammar.

window.request narrows the address to a slice instead of re-downloading the corpus. Only public.https and authenticated.https.api_key@1.0.0 compose their request from the declared address, so only those two take a window. Every other adapter refuses one by name.

window.snapshot records an address whose current state has no time in its URL. The partition key is the run’s own acquisition day, [D, D+1), and the worker stamps every decoded row column = D. max_partitions runs 1 to 3660. A snapshot source is never closed and carries neither physical_scopes nor bootstrap. It has no normal-refresh action: Studio returns RESYNC_REQUIRED before acquisition. An explicit resync may acquire another whole snapshot.

closed: true says this source’s range is over. A full build or explicit resync acquires it. With a predecessor, it is the only condition that authorizes reuse_predecessor in a strict refresh plan. A closed source rides beside a request window in that plan: the window acquires its slice and the closed source reuses its sealed bytes. A closed-only plan has no bounded materializer, so Studio returns RESYNC_REQUIRED before acquisition.

Studio may record a legacy settled-source status when its most recent consecutive cadence observations reported changed: false under the same digest. There must be at least STUDIO_V4_SETTLED_AFTER of them (default 2, deployment-configurable). One changed observation, or one differing digest, ends the streak. Only an observation where the worker actually asked the publisher counts. This status does not authorize reuse_predecessor in a strict refresh plan. Studio re-verifies a settled source once nobody has looked at it for max(cadence interval × 8, 24 h). It also re-verifies the least-recently-verified settled sources of each run, whatever their age.

HTTPS acquisition sends conditional requests with If-None-Match and If-Modified-Since where it holds validators. A 304 is the publisher saying the bytes have not changed, and it hands back no body. It records an unchanged acquisition; it does not authorize reuse_predecessor in a strict refresh plan or turn an unwindowed whole representation into an incremental refresh. An unwindowed source has no normal-refresh action: Studio returns RESYNC_REQUIRED before acquisition, and an explicit resync is required to read it again.

Finding a source

mr-data catalog search asks the public-source catalogue one question and ranks what it holds.

mr-data catalog search "daily rainfall by station, United States, since 2015" \
  --format csv --limit 25 --json

--limit runs 1 to 25 and defaults to 10. --format states a requirement and does not filter. An entry that does not declare the format still comes back, ranked, with disposition: "refused" and filters_match: false.

Each ranked entry includes a disposition of admitted, human_escalation_required or refused. The search itself reports catalog_searched, catalog_no_admitted_source or catalog_search_incomplete.

The search fetches nothing and registers no source. A catalogue entry is discovery evidence over one provider’s partial sweep. Picking one neither acquires the resource nor establishes usage rights. Route it through one of the adapters and write the rights_claim yourself.

Where the limits are

Limits and ceilings collects every bound on this page, plus the ones the run and the table carry. Refusal codes sit on Run states and errors.