Build your first dataset

Follow nine steps to turn one public CSV into a live Parquet table with a documented schema and passing checks.

Mostly Right hosts a small CSV for this walkthrough. It contains two rows and four columns, needs no credentials and republishes every minute in UTC.

curl "https://mostlyright.md/api/demo/sources/refresh.csv?interval=60"

station_id,observed_at,temperature_c,edition
prague-demo,2026-09-12T08:09:00.000Z,20.0,29820009
brno-demo,2026-09-12T08:09:00.000Z,19.5,29820009

Before you start, install mr-data and sign in. The identifiers shown below are examples. Your commands will print different ones.

1. Create the dataset page

Start by creating the dataset page. The command returns a dataset_id, which you will add to the recipe later.

mr-data dataset create --name "Demo station readings" --json

{
  "schema_version": "mostlyright-thin-client-v4-dataset-created.v1",
  "status": "dataset_created",
  "lane": "hosted",
  "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "name": "Demo station readings",
  "version": 1,
  "dashboard_url": "https://mostlyright.md/datasets/0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "navigation": {
    "url": "https://mostlyright.md/datasets/0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "target": "/datasets/0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "open_before_research": true,
    "follow_agent": true,
    "status": "dashboard_ready"
  },
  "note": "The page exists now and is empty. …"
}

Keep the dataset_id. You can open the page now and watch it fill in as the build runs. mr-data open /datasets/DATASET_ID --open uses a single-use address to open the page with your current terminal session.

2. Pick a category

Every dataset needs one primary category before you can register a recipe. Categories come from a fixed list. You can list them locally without contacting the backend.

mr-data dataset categories --json

{
  "schema_version": "mostlyright-dataset-categories.v1",
  "version": 1,
  "assignment_policy": "Exactly one primary category, chosen from the dataset purpose, tables and measured entities. …",
  "categories": [
    {
      "id": "climate-environment",
      "label": "Climate & Environment",
      "definition": "Weather, climate, pollution, ecosystems and environmental measurements.",
      "exclusions": "Agricultural output belongs to Agriculture & Food; energy supply belongs to Energy.",
      "examples": "Hourly weather observations; air quality; ocean temperatures"
    }
  ]
}

The eighteen ids are climate-environment, crypto, finance, economics-business, government-politics, people-society, health-medicine, science-research, technology, energy, agriculture-food, transport-mobility, places-infrastructure, education, sports, arts-media-entertainment, law-public-safety, other.

3. Save the category

mr-data dataset set 0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90 \
  --category climate-environment \
  --json

{
  "schema_version": "mostlyright-thin-client-v4-dataset-updated.v1",
  "status": "dataset_updated",
  "lane": "hosted",
  "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "version": 2,
  "fields": ["category_id"],
  "retried_after_conflict": false
}

The same command can update --name, --topics, --license and --description-file. Descriptions come from a file, or from - for stdin, because they may contain several paragraphs.

4. Write the recipe

The recipe is one JSON file that describes the whole build. It names the dataset, question, sources, table, SQL, checks, units and timezone. Save it as recipe.json and replace dataset.id with the identifier from step 1.

{
  "dataset": {
    "id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "name": "Demo station readings",
    "description": "Two synthetic weather stations read from the Mostly Right demo CSV endpoint, one row per station per published edition."
  },
  "question": {
    "text": "What temperature did each demo station report in each published edition?"
  },
  "table": {
    "name": "demo_station_readings",
    "description": "One row per station per edition of the demo feed.",
    "grain": ["station_id", "edition"],
    "columns": [
      {
        "name": "station_id",
        "type": "string",
        "nullable": false,
        "description": "The demo station the reading came from."
      },
      {
        "name": "observed_at",
        "type": "timestamp",
        "nullable": false,
        "description": "The UTC instant the edition states the reading was taken at."
      },
      {
        "name": "temperature_c",
        "type": "decimal",
        "nullable": false,
        "description": "Air temperature the station reported, in degrees Celsius."
      },
      {
        "name": "edition",
        "type": "integer",
        "nullable": false,
        "description": "The edition number the endpoint stamps on every row it published in one interval."
      }
    ]
  },
  "sources": [
    {
      "name": "demo_refresh_feed",
      "description": "The Mostly Right demo CSV endpoint, which publishes one stable edition per UTC minute.",
      "source_class": "user_url",
      "data_classification": "public",
      "locator": {
        "kind": "https_url",
        "display_locator": "https://mostlyright.md/api/demo/sources/refresh.csv?interval=60"
      },
      "rights_claim": {
        "claimed_basis": "terms_of_service_asserted",
        "claim_evidence_digest": "sha256:f8ecbfa29a29daf00908971739813f7bfebd42cab27b5a5d53824ea02cffcc0b",
        "claim_note": "Mostly Right publishes this endpoint as credential-free demonstration data; the digest is of https://mostlyright.md/terms/ as read on 12 September 2026."
      },
      "connector": {
        "adapter_id": "public.https",
        "credential_mode": "none",
        "origin": "https://mostlyright.md",
        "parameters": [{ "name": "accept", "value": "text/csv" }]
      },
      "limits": {
        "max_source_bytes": 65536,
        "max_rows": 1000,
        "max_requests": 1
      }
    }
  ],
  "transform": {
    "engine": "duckdb_sql",
    "steps": [
      {
        "step_id": "demo_station_readings",
        "sql": "select cast(station_id as varchar) as station_id, cast(observed_at as timestamp) as observed_at, cast(temperature_c as decimal(5,1)) as temperature_c, cast(edition as integer) as edition from demo_refresh_feed",
        "description": "Cast the four CSV fields into the types the table declares; a merged source relation reaches the transform as text."
      }
    ]
  },
  "checks": [
    {
      "check_id": "one_row_per_station_edition",
      "kind": "key_uniqueness",
      "enforcement": "required",
      "columns": ["station_id", "edition"],
      "description": "One reading per station per edition."
    },
    {
      "check_id": "temperature_present",
      "kind": "null_ceiling",
      "enforcement": "required",
      "columns": ["temperature_c"],
      "max_null_ppm": 0
    },
    {
      "check_id": "edition_is_not_empty",
      "kind": "row_expectation",
      "enforcement": "required",
      "min_rows": 1,
      "max_rows": 1000
    }
  ],
  "units": [{ "column": "temperature_c", "unit": "celsius" }],
  "timezone": "UTC"
}

Five details matter for this example. Write a recipe covers the full document.

The SQL reads each source using its sources[].name. In this recipe, the source named demo_refresh_feed becomes from demo_refresh_feed. A step_id cannot reuse a source name.

Steps run in the order listed. Each step creates a relation under its step_id, and the final step becomes the table.

Source columns arrive at the transform as text. The SQL therefore casts every column to the type declared by the table.

The rights_claim records your claim rather than looking one up. claimed_basis is one of unknown, prohibited, permission_asserted, public_domain_asserted, contractual_license_asserted and terms_of_service_asserted. claim_evidence_digest is the sha256 of the terms or licence bytes you actually read.

Recipe documents cannot contain fractional JSON numbers. Use integers for row counts, byte limits, parts-per-million bounds and spans in seconds. The client rejects a float when it reads the file.

5. Register it

mr-data recipe recipe.json --json

The server canonicalizes the document and computes its digest. If you send the same bytes again, the server returns the same recipe instead of creating another one.

{
  "schema_version": "mostlyright-thin-client-recipe.v1",
  "status": "recipe_registered",
  "lane": "hosted",
  "recipe_id": "b41d7a2c-9e55-4d18-8b07-1f3a6c95de24",
  "recipe_digest": "9f2c7b1e6a4d05c38e7f21ab94d6c0538172be49fa3d6c8b70e5a1d2c4f80963",
  "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "table_id": "7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45",
  "source_ids": ["d5a1c308-62b7-4e09-9f24-8ab35c1e07d6"],
  "idempotent_by": "recipe_digest"
}

Keep the recipe_digest. It is lowercase hexadecimal without a sha256: prefix. The run command checks the digest you provide and does not derive it again. A mismatched digest stops before a run is created.

Registration can return two warnings without failing. One warns when limits.max_source_bytes exceeds what a reader can reliably hold. The other lists columns without descriptions. Fix the document and register it again. A revision costs nothing and keeps the same table_id.

6. Start one full build

Full builds use progressive acquisition automatically. A long build exposes an inspection checkpoint after about five minutes and continues in the same run without a second approval or acquisition. This tiny example usually finishes before a checkpoint is useful. If a larger build is held for confirmation, review its projection before authorizing it as described in step 9.

mr-data run \
  --recipe b41d7a2c-9e55-4d18-8b07-1f3a6c95de24 \
  --digest 9f2c7b1e6a4d05c38e7f21ab94d6c0538172be49fa3d6c8b70e5a1d2c4f80963 \
  --full \
  --json

{
  "schema_version": "mostlyright-thin-client-v4-run.v1",
  "status": "run_queued",
  "lane": "hosted",
  "dashboard_url": "https://mostlyright.md/datasets/0f5c9a4e-…?run=3e8b17cd-…",
  "watch_command": "mr-data watch 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "run": {
    "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
    "workspace_id": "39ffafde-86ed-5d7b-b337-1c205ea79390",
    "recipe_id": "b41d7a2c-9e55-4d18-8b07-1f3a6c95de24",
    "recipe_digest": "9f2c7b1e…",
    "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "table_id": "7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45",
    "mode": "full",
    "status": "queued",
    "clamps": {},
    "created_at": "2026-09-12T08:10:04Z"
  }
}

For a deliberately bounded experiment, --sample remains available with an explicit ceiling. Its row ceiling limits output, not acquisition time, and a separate sample is not the full build’s reusable checkpoint. It is not a required step before --full.

7. Follow it

mr-data watch 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05

watch streams the run’s event log until the run finishes. If the stream disconnects, it can resume with --last-event-id and --from-seq.

{
  "schema_version": "mostlyright-thin-client-v4-watch.v1",
  "status": "run_terminal",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "terminal_event": "run_succeeded",
  "dashboard_url": "https://mostlyright.md/datasets/0f5c9a4e-…?run=3e8b17cd-…"
}

Use mr-data status RUN_ID when you want the current state once instead of a live stream.

{
  "schema_version": "mostlyright-thin-client-v4-run-status.v1",
  "status": "run_status_reported",
  "lane": "hosted",
  "terminal": true,
  "run": {
    "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
    "mode": "full",
    "status": "succeeded",
    "table_version_id": "5a91c7d4-8e30-4b62-91f7-c0d8e3b95a12",
    "candidate_digest": "3d9a…",
    "coverage": { "truncated": false },
    "created_at": "2026-09-12T08:10:04Z",
    "completed_at": "2026-09-12T08:10:51Z"
  }
}

When a run fails, failure contains failure_code, failure_detail and failed_stage. The result also includes a refusal object that scripts can branch on, and the command exits 2. See Fix a failed run.

8. Read what it built

Use mr-data checks to see the result of every declared check.

mr-data checks 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 --json

{
  "schema_version": "mostlyright-thin-client-v4-checks.v1",
  "status": "checks_reported",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "run_status": "succeeded",
  "checks_scope": "version",
  "check_count": 3,
  "checks_passed": 3,
  "checks_failed": 0,
  "checks": {
    "check 1": { "check_id": "one_row_per_station_edition", "passed": true, "detail": null },
    "check 2": { "check_id": "temperature_present", "passed": true, "detail": null },
    "check 3": { "check_id": "edition_is_not_empty", "passed": true, "detail": null }
  }
}

A run only reaches succeeded after every required check passes.

Use peek to inspect the rows.

mr-data peek 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 --rows 5 --json

{
  "schema_version": "mostlyright-thin-client-v4-peek.v1",
  "status": "peeked",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "row_count": 2,
  "columns": ["station_id", "observed_at", "temperature_c", "edition"],
  "schema": {
    "column 1": { "name": "station_id", "type": "string" },
    "column 2": { "name": "observed_at", "type": "timestamp" },
    "column 3": { "name": "temperature_c", "type": "decimal" },
    "column 4": { "name": "edition", "type": "integer" }
  },
  "sample": {
    "row 1": {
      "station_id": "prague-demo",
      "observed_at": "2026-09-12T08:09:00Z",
      "temperature_c": "20.0",
      "edition": "29820009"
    },
    "row 2": {
      "station_id": "brno-demo",
      "observed_at": "2026-09-12T08:09:00Z",
      "temperature_c": "19.5",
      "edition": "29820009"
    }
  }
}

peek reads the preview written by the run. --rows can return fewer preview rows, but it cannot expand the saved preview.

You can also query the table with SQL.

mr-data query 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 \
  "select station_id, max(temperature_c) as high from run_table group by station_id" \
  --json

The table is called run_table inside the query, regardless of its recipe name. Queries are read-only and contain one statement beginning with SELECT, WITH, EXPLAIN or DESCRIBE.

{
  "schema_version": "mostlyright-thin-client-v4-query.v1",
  "status": "query_answered",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "query_id": "f0a27c85-4b16-4d93-8e5a-91c37d0b6482",
  "state": "succeeded",
  "sql": "select station_id, max(temperature_c) as high from run_table group by station_id",
  "max_rows": 1000,
  "part_count": 1,
  "row_count": 2,
  "truncated": false,
  "elapsed_ms": 312,
  "result_digest": "c71e…",
  "replayed": false,
  "polls": 3,
  "rows": {
    "row 1": { "station_id": "prague-demo", "high": "20.0" },
    "row 2": { "station_id": "brno-demo", "high": "19.5" }
  }
}

The statement, row limit and prune block determine the query identifier. If you repeat the same command, Mostly Right returns the existing result instead of running the query again. Narrow a wide table with --field and --from/--to, or with --partition NAME=VALUE. If the query still selects too many parts, it returns QUERY_TOO_MANY_PARTS with the selected count and the limit.

9. Confirm a larger build when required

The full run started in step 6 is the build. Once it succeeds with complete coverage, no second run is needed. For larger datasets, the original start command may instead return this hold.

A full run starts immediately when its projection is below the threshold. Above the threshold, the platform saves it as awaiting_confirmation. The command prints the projection and exits 2.

{
  "schema_version": "mostlyright-thin-client-v4-run.v1",
  "status": "run_awaiting_confirmation",
  "lane": "hosted",
  "run_id": "a6f3c218-77d4-4b1e-84a0-3e9c5db02f71",
  "projected_bytes": 1073741824,
  "projected_runtime_s": 1840,
  "projected_cost": "4.20",
  "confirm_command": "mr-data run --recipe b41d7a2c-… --digest 9f2c7b1e… --mode full --confirm"
}

Paste confirm_command to approve the projected work. This two-row demo will not reach the threshold. Confirmation happens before acquisition; the progressive inspection checkpoint later continues automatically. New full builds do not create a sample/full pair. See How a build works.

The table is live on its first success

The table becomes live after its first successful run, such as the one in step 6. You do not need a separate release step. The live pointer then follows the newest version that passes its checks.

mr-data table 7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45 --json

{
  "schema_version": "mostlyright-thin-client-v4-promotion.v1",
  "status": "table_reported",
  "lane": "hosted",
  "table": {
    "table_id": "7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45",
    "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "name": "demo_station_readings",
    "promotion_status": "live",
    "live_version_id": "5a91c7d4-8e30-4b62-91f7-c0d8e3b95a12",
    "pinned_version_id": null,
    "schedule": null,
    "next_refresh_at": null,
    "last_refresh_at": null,
    "last_failure": null
  }
}

promotion_status is unpromoted, live or demoted. Once the cadence loop has observations, the same object includes an evidence member. It is absent until then.

demo_refresh_feed declares no window, no closed and no collection, so this table is resync-only. Studio refuses a scheduled refresh of it with RESYNC_REQUIRED before acquisition, and reading the feed again is an explicit resync. A table that refreshes on a schedule declares a continuation on every source. Publish and keep it fresh carries both gates.

Use mr-data promote to record how often the table should refresh and why. The command does not control whether the table is readable, and it does not make a resync-only table refresh.

mr-data promote 7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45 \
  --cadence "every 1h" \
  --why "The endpoint republishes once a minute, so hourly matches the source." \
  --json

The cadence starts as a seed. Studio adjusts the applied schedule using what each refresh observes about the source. Use mr-data reschedule TABLE --cadence … --lock to freeze it. An accepted schedule does not mean that a refresh has already run, and on this recipe none will. See Publish and keep it fresh.

Where it shows up

The dataset page at https://mostlyright.md/datasets/DATASET_ID carries the schema, the rows, the versions, the runs that built them, and the decision record mr-data dataset note writes.
The Public Dataset API supports bounded JSON queries and serves the Parquet download, once the dataset is published. See Use a public dataset and the Public Dataset API reference.
MCP, for an AI tool reading it directly. See Connect an AI tool.

New datasets are private to your workspace. Publish the dataset when you are ready to let others read it.

mr-data dataset publish 0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90 --mode public --json

--mode public lists the dataset in the public directory. link makes it available at an unlisted address, and private limits it to the workspace again. --show reports the current setting without changing it.

Bring the Parquet here

mr-data download 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 --output ./out --json

{
  "schema_version": "mostlyright-thin-client-v4-download.v1",
  "status": "artifacts_downloaded",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "output": "out",
  "version": {
    "manifest_digest": "8b14…",
    "part_count": 1,
    "row_count": 2,
    "parts_sealed_by_earlier_runs": 0
  },
  "artifacts": [
    {
      "artifact_id": "c0d95e71-…",
      "kind": "table_part",
      "media_type": "application/vnd.apache.parquet",
      "size_bytes": 2841,
      "digest": "sha256:…",
      "path": "out/table/parts/part-0000.parquet",
      "transferred": true
    }
  ]
}

The client verifies each artifact digest during the download and does not overwrite files already in the output folder. It downloads every part in the version before completing. The manifest may include parts written by earlier runs. If any part cannot be fetched, the download fails and reports how many parts are missing. It does not leave a partial table behind. --kind selects among table_parquet, table_manifest, table_part, column_profile, preview, receipt and raw_snapshot.

How a build works: the objects, the seven run states, and what is immutable.
Probe a source: check how a source responds before you write a recipe around it.
Write a recipe: every member of the document, with the rules that decide if it runs.
Fix a failed run: failure codes by stage, and what each one means.
Recipe examples: complete documents for JSON APIs, HTML pages, PDFs, GRIB2 and WebSocket venues.