# Build your first dataset

Source: https://mostlyright.md/docs/start/first-dataset/

Follow nine steps to turn one public CSV into a live Parquet table with a documented schema and passing checks.

Mostly Right hosts a small CSV for this walkthrough. It contains two rows and four columns, needs no
credentials and republishes every minute in UTC.

```bash
curl "https://mostlyright.md/api/demo/sources/refresh.csv?interval=60"
```

```text
station_id,observed_at,temperature_c,edition
prague-demo,2026-09-12T08:09:00.000Z,20.0,29820009
brno-demo,2026-09-12T08:09:00.000Z,19.5,29820009
```

Before you start, [install `mr-data` and sign in](/docs/start/install/). The identifiers shown below
are examples. Your commands will print different ones.

**Already have a dataset page?** (caution)
If the dataset was created on mostlyright.md and you were handed its `dataset_id`, its page is
already open and waiting for this build. Skip step 1 and the `mr-data open` command: do not create
another dataset. Put that `dataset_id` in the recipe's `dataset.id`, or registering the recipe
creates a second dataset under the recipe's name. From mr-data 0.28.0, `dataset create` finds that
page itself and answers `open_request_reused` with its `dataset_id` instead of creating another.


## 1. Create the dataset page

Start by creating the dataset page. The command returns a `dataset_id`, which you will add to the
recipe later.

```bash
mr-data dataset create --name "Demo station readings" --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-dataset-created.v1",
  "status": "dataset_created",
  "lane": "hosted",
  "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "name": "Demo station readings",
  "version": 1,
  "dashboard_url": "https://mostlyright.md/datasets/0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "navigation": {
    "url": "https://mostlyright.md/datasets/0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "target": "/datasets/0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "open_before_research": true,
    "follow_agent": true,
    "status": "dashboard_ready"
  },
  "note": "The page exists now and is empty. …"
}
```

Keep the `dataset_id`. You can open the page now and watch it fill in as the build runs.
`mr-data open /datasets/DATASET_ID --open` uses a single-use address to open the page with your
current terminal session.

## 2. Pick a category

Every dataset needs one primary category before you can register a recipe. Categories come from a
fixed list. You can list them locally without contacting the backend.

```bash
mr-data dataset categories --json
```

```json
{
  "schema_version": "mostlyright-dataset-categories.v1",
  "version": 1,
  "assignment_policy": "Exactly one primary category, chosen from the dataset purpose, tables and measured entities. …",
  "categories": [
    {
      "id": "climate-environment",
      "label": "Climate & Environment",
      "definition": "Weather, climate, pollution, ecosystems and environmental measurements.",
      "exclusions": "Agricultural output belongs to Agriculture & Food; energy supply belongs to Energy.",
      "examples": "Hourly weather observations; air quality; ocean temperatures"
    }
  ]
}
```

The eighteen ids are `climate-environment`, `crypto`, `finance`, `economics-business`,
`government-politics`, `people-society`, `health-medicine`, `science-research`, `technology`,
`energy`, `agriculture-food`, `transport-mobility`, `places-infrastructure`, `education`,
`sports`, `arts-media-entertainment`, `law-public-safety`, `other`.

## 3. Save the category

```bash
mr-data dataset set 0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90 \
  --category climate-environment \
  --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-dataset-updated.v1",
  "status": "dataset_updated",
  "lane": "hosted",
  "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "version": 2,
  "fields": ["category_id"],
  "retried_after_conflict": false
}
```

The same command can update `--name`, `--topics`, `--license` and `--description-file`. When you change the description, send the title and topics in the same command: a description sent on its own can make Studio write a new title and new topics from it.
Descriptions come from a file, or from `-` for stdin, because they may contain several paragraphs.

## 4. Write the recipe

The recipe is one JSON file that describes the whole build. It names the dataset, question,
sources, table, SQL, checks, units and timezone. Save it as `recipe.json` and replace `dataset.id`
with the identifier from step 1.

```json title="recipe.json"
{
  "dataset": {
    "id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "name": "Demo station readings"
  },
  "question": {
    "text": "What temperature did each demo station report in each published edition?"
  },
  "table": {
    "name": "demo_station_readings",
    "description": "One row per station per edition of the demo feed.",
    "grain": ["station_id", "edition"],
    "columns": [
      {
        "name": "station_id",
        "type": "string",
        "nullable": false,
        "description": "The demo station the reading came from."
      },
      {
        "name": "observed_at",
        "type": "timestamp",
        "nullable": false,
        "description": "The UTC instant the edition states the reading was taken at."
      },
      {
        "name": "temperature_c",
        "type": "decimal",
        "nullable": false,
        "description": "Air temperature the station reported, in degrees Celsius."
      },
      {
        "name": "edition",
        "type": "integer",
        "nullable": false,
        "description": "The edition number the endpoint stamps on every row it published in one interval."
      }
    ]
  },
  "sources": [
    {
      "name": "demo_refresh_feed",
      "description": "The Mostly Right demo CSV endpoint, which publishes one stable edition per UTC minute.",
      "source_class": "user_url",
      "data_classification": "public",
      "locator": {
        "kind": "https_url",
        "display_locator": "https://mostlyright.md/api/demo/sources/refresh.csv?interval=60"
      },
      "rights_claim": {
        "claimed_basis": "terms_of_service_asserted",
        "claim_evidence_digest": "sha256:f8ecbfa29a29daf00908971739813f7bfebd42cab27b5a5d53824ea02cffcc0b",
        "claim_note": "Mostly Right publishes this endpoint as credential-free demonstration data; the digest is of https://mostlyright.md/terms/ as read on 12 September 2026."
      },
      "connector": {
        "adapter_id": "public.https",
        "credential_mode": "none",
        "origin": "https://mostlyright.md"
      },
      "limits": {
        "max_source_bytes": 65536,
        "max_rows": 1000,
        "max_requests": 1
      }
    }
  ],
  "transform": {
    "engine": "duckdb_sql",
    "steps": [
      {
        "step_id": "demo_station_readings",
        "sql": "select cast(station_id as varchar) as station_id, cast(observed_at as timestamp) as observed_at, cast(temperature_c as decimal(5,1)) as temperature_c, cast(edition as integer) as edition from demo_refresh_feed",
        "description": "Cast the four CSV fields into the types the table declares; a merged source relation reaches the transform as text."
      }
    ]
  },
  "checks": [
    {
      "check_id": "one_row_per_station_edition",
      "kind": "key_uniqueness",
      "enforcement": "required",
      "columns": ["station_id", "edition"],
      "description": "One reading per station per edition."
    },
    {
      "check_id": "temperature_present",
      "kind": "null_ceiling",
      "enforcement": "required",
      "columns": ["temperature_c"],
      "max_null_ppm": 0
    },
    {
      "check_id": "edition_is_not_empty",
      "kind": "row_expectation",
      "enforcement": "required",
      "min_rows": 1,
      "max_rows": 1000
    }
  ],
  "units": [{ "column": "temperature_c", "unit": "celsius" }],
  "timezone": "UTC"
}
```

Five details matter for this example. [Write a recipe](/docs/build/write-a-recipe/) covers the full
document.

The SQL reads each source using its `sources[].name`. In this recipe, the source named
`demo_refresh_feed` becomes `from demo_refresh_feed`. A `step_id` cannot reuse a source name.

Steps run in the order listed. Each step creates a relation under its `step_id`, and the final step
becomes the table.

Source columns arrive at the transform as text. The SQL therefore casts every column to the type
declared by the table.

The `rights_claim` records your claim rather than looking one up. `claimed_basis` is one of `unknown`, `prohibited`,
`permission_asserted`, `public_domain_asserted`, `contractual_license_asserted` and
`terms_of_service_asserted`. `claim_evidence_digest` is the sha256 of the terms or licence bytes you
actually read.

Recipe documents cannot contain fractional JSON numbers. Use integers for row counts, byte limits,
parts-per-million bounds and spans in seconds. The client rejects a float when it reads the file.

## 5. Explain and register it

```bash
mr-data recipe explain recipe.json --json
mr-data recipe recipe.json --json
```

The server canonicalizes the document and computes its digest. If you send the same bytes again,
the server returns the same recipe instead of creating another one.

```json
{
  "schema_version": "mostlyright-thin-client-recipe.v1",
  "status": "recipe_registered",
  "lane": "hosted",
  "recipe_id": "b41d7a2c-9e55-4d18-8b07-1f3a6c95de24",
  "recipe_digest": "9f2c7b1e6a4d05c38e7f21ab94d6c0538172be49fa3d6c8b70e5a1d2c4f80963",
  "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
  "table_id": "7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45",
  "source_ids": ["d5a1c308-62b7-4e09-9f24-8ab35c1e07d6"],
  "idempotent_by": "recipe_digest"
}
```

Keep the `recipe_digest`. It is lowercase hexadecimal without a `sha256:` prefix. The `run` command
checks the digest you provide and does not derive it again. A mismatched digest stops before a run
is created.

Read every registration warning. They can flag source limits, missing column descriptions, refresh readiness or a source with no completed window. Fix relevant findings before running. A revision keeps the same `table_id` but changes the digest.

The demo source has no incremental continuation: this walkthrough builds one current edition. A scheduled refresh requires a supported plan or declared rebuild. Use `recipe explain` to see why a schedule would be held.

Forecast the registered revision before starting it:

```sh
mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json
```

Use the identifiers registration returned. Exit 3 means resize before running. This demo has no provider-paid calls.

## 6. Start one full build

Full builds use progressive acquisition automatically. A long build exposes an inspection
checkpoint after about five minutes and continues in the same run without a second approval or
acquisition. This tiny example usually finishes before a checkpoint is useful. If a larger build
is held for confirmation, review its projection before authorizing it as described in step 9.

```bash
mr-data run \
  --recipe b41d7a2c-9e55-4d18-8b07-1f3a6c95de24 \
  --digest 9f2c7b1e6a4d05c38e7f21ab94d6c0538172be49fa3d6c8b70e5a1d2c4f80963 \
  --full \
  --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-run.v1",
  "status": "run_queued",
  "lane": "hosted",
  "dashboard_url": "https://mostlyright.md/datasets/0f5c9a4e-…?run=3e8b17cd-…",
  "watch_command": "mr-data watch 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "run": {
    "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
    "workspace_id": "39ffafde-86ed-5d7b-b337-1c205ea79390",
    "recipe_id": "b41d7a2c-9e55-4d18-8b07-1f3a6c95de24",
    "recipe_digest": "9f2c7b1e…",
    "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "table_id": "7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45",
    "mode": "full",
    "status": "queued",
    "clamps": {},
    "created_at": "2026-09-12T08:10:04Z"
  }
}
```

For a deliberately bounded experiment, `--sample` remains available with an explicit ceiling.
Its row ceiling limits output, not acquisition time, and a separate sample is not the full
build's reusable checkpoint. It is not a required step before `--full`.

## 7. Follow it

```bash
mr-data watch 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05
```

`watch` streams the run's event log until the run finishes. If the stream disconnects, it can resume
with `--last-event-id` and `--from-seq`.

```json
{
  "schema_version": "mostlyright-thin-client-v4-watch.v1",
  "status": "run_terminal",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "terminal_event": "run_succeeded",
  "dashboard_url": "https://mostlyright.md/datasets/0f5c9a4e-…?run=3e8b17cd-…"
}
```

Use `mr-data status RUN_ID` when you want the current state once instead of a live stream.

```json
{
  "schema_version": "mostlyright-thin-client-v4-run-status.v1",
  "status": "run_status_reported",
  "lane": "hosted",
  "terminal": true,
  "run": {
    "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
    "mode": "full",
    "status": "succeeded",
    "table_version_id": "5a91c7d4-8e30-4b62-91f7-c0d8e3b95a12",
    "candidate_digest": "3d9a…",
    "coverage": { "truncated": false },
    "created_at": "2026-09-12T08:10:04Z",
    "completed_at": "2026-09-12T08:10:51Z"
  }
}
```

When a run fails, `failure` contains `failure_code`, `failure_detail` and `failed_stage`. The result
also includes a `refusal` object that scripts can branch on, and the command exits `2`. See [Fix a
failed run](/docs/build/troubleshooting/).

## 8. Read what it built

Use `mr-data checks` to see the result of every declared check.

```bash
mr-data checks 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-checks.v1",
  "status": "checks_reported",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "run_status": "succeeded",
  "checks_scope": "version",
  "check_count": 3,
  "checks_passed": 3,
  "checks_failed": 0,
  "checks": {
    "check 1": { "check_id": "one_row_per_station_edition", "passed": true, "detail": null },
    "check 2": { "check_id": "temperature_present", "passed": true, "detail": null },
    "check 3": { "check_id": "edition_is_not_empty", "passed": true, "detail": null }
  }
}
```

A run only reaches `succeeded` after every required check passes.

Use `peek` to inspect the rows.

```bash
mr-data peek 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 --rows 5 --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-peek.v1",
  "status": "peeked",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "row_count": 2,
  "columns": ["station_id", "observed_at", "temperature_c", "edition"],
  "schema": {
    "column 1": { "name": "station_id", "type": "string" },
    "column 2": { "name": "observed_at", "type": "timestamp" },
    "column 3": { "name": "temperature_c", "type": "decimal" },
    "column 4": { "name": "edition", "type": "integer" }
  },
  "sample": {
    "row 1": {
      "station_id": "prague-demo",
      "observed_at": "2026-09-12T08:09:00Z",
      "temperature_c": "20.0",
      "edition": "29820009"
    },
    "row 2": {
      "station_id": "brno-demo",
      "observed_at": "2026-09-12T08:09:00Z",
      "temperature_c": "19.5",
      "edition": "29820009"
    }
  }
}
```

`peek` reads the preview written by the run. `--rows` can return fewer preview rows, but it cannot
expand the saved preview.

You can also query the table with SQL.

```bash
mr-data query 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 \
  "select station_id, max(temperature_c) as high from run_table group by station_id" \
  --json
```

The table is called `run_table` inside the query, regardless of its recipe name. Queries are
read-only and contain one statement beginning with `SELECT`, `WITH`, `EXPLAIN` or `DESCRIBE`.

```json
{
  "schema_version": "mostlyright-thin-client-v4-query.v1",
  "status": "query_answered",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "query_id": "f0a27c85-4b16-4d93-8e5a-91c37d0b6482",
  "state": "succeeded",
  "sql": "select station_id, max(temperature_c) as high from run_table group by station_id",
  "max_rows": 1000,
  "part_count": 1,
  "row_count": 2,
  "truncated": false,
  "elapsed_ms": 312,
  "result_digest": "c71e…",
  "replayed": false,
  "polls": 3,
  "rows": {
    "row 1": { "station_id": "prague-demo", "high": "20.0" },
    "row 2": { "station_id": "brno-demo", "high": "19.5" }
  }
}
```

The statement, row limit and prune block determine the query identifier. If you repeat the same
command, Mostly Right returns the existing result instead of running the query again. Narrow a wide
table with `--field` and `--from`/`--to`, or with `--partition NAME=VALUE`. If the query still selects
too many parts, it returns `QUERY_TOO_MANY_PARTS` with the selected count and the limit.

## 9. Confirm a larger build when required

The full run started in step 6 is the build. Once it succeeds with complete coverage, no second
run is needed. For larger datasets, the original start command may instead return this hold.

A full run starts immediately when its projection is below the threshold. Above the threshold, the
platform saves it as `awaiting_confirmation`. The command prints the projection and exits `2`.

```json
{
  "schema_version": "mostlyright-thin-client-v4-run.v1",
  "status": "run_awaiting_confirmation",
  "lane": "hosted",
  "run_id": "a6f3c218-77d4-4b1e-84a0-3e9c5db02f71",
  "projected_bytes": 1073741824,
  "projected_runtime_s": 1840,
  "projected_cost": "4.20",
  "confirm_command": "mr-data run --recipe b41d7a2c-… --digest 9f2c7b1e… --mode full --confirm"
}
```

Paste `confirm_command` to approve the projected work. This two-row demo will not reach the
threshold. Confirmation happens before acquisition; the progressive inspection checkpoint later
continues automatically. New full builds do not create a sample/full pair. See [How a build
works](/docs/start/how-it-works/#the-confirmation-gate-and-progressive-checkpoint).

## The table is live on its first success

The table becomes live after its first successful run, such as the one in step
6. You do not need a separate release step. The live pointer then follows the newest version that
passes its checks.

```bash
mr-data table 7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45 --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-promotion.v1",
  "status": "table_reported",
  "lane": "hosted",
  "table": {
    "table_id": "7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45",
    "dataset_id": "0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90",
    "name": "demo_station_readings",
    "promotion_status": "live",
    "live_version_id": "5a91c7d4-8e30-4b62-91f7-c0d8e3b95a12",
    "pinned_version_id": null,
    "schedule": null,
    "next_refresh_at": null,
    "last_refresh_at": null,
    "last_failure": null
  }
}
```

`promotion_status` is `unpromoted`, `live` or `demoted`. Once the cadence loop has observations, the
same object includes an `evidence` member. It is absent until then.

`demo_refresh_feed` declares no window, no `closed` and no collection, so this table is
resync-only. Studio refuses a scheduled refresh of it with `RESYNC_REQUIRED` before acquisition, and
reading the feed again is an explicit resync. A table that refreshes on a schedule declares a
continuation on every source. [Publish and keep it fresh](/docs/build/publish-and-refresh/) carries
both gates.

Use `mr-data promote` to record how often the table should refresh and why. The command does not
control whether the table is readable, and it does not make a resync-only table refresh.

```bash
mr-data promote 7c2e4b90-3a16-4f5d-9c81-0d6b2e73af45 \
  --cadence "every 1h" \
  --why "The endpoint republishes once a minute, so hourly matches the source." \
  --json
```

The cadence starts as a seed. Studio adjusts the applied schedule using what each refresh observes
about the source. Use `mr-data reschedule TABLE --cadence … --lock` to freeze it. An accepted
schedule does not mean that a refresh has already run, and on this recipe none will. See [Publish and
keep it fresh](/docs/build/publish-and-refresh/).

## Where it shows up

- The dataset page at `https://mostlyright.md/datasets/DATASET_ID` carries the schema, the rows, the
  versions, the runs that built them, and the decision record `mr-data dataset note` writes.
- The Public Dataset API supports bounded JSON queries and serves the Parquet download, once the
  dataset is published. See [Use a public dataset](/docs/guides/use-public-datasets/) and the
  [Public Dataset API reference](/docs/reference/public-dataset-api/).
- MCP, for an AI tool reading it directly. See [Connect an AI tool](/docs/guides/mcp/).

New datasets are private to your workspace. Publish the dataset when you are ready to let others
read it.

```bash
mr-data dataset publish 0f5c9a4e-7d21-4c0b-9a3e-6b1f2d8c4e90 --mode public --json
```

`--mode public` lists the dataset in the public directory. `link` makes it available at an unlisted
address, and `private` limits it to the workspace again. `--show` reports the current setting
without changing it.

## Bring the Parquet here

```bash
mr-data download 3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05 --output ./out --json
```

```json
{
  "schema_version": "mostlyright-thin-client-v4-download.v1",
  "status": "artifacts_downloaded",
  "lane": "hosted",
  "run_id": "3e8b17cd-5f42-4a6e-b930-27c8ad1e6f05",
  "output": "out",
  "version": {
    "manifest_digest": "8b14…",
    "part_count": 1,
    "row_count": 2,
    "parts_sealed_by_earlier_runs": 0
  },
  "artifacts": [
    {
      "artifact_id": "c0d95e71-…",
      "kind": "table_part",
      "media_type": "application/vnd.apache.parquet",
      "size_bytes": 2841,
      "digest": "sha256:…",
      "path": "out/table/parts/part-0000.parquet",
      "transferred": true
    }
  ]
}
```

The client verifies each artifact digest during the download and does not overwrite files already
in the output folder. It downloads every part in the version before completing. The manifest may
include parts written by earlier runs. If any part cannot be fetched, the download fails and reports
how many parts are missing. It does not leave a partial table behind. `--kind` selects among `table_parquet`,
`table_manifest`, `table_part`, `column_profile`, `preview`, `receipt` and `raw_snapshot`.

## Next

- [How a build works](/docs/start/how-it-works/): the objects, the seven run states, and what is
  immutable.
- [Probe a source](/docs/build/probe-sources/): check how a source responds before you write a recipe
  around it.
- [Write a recipe](/docs/build/write-a-recipe/): every member of the document, with the rules that
  decide if it runs.
- [Fix a failed run](/docs/build/troubleshooting/): failure codes by stage, and what each one means.
- [Recipe examples](/docs/recipes/): complete documents for JSON APIs, HTML pages, PDFs, GRIB2 and
  WebSocket venues.