# Benchmark scores and release dates Source: https://mostlyright.md/docs/recipes/benchmark-scores-join/ Join LLM benchmark scores to the release date and organization Epoch AI records for the same model. ## The question The Open LLM Leaderboard scores a model on MMLU-Pro and IFEval. Epoch AI records when a notable model was published, and by whom. The two files do not share an identifier. A benchmark row names a Hugging Face repository. A release row names the model in plain words. For each model on the first page of the leaderboard, submitted in 2019 or later, what were its MMLU-Pro and IFEval scores, and which Epoch release date and organization match its name? ## The sources | | Leaderboard page | Notable models | | --- | --- | --- | | Publisher | Hugging Face, the Open LLM Leaderboard `contents` dataset | Epoch AI | | Address | `https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard/contents&config=default&split=train&offset=0&length=100` | `https://epoch.ai/data/notable_ai_models.csv` | | Format | JSON, records under `/rows`, fields under `/row` | CSV, heading row, `text/csv` | | Cadence | the dataset card was last modified on 20 March 2025; this address is one page of 100 rows | the CSV is republished daily | | Rights | `unknown`. The dataset card states no licence | `contractual_license_asserted`. Epoch publishes the file under Creative Commons Attribution | | Why | `fullname`, `MMLU-PRO`, `IFEval` and `Submission Date` | `Model`, `Organization` and `Publication date` | The leaderboard serves 100 rows a page and 4,576 rows in total. `public.https` performs one request, so this document reads the first page. The last section says how the remaining pages join the same way. The two publishers share no key. The recipe builds one by lowercasing each name and deleting every character that is not a letter or a digit. `openai/gpt-4` and `GPT-4` both become `gpt4`. ## The recipe ```json title="recipe.json" { "dataset": { "name": "LLM benchmark scores by model since 2019" }, "question": { "text": "For each model on the first page of the Open LLM Leaderboard submitted since 2019, what were its MMLU-Pro and IFEval scores, and which Epoch AI release date and organization match its name?" }, "table": { "name": "benchmark_scores", "description": "One row per leaderboard model on the page.", "grain": [ "fullname" ], "columns": [ { "name": "fullname", "type": "string", "nullable": false, "description": "The Hugging Face repository the leaderboard scored, owner and model name.", "presentation": { "chart": "none" } }, { "name": "model_key", "type": "string", "nullable": false, "description": "The join key the recipe built from the repository name: lowercase letters and digits only." }, { "name": "mmlu_pro", "type": "decimal", "nullable": true, "description": "The model's MMLU-Pro score, as the leaderboard normalized it.", "presentation": { "chart": "histogram", "bins": 10 } }, { "name": "ifeval", "type": "decimal", "nullable": true, "description": "The model's IFEval score, as the leaderboard normalized it." }, { "name": "submitted_on", "type": "string", "nullable": true, "description": "The date the score was submitted, as the leaderboard prints it, YYYY-MM-DD." }, { "name": "published_on", "type": "string", "nullable": true, "description": "The earliest publication date Epoch AI records for a notable model with the same key, null when none matches." }, { "name": "organization", "type": "string", "nullable": true, "description": "The organization Epoch AI names for that model, null when none matches." }, { "name": "release_name", "type": "string", "nullable": true, "description": "The model name Epoch AI prints for that key, null when none matches." }, { "name": "has_release_date", "type": "boolean", "nullable": false, "description": "Whether an Epoch AI row matched the join key.", "presentation": { "chart": "share" } } ] }, "sources": [ { "name": "leaderboard_page", "description": "The first 100 rows of the Open LLM Leaderboard contents dataset.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard/contents&config=default&split=train&offset=0&length=100" }, "rights_claim": { "claimed_basis": "unknown", "claim_evidence_digest": "sha256:679356c3eb8167ae09fe9e46291c9f9b201ac303f20cfa3a9977e7ec43ce5133", "claim_note": "The dataset card at huggingface.co/datasets/open-llm-leaderboard/contents states no licence. This recipe claims no permission and no open licence. The digest is the card as read on 22 October 2026." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://datasets-server.huggingface.co", "parameters": [ { "name": "reader.family_id", "value": "json.tabular" }, { "name": "reader.family_version", "value": "1.1.0" }, { "name": "reader.decode_options", "value": "{\"columns\":[{\"name\":\"fullname\",\"pointer\":\"/row/fullname\",\"required\":true},{\"name\":\"mmlu_pro\",\"pointer\":\"/row/MMLU-PRO\",\"required\":false},{\"name\":\"ifeval\",\"pointer\":\"/row/IFEval\",\"required\":false},{\"name\":\"submitted_on\",\"pointer\":\"/row/Submission Date\",\"required\":false}],\"document_format\":\"json\",\"expand\":[],\"records_pointer\":\"/rows\"}" } ] }, "limits": { "max_source_bytes": 2097152, "max_rows": 100, "max_requests": 2 } }, { "name": "notable_models", "description": "Epoch AI's notable-models CSV: model name, organization and publication date.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://epoch.ai/data/notable_ai_models.csv" }, "rights_claim": { "claimed_basis": "contractual_license_asserted", "claim_evidence_digest": "sha256:d8423a724aa1e25dd4c930e158172e73380f7fba6015775b2db93a136b325e63", "claim_note": "Epoch AI states that the file is free to use, distribute and reproduce with credit, under Creative Commons Attribution. The digest is the documentation page as read on 22 October 2026." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://epoch.ai", "parameters": [ { "name": "reader.family_id", "value": "delimited_text" }, { "name": "reader.family_version", "value": "2.0.0" }, { "name": "reader.decode_options", "value": "{\"columns\":[{\"expected_header\":\"Model\",\"index\":0,\"name\":\"model_name\"},{\"expected_header\":\"Organization\",\"index\":1,\"name\":\"organization\"},{\"expected_header\":\"Publication date\",\"index\":2,\"name\":\"published_on\"}],\"delimiter\":\",\",\"doublequote\":true,\"encoding\":\"utf-8\",\"escapechar\":null,\"header\":true,\"quotechar\":\"\\\"\",\"skip_blank_rows\":false,\"skip_rows\":0,\"unselected_columns\":\"allow\"}" } ] }, "limits": { "max_source_bytes": 8388608, "max_rows": 20000, "max_requests": 2 } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "scores", "sql": "select fullname, lower(regexp_replace(split_part(fullname, '/', 2), '[^A-Za-z0-9]', '', 'g')) as model_key, try_cast(mmlu_pro as decimal(8, 3)) as mmlu_pro, try_cast(ifeval as decimal(8, 3)) as ifeval, submitted_on from leaderboard_page where submitted_on >= '2019-01-01'", "description": "Keep scores submitted since 2019 and build the join key from the repository name, the part after the slash." }, { "step_id": "releases", "sql": "select lower(regexp_replace(model_name, '[^A-Za-z0-9]', '', 'g')) as model_key, min(published_on) as published_on, any_value(organization) as organization, any_value(model_name) as release_name from notable_models where published_on >= '2019-01-01' group by 1", "description": "One row per normalized Epoch name, so the join cannot fan out. The date kept is the earliest." }, { "step_id": "benchmark_scores", "sql": "select s.fullname, s.model_key, s.mmlu_pro, s.ifeval, s.submitted_on, r.published_on, r.organization, r.release_name, r.model_key is not null as has_release_date from scores s left join releases r on s.model_key = r.model_key", "description": "Keep every scored model. Attach the release date where the built key matches." } ] }, "checks": [ { "check_id": "one_row_per_model", "kind": "key_uniqueness", "enforcement": "required", "columns": [ "fullname" ], "description": "One row per leaderboard repository. A duplicate means the release side was not unique on the key." }, { "check_id": "mmlu_pro_in_range", "kind": "value_range", "enforcement": "required", "columns": [ "mmlu_pro" ], "min_value": "0", "max_value": "100" }, { "check_id": "scores_present", "kind": "row_expectation", "enforcement": "required", "min_rows": 1 } ], "units": [], "timezone": "UTC" } ``` ## Part by part ### A key the recipe builds `fullname` looks like `openai/gpt-4`. `split_part(fullname, '/', 2)` keeps `gpt-4`. `regexp_replace` then deletes hyphens, dots and any other character that is not a letter or a digit, and `lower` finishes the key. The same expression runs on Epoch's `Model` column. A match is a coincidence of those reduced names, not an identifier either publisher issued. A leaderboard row whose name does not reduce to an Epoch name stays in the table with `has_release_date` false. That is the ordinary case for a fine-tune Epoch does not list. ### Two readers The leaderboard is one JSON document. Records sit at `/rows`, and the fields this table wants sit one level down at `/row`. `MMLU-PRO` and `Submission Date` are the publisher's names. The column `name` is what SQL sees (`mmlu_pro`, `submitted_on`), so the statement never has to quote a space. The Epoch file is a wide CSV. `delimited_text@2.0.0` projects three columns by index and checks the heading with `expected_header`. `unselected_columns` is `allow` because the rest of the file is populated and this table does not read it. `require_empty` would refuse the file. ### The join The release step groups by the key before the join. Two Epoch rows that reduce to the same key would otherwise duplicate every leaderboard row they match, and the totals computed from the table would be wrong. `min(published_on)` keeps the earliest ISO date. `any_value` keeps one organization and one printed name. Those two are labels, not the key. The join is a left join. An inner join would answer a different question: which scored models Epoch also lists. The flag is `r.model_key is not null`, not `r.published_on is not null`. A matched row with a missing date is still a match. `submitted_on` and `published_on` stay text. Both publishers print `YYYY-MM-DD`, and a string comparison against `'2019-01-01'` keeps that order. Casting them to `timestamp` is a later revision, once a sample shows every value parses. ### Checks `key_uniqueness` on `fullname` is the check that catches a join that fanned out. `value_range` bounds a normalized score between `"0"` and `"100"`, as decimal strings, because the document includes no fractional JSON number. `row_expectation` catches a page that came back empty. A score the leaderboard left null is allowed. The range check does not require the column to be present on every row. ## The rest of the leaderboard This document reads `offset=0`. The server returns 100 rows a page. The remaining pages are the same source repeated with `offset` set to 100, 200, and so on, up to the `num_rows_total` the first response reports, and the transform unions them before `scores`. That shape is the one in [JSON API](/docs/recipes/json-api/). A recipe names at most 256 sources, and 46 pages fit. ## Register, run, inspect Use the real table for one full build. For a bounded test first, follow the [scratch-table workflow](/docs/build/run-and-inspect/#test-before-the-full-build). A successful sample on the real table can publish truncated data. Stop if the forecast refuses; size the recipe before running. ```sh mr-data dataset create --name "LLM benchmark scores by model since 2019" --json mr-data recipe recipe.json --json mr-data recipe explain RECIPE_ID --json mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json mr-data watch RUN_ID --json mr-data peek RUN_ID --json mr-data checks RUN_ID --json mr-data query RUN_ID "select has_release_date, count(*) from run_table group by has_release_date" --json ``` The full run reads the Epoch file to its end. Verify unmatched rows and join coverage; a passing run does not establish that every model name matched correctly.