Benchmark scores and release dates
Join LLM benchmark scores to the release date and organization Epoch AI records for the same model.
The question
Section titled “The question”The Open LLM Leaderboard scores a model on MMLU-Pro and IFEval. Epoch AI records when a notable model was published, and by whom. The two files do not share an identifier. A benchmark row names a Hugging Face repository. A release row names the model in plain words.
For each model on the first page of the leaderboard, submitted in 2019 or later, what were its MMLU-Pro and IFEval scores, and which Epoch release date and organization match its name?
The sources
Section titled “The sources”| Leaderboard page | Notable models | |
|---|---|---|
| Publisher | Hugging Face, the Open LLM Leaderboard contents dataset |
Epoch AI |
| Address | https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard/contents&config=default&split=train&offset=0&length=100 |
https://epoch.ai/data/notable_ai_models.csv |
| Format | JSON, records under /rows, fields under /row |
CSV, heading row, text/csv |
| Cadence | the dataset card was last modified on 20 March 2025; this address is one page of 100 rows | the CSV is republished daily |
| Rights | unknown. The dataset card states no licence |
contractual_license_asserted. Epoch publishes the file under Creative Commons Attribution |
| Why | fullname, MMLU-PRO, IFEval and Submission Date |
Model, Organization and Publication date |
The leaderboard serves 100 rows a page and 4,576 rows in total. public.https performs one
request, so this document reads the first page. The last section says how the remaining pages join
the same way.
The two publishers share no key. The recipe builds one by lowercasing each name and deleting every
character that is not a letter or a digit. openai/gpt-4 and GPT-4 both become gpt4.
The recipe
Section titled “The recipe”{ "dataset": { "name": "LLM benchmark scores by model since 2019" }, "question": { "text": "For each model on the first page of the Open LLM Leaderboard submitted since 2019, what were its MMLU-Pro and IFEval scores, and which Epoch AI release date and organization match its name?" }, "table": { "name": "benchmark_scores", "description": "One row per leaderboard model on the page.", "grain": [ "fullname" ], "columns": [ { "name": "fullname", "type": "string", "nullable": false, "description": "The Hugging Face repository the leaderboard scored, owner and model name.", "presentation": { "chart": "none" } }, { "name": "model_key", "type": "string", "nullable": false, "description": "The join key the recipe built from the repository name: lowercase letters and digits only." }, { "name": "mmlu_pro", "type": "decimal", "nullable": true, "description": "The model's MMLU-Pro score, as the leaderboard normalized it.", "presentation": { "chart": "histogram", "bins": 10 } }, { "name": "ifeval", "type": "decimal", "nullable": true, "description": "The model's IFEval score, as the leaderboard normalized it." }, { "name": "submitted_on", "type": "string", "nullable": true, "description": "The date the score was submitted, as the leaderboard prints it, YYYY-MM-DD." }, { "name": "published_on", "type": "string", "nullable": true, "description": "The earliest publication date Epoch AI records for a notable model with the same key, null when none matches." }, { "name": "organization", "type": "string", "nullable": true, "description": "The organization Epoch AI names for that model, null when none matches." }, { "name": "release_name", "type": "string", "nullable": true, "description": "The model name Epoch AI prints for that key, null when none matches." }, { "name": "has_release_date", "type": "boolean", "nullable": false, "description": "Whether an Epoch AI row matched the join key.", "presentation": { "chart": "share" } } ] }, "sources": [ { "name": "leaderboard_page", "description": "The first 100 rows of the Open LLM Leaderboard contents dataset.", "source_class": "user_api", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard/contents&config=default&split=train&offset=0&length=100" }, "rights_claim": { "claimed_basis": "unknown", "claim_evidence_digest": "sha256:679356c3eb8167ae09fe9e46291c9f9b201ac303f20cfa3a9977e7ec43ce5133", "claim_note": "The dataset card at huggingface.co/datasets/open-llm-leaderboard/contents states no licence. This recipe claims no permission and no open licence. The digest is the card as read on 22 October 2026." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://datasets-server.huggingface.co", "parameters": [ { "name": "reader.family_id", "value": "json.tabular" }, { "name": "reader.family_version", "value": "1.1.0" }, { "name": "reader.decode_options", "value": "{\"columns\":[{\"name\":\"fullname\",\"pointer\":\"/row/fullname\",\"required\":true},{\"name\":\"mmlu_pro\",\"pointer\":\"/row/MMLU-PRO\",\"required\":false},{\"name\":\"ifeval\",\"pointer\":\"/row/IFEval\",\"required\":false},{\"name\":\"submitted_on\",\"pointer\":\"/row/Submission Date\",\"required\":false}],\"document_format\":\"json\",\"expand\":[],\"records_pointer\":\"/rows\"}" } ] }, "limits": { "max_source_bytes": 2097152, "max_rows": 100, "max_requests": 2 } }, { "name": "notable_models", "description": "Epoch AI's notable-models CSV: model name, organization and publication date.", "source_class": "user_url", "data_classification": "public", "locator": { "kind": "https_url", "display_locator": "https://epoch.ai/data/notable_ai_models.csv" }, "rights_claim": { "claimed_basis": "contractual_license_asserted", "claim_evidence_digest": "sha256:d8423a724aa1e25dd4c930e158172e73380f7fba6015775b2db93a136b325e63", "claim_note": "Epoch AI states that the file is free to use, distribute and reproduce with credit, under Creative Commons Attribution. The digest is the documentation page as read on 22 October 2026." }, "connector": { "adapter_id": "public.https", "credential_mode": "none", "origin": "https://epoch.ai", "parameters": [ { "name": "reader.family_id", "value": "delimited_text" }, { "name": "reader.family_version", "value": "2.0.0" }, { "name": "reader.decode_options", "value": "{\"columns\":[{\"expected_header\":\"Model\",\"index\":0,\"name\":\"model_name\"},{\"expected_header\":\"Organization\",\"index\":1,\"name\":\"organization\"},{\"expected_header\":\"Publication date\",\"index\":2,\"name\":\"published_on\"}],\"delimiter\":\",\",\"doublequote\":true,\"encoding\":\"utf-8\",\"escapechar\":null,\"header\":true,\"quotechar\":\"\\\"\",\"skip_blank_rows\":false,\"skip_rows\":0,\"unselected_columns\":\"allow\"}" } ] }, "limits": { "max_source_bytes": 8388608, "max_rows": 20000, "max_requests": 2 } } ], "transform": { "engine": "duckdb_sql", "steps": [ { "step_id": "scores", "sql": "select fullname, lower(regexp_replace(split_part(fullname, '/', 2), '[^A-Za-z0-9]', '', 'g')) as model_key, try_cast(mmlu_pro as decimal(8, 3)) as mmlu_pro, try_cast(ifeval as decimal(8, 3)) as ifeval, submitted_on from leaderboard_page where submitted_on >= '2019-01-01'", "description": "Keep scores submitted since 2019 and build the join key from the repository name, the part after the slash." }, { "step_id": "releases", "sql": "select lower(regexp_replace(model_name, '[^A-Za-z0-9]', '', 'g')) as model_key, min(published_on) as published_on, any_value(organization) as organization, any_value(model_name) as release_name from notable_models where published_on >= '2019-01-01' group by 1", "description": "One row per normalized Epoch name, so the join cannot fan out. The date kept is the earliest." }, { "step_id": "benchmark_scores", "sql": "select s.fullname, s.model_key, s.mmlu_pro, s.ifeval, s.submitted_on, r.published_on, r.organization, r.release_name, r.model_key is not null as has_release_date from scores s left join releases r on s.model_key = r.model_key", "description": "Keep every scored model. Attach the release date where the built key matches." } ] }, "checks": [ { "check_id": "one_row_per_model", "kind": "key_uniqueness", "enforcement": "required", "columns": [ "fullname" ], "description": "One row per leaderboard repository. A duplicate means the release side was not unique on the key." }, { "check_id": "mmlu_pro_in_range", "kind": "value_range", "enforcement": "required", "columns": [ "mmlu_pro" ], "min_value": "0", "max_value": "100" }, { "check_id": "scores_present", "kind": "row_expectation", "enforcement": "required", "min_rows": 1 } ], "units": [], "timezone": "UTC"}Part by part
Section titled “Part by part”A key the recipe builds
Section titled “A key the recipe builds”fullname looks like openai/gpt-4. split_part(fullname, '/', 2) keeps gpt-4.
regexp_replace then deletes hyphens, dots and any other character that is not a letter or a
digit, and lower finishes the key. The same expression runs on Epoch’s Model column.
A match is a coincidence of those reduced names, not an identifier either publisher issued. A
leaderboard row whose name does not reduce to an Epoch name stays in the table with
has_release_date false. That is the ordinary case for a fine-tune Epoch does not list.
Two readers
Section titled “Two readers”The leaderboard is one JSON document. Records sit at /rows, and the fields this table wants sit
one level down at /row. MMLU-PRO and Submission Date are the publisher’s names. The column
name is what SQL sees (mmlu_pro, submitted_on), so the statement never has to quote a space.
The Epoch file is a wide CSV. delimited_text@2.0.0 projects three columns by index and checks
the heading with expected_header. unselected_columns is allow because the rest of the file
is populated and this table does not read it. require_empty would refuse the file.
The join
Section titled “The join”The release step groups by the key before the join. Two Epoch rows that reduce to the same key
would otherwise duplicate every leaderboard row they match, and the totals computed from the table
would be wrong. min(published_on) keeps the earliest ISO date. any_value keeps one
organization and one printed name. Those two are labels, not the key.
The join is a left join. An inner join would answer a different question: which scored models
Epoch also lists. The flag is r.model_key is not null, not r.published_on is not null. A
matched row with a missing date is still a match.
submitted_on and published_on stay text. Both publishers print YYYY-MM-DD, and a string
comparison against '2019-01-01' keeps that order. Casting them to timestamp is a later
revision, once a sample shows every value parses.
Checks
Section titled “Checks”key_uniqueness on fullname is the check that catches a join that fanned out.
value_range bounds a normalized score between "0" and "100", as decimal strings, because
the document includes no fractional JSON number. row_expectation catches a page that came back
empty. A score the leaderboard left null is allowed. The range check does not require the column
to be present on every row.
The rest of the leaderboard
Section titled “The rest of the leaderboard”This document reads offset=0. The server returns 100 rows a page. The remaining pages are the
same source repeated with offset set to 100, 200, and so on, up to the num_rows_total the
first response reports, and the transform unions them before scores. That shape is the one in
JSON API. A recipe names at most 256 sources, and 46 pages fit.
Register, run, inspect
Section titled “Register, run, inspect”Use the real table for one full build. For a bounded test first, follow the scratch-table workflow. A successful sample on the real table can publish truncated data. Stop if the forecast refuses; size the recipe before running.
mr-data dataset create --name "LLM benchmark scores by model since 2019" --jsonmr-data recipe recipe.json --json
mr-data recipe explain RECIPE_ID --jsonmr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --jsonmr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --jsonmr-data watch RUN_ID --jsonmr-data peek RUN_ID --jsonmr-data checks RUN_ID --jsonmr-data query RUN_ID "select has_release_date, count(*) from run_table group by has_release_date" --jsonThe full run reads the Epoch file to its end. Verify unmatched rows and join coverage; a passing run does not establish that every model name matched correctly.