RISE Humanities Data Benchmark: news

Release v0.8.1

Tue, 03 Nov 2026 00:00:00 +0000

Added

Dataset release 2026-10-29.5 deposited (10.5281/zenodo.23821533, concept DOI 10.5281/zenodo.23821532) and recorded in scripts/data/dataset_releases.json; README and CITATION.cff link it, and CITATION.cff lists the v0.8.0 DOI.
.zenodo.json, generated from CITATION.cff by scripts/zenodo_metadata.py: the software deposit links the dataset and the JOHD articles and describes itself.

Changed

library_cards 3.0.0: format entered as pages moved to format on five cards; 00639255 pages, 00443771 shelfmark and the cross-reference 00414956 corrected; stored runs re-scored.
company_lists 3.0.0: swapped locations on 156089_1321088_41 corrected; stored runs re-scored.
business_letters 3.0.0: names matched ignoring case and whitespace, empty names no longer match; persons.json adds four spellings and drops four bad ones; the letter46 sender is marked as inferred; stored runs re-scored.
CITATION.cff titled "RISE Humanities Data Benchmark"; the README's DOI badges name the project and the dataset; the How it works graphic redrawn (docs/how-it-works.svg).
Stored rescored labels from before the version merge say 2.0.0 instead of 3.0.0 (194 runs); published_as removed from meta.json.
CITATION.cff records the University of Basel affiliation of 13 authors; the dataset deposit's description explains the benchmark without the README, and the dataset's CITATION.cff and deposit name the two JOHD articles that describe it; one list of 15 keywords for datapackage.json, CITATION.cff and the deposit; the dataset is titled "RISE Humanities Data Benchmark Dataset: Complete Stored Results".

Removed

Benchmark.remove_none_values() and resize_images(), hooks the runner never called.

Fixed

create_benchmark.py: class names with digits match the runner's; backslashes or """ in a title or description no longer break benchmark.py; stops at end of input instead of looping; works from any directory; no longer asks for a role description it discarded or a ranking metric its scorer does not return; no crash on redirected Windows output.
Card template: Judge Uses LLM defaults to false; cards name Arabic (ar).
Benchmark.convert_truth_to_json() overrides take effect.

The results as a citable dataset

Fri, 30 Oct 2026 00:00:00 +0000

Every result behind this website is now published as a dataset on Zenodo: 10.5281/zenodo.23821532. It holds every stored run and request since 2025, 3,805 runs and 166,380 requests in the first release, as Parquet tables with a metric dictionary, together with the original request and scoring records of every model answer. It is licensed CC BY 4.0, and the files live in their own repository.

What it is for. The website answers which model to use; the dataset lets you ask your own questions: cost per correct answer, how a model changed between versions, where models fail on the same input. Apart from the framework's own test benchmarks, nothing is filtered out: failed requests, unscored runs and retired configurations are rows with a status, so you choose what to include.

Versions. A new dataset release follows each software release. Every release is a complete snapshot, and its changelog says what changed, including which benchmarks were re-scored. Each run records the benchmark version that produced its score, so you can tell which scores are comparable. To cite the results, use the DOI above and name the dataset version you analysed; the About page has the details.

Release v0.8.0

Fri, 30 Oct 2026 00:00:00 +0000

Added

scripts/export_dataset/: builds dataset/, with runs, requests, scores_long and rescored_fields tables in Parquet and CSV, a metric dictionary, payload sidecars, coverage, diagnostics and a source manifest of hashed inputs.
Each run carries the benchmark version it was made and scored with (benchmark_version, benchmark_version_scored, benchmark_version_basis); dataset schema 1.3.0.
Per-request cost from recorded tokens at the price in force on the run's date, reasoning included, beside the stored cost; reasoning_tokens and reasoning cost columns on requests and runs.
dataset/ documentation: datapackage.json, README, changelog and CITATION.cff; data licensed CC BY 4.0.
dataset/examples/business_letters_cost.py: cost per correct extraction by provider and date.
scripts/package_dataset.py: packages a release, payload sidecars included, with SHA256SUMS; --release-repo syncs it into a clone of the release repository and writes its .zenodo.json, LICENSE and .gitattributes.
scripts/export_dataset/releases.py: records each deposited release and its DOI in scripts/data/dataset_releases.json; later builds list them in the dataset's CITATION.cff and changelog, with the benchmark versions added per release. The test_* benchmarks are left out of the dataset.
scripts/supervise_run.py (restarts a stalled run, the date pinned for the runner) and scripts/watch_run.py (reports spend and waste during a run), with tests.
scripts/export_dataset/crosscheck.py: reconciles the dataset with the website's run export from benchmark-frontend before packaging.
scripts/results_index.py: one shared reader for results/ and benchmarks_tests.csv, also used by scripts/compare/.
README: slides from DaSCHCon 2026 and the SGG workshop under Publications; the Results section describes the new website.
Version history of every benchmark in meta.json versions (version, date, commit, kind of change, summary), reconstructed from the git history, with 0.y.z before the cards; the cards' Versions list is generated from it, every version dated, and an integrity test checks the history; it includes the default output cap lowered to 16,384 tokens on 2026-09-18.
scripts/compare/generate_version_runs.py: run counts per benchmark version and each run's version for the website (collected_results/benchmark_versions.json, run_versions.json), a step of generate_all.
Each run records the benchmark version in scoring.json benchmark_version (version, commit, uncommitted changes to the benchmark's files), and each answer in its request file; a resumed run keeps the version of its earliest answer, marked mixed when its answers were made with different benchmark states. The version assignment uses it before the run's time (benchmark_version_basis = recorded).
Results for 2026-10-29: T1956, T1960, T1962 (3 tests, 22 requests) for ministral-3b-2512, verification runs.

Changed

magazine_pages example image: a page with seven advertisements.
Card versions published on the same day are merged: company_lists, duty_rosters, fraktur_adverts, general_meeting_minutes and library_cards 3.0.0 become 2.0.0, and business_letters 1.0.1 is part of 2.0.0; published_as records the former numbers.
business_letters is ranked on all 57 letters: meta.json ranking names the ranked letter set (label, rules) and ranks the 32 signed and the 25 unsigned letters apart (variants). Before, a model's best run on any of the three sets counted, and the 25 unsigned letters, which score higher, made up the top of the ranking.
Website code moved to the benchmark-frontend repository; scripts/ndr_export/ is now scripts/compare/, which builds only the comparison data, and pricing_resolver.py moved to scripts/.

Removed

NDR Core exports (multi-select data, visualizations, vars), scripts/sync_benchmark_pages.py, the site fields of contributors.json (photo, website, core) and the unused mkdocs.yml.

Fixed

An answer billed with finish_reason "error" (OpenRouter, Mistral) is saved and scored as a miss instead of dropped by the fatal-error check.
The 173 answers it dropped from 62 runs (2026-09-16 to 2026-10-26) are added as misses, with tokens and cost unknown; so are 28 inputs of 10 runs (2026-09-16 to 09-18) whose error records were deleted or whose answers a scorer exception dropped.
A scorer that raises no longer drops the answer: it is saved and scored as a miss. magazine_pages 1.0.1 counts a box that is not four numbers as a false positive, medieval_manuscripts 2.0.1 a folio entry that is not an object as missed; no stored score changes.
generate_compare_detail.py: removes a detail file once its run no longer needs one.

A new website for the benchmark

Thu, 29 Oct 2026 00:00:00 +0000

The benchmark website has been rebuilt. The address stays the same and old links keep working, but every page is new: a static site in the University of Basel's corporate design, generated directly from the repository and updated every night when the benchmarks or results change. Pages load faster, and what you see is always what the repository holds.

Comparing models. The leaderboard now has five views. The matrix shows each model's best run on all twelve benchmarks, and its column headings stay in view as you scroll. Which model for me? recommends a model for your material. The run list holds every run, Cost and speed sets scores against price and time, and the new Compare models view puts up to four models side by side. A filterable model browser helps you pick them, and for any benchmark the view shows where the models disagree input by input, with a link from each score to that input's answer and ground truth.

How sure a ranking is. Every score now carries its sampling margin (±, 95%), worked out from the run's individual inputs. Models whose difference is within that margin share a rank, marked ≈, so the leaderboard no longer separates models that cannot be told apart. Costs are given per 1,000 documents, so benchmarks of different size compare fairly. Scores from a run on a different prompt or more than a year old are marked.

A page for every model. All 158 models tested since 2025 have their own page, including the 37 legacy models that are no longer run. A page shows the model's results per benchmark and task, every run and its price history. Legacy models can also be switched on in the leaderboard; they are shown with their best legacy run and are not ranked.

Benchmarks, news and data. Each benchmark page shows its card, an example input with its ground truth, the results and the newest runs. In the leaderboard, explanations now appear on mouseover next to the figures they explain. The leaderboard, every benchmark's ranking and every run can be downloaded as CSV or JSON, and the news has an RSS feed.

Two talks in Bern: DaSCHCon 2026 and the SGG workshop on AI and historical materials

Mon, 26 Oct 2026 00:00:00 +0000

In the week of 19 October Maximilian Hindermann gave two talks on the benchmark at the University of Bern, to two different audiences: research data infrastructure at DaSCHCon, and historians, archivists and librarians at the Swiss Historical Society (SGG). All figures quoted in both talks come from benchmark v0.7.0, with runs up to 17 October 2026.

DaSCHCon 2026, 21 October. At "AI in Practice: Experiences, Workflows, and Platforms", the talk Benchmark First: Making Model Choice a Routine Step in Research Data Workflows showed how the benchmark structures RISE's consultations: settle which models the data may go to and what result is needed, reuse or build a matching benchmark, then compare models on it. The worked case was the nursing staff duty rosters of the INTERSCALE project. Across 93 models the price per sheet varied nearly five-hundredfold, and Claude Opus 5, first on Document Arena, placed 87th, because most of its answers came wrapped in a key the task did not ask for. The extended abstract is in the conference's abstract booklet, compiled by G. Bachmann, R. Gautschy and J. L. Losada Palenzuela (DaSCH, 2026); the slides are on Zenodo: https://doi.org/10.5281/zenodo.23761273.

SGG workshop, 23 October. At the workshop «Künstliche Intelligenz, historische Materialien und Methoden» of the SGG section «Grundlagenerschliessung und Digitalisierung», the talk Der RISE Humanities Data Benchmark: KI-Modelle an echten Projektdaten messen started from a single personnel card and the two errors a model made on it, and reported two results from the twelve datasets: no model leads on all of them (ten different models hold a best score, counting ties), and similar results can come at very different prices. On Fraktur Adverts, gpt-4.1-mini scores 96.8 at $0.016 per run and gpt-6-astra 96.7 at $0.98, a factor of 63. The slides are on Zenodo: https://doi.org/10.5281/zenodo.23596168.

Both talks ended with the same offer: researchers who want to know which model suits their own material are welcome to get in touch with RISE.

Release v0.7.1

Mon, 26 Oct 2026 00:00:00 +0000

Added

21 new models with 14 benchmark test configurations each (T1900-T2221), company_lists on prompt_v2.txt only: claude-haiku-5-5 (Anthropic), gpt-6.1-sol (OpenAI), mistral-large-4, mistral-small-2603, ministral-3b-2512 (Mistral), command-a-plus-05-2026 (Cohere), and via OpenRouter minimax/minimax-m3.1-flash, nvidia/nemotron-3.5-super-vl, meta/muse-spark-open, meta/muse-wonder, stepfun/step-5-preview, qwen/qwen3.8-max-prime, xiaomi/mimo-v2.6-pro, xiaomi/mimo-v2.6-flash, inclusionai/ling-3.0-flash-vl, perceptron/perceptron-mk1.5, elorian/glimpse, fireworks/ember-1, bytedance-seed/seed-2-1-turbo, nex-agi/nex-n2.5-mini, nex-agi/nex-n2.5-pro; pricing.json metadata bumped to version 1.58.
Results for 2026-10-24: 175 tests in T1900-T2185 (7,972 requests) for 15 of the new models, mistral-large-4 with one; inclusionai/ling-3.0-flash-vl not run.
Results for 2026-10-25: 88 tests in T2068-T2218 (3,956 requests) for qwen/qwen3.8-max-prime, xiaomi/mimo-v2.6-pro, xiaomi/mimo-v2.6-flash, fireworks/ember-1, bytedance-seed/seed-2-1-turbo, nex-agi/nex-n2.5-mini, nex-agi/nex-n2.5-pro.
Results for 2026-10-26: T2219-T2221 (3 tests, 70 requests) for nex-agi/nex-n2.5-pro.

Fixed

Mistral requests at temperature 0 send top_p=1, which mistral-large-4 requires; its chunked thinking/text answers are parsed from the text chunks. command-a-plus-05-2026 output cap clamped to its 64,000 ceiling.

Release v0.7.0

Mon, 19 Oct 2026 00:00:00 +0000

Added

New model gemini-4-argon (GenAI), with 15 benchmark test configurations (T1885-T1899); pricing.json metadata bumped to version 1.56.
Results for 2026-10-14: T1885-T1899 (15 tests, 649 requests) for gemini-4-argon; magazine_pages on prompt_explicit-json.txt.
Results for 2026-10-17: T0203-T0208, T0285, T0317, T0363, T0364, T0460, T0589, T0785, T1229, T1282 (15 tests, 649 requests) for gemini-2.5-flash-lite, a regression check of the 2.0.0 benchmarks.
BenchmarkCard READMEs (#89) for all 12 benchmarks, each with benchmark_card.json and a generated How to Cite; scripts/benchmark_cards.py and an integrity test; scripts/sync_benchmark_pages.py points a card's site page at the benchmark_card widget and the result page at benchmark_results, the test-run search at test_runs and the leaderboard at dashboard at deploy; benchmark_export.json runs carry cost_usd, prompt_file and legacy; one card template replaces both README templates; benchmark versions are MAJOR.MINOR.PATCH. README.md introduces the cards and benchmark versions and takes its descriptions from them; its benchmark guide moved into CONTRIBUTING.md, which adds scorer tests. Reviewed all cards and both fixture READMEs against prompts, schemas, data and scorers; corrected score interpretation, failure handling, personnel interpretation rules and provenance claims, added run guidance and reproducible diagnostics, and recorded per-benchmark findings and open issues in benchmarks/DOCUMENTATION_REVIEW.md.
Site data layer: scripts/ndr_export/generate_site_data.py (last step of generate_all) collects everything the website's widgets read into collected_results/site/ (cards with web-sized example images, export, per-benchmark scoring, test configurations, CHANGELOG, CITATION, CONTRIBUTING, contributors, and news/about from the new benchmark-frontend repository); the deploy serves it at /benchmarks/media/site/ and the widgets no longer read GitHub. News, Contribute and About pages built from it. CONTRIBUTING.md gains a step-by-step guide to creating and submitting a benchmark; contributors.json gains photo, website, core.
Benchmark cards state two licences: the source material's and CC BY 4.0 for all data the benchmark creates; Audience may name humanities readers (an extension of the EvalEval vocabulary).
Scorer tests for duty_rosters, general_meeting_minutes and magazine_pages; an integrity test requires them for every benchmark.
create_benchmark.py scaffolds a working fuzzy scorer that scores unparseable answers as misses, a scorer test file and meta.json added, and no .gitkeep in input folders; the card validator accepts Not specified for Audience.

Removed

collected_results/compare_detail/ is no longer committed; the deploy regenerates it on the VM and copies it into the comparison widget's media directory, which now reads it from there instead of raw.githubusercontent. export_dataset still hashes it into source_manifest.jsonl as if committed.

Fixed

business_letters 2.0.0: unparseable responses scored as misses, stored runs re-scored; ground truth is JSON only; its CSV exports, per-run regeneration and dead scorer code removed, and NaN in persons.json replaced by null, scores unchanged.
company_lists 3.0.0: page_id and entry_id no longer scored; unparseable responses scored as misses; all stored runs re-scored with scripts/rewrite_scores.py; prompt_v2.txt for new tests.
duty_rosters 3.0.0: ground truth of rows SaA410, S402 (staff ids) and 109 (29 April) corrected; profession no longer scored (not in the row images); unparseable responses scored as misses; stored runs re-scored.
personnel_cards 2.0.0: unparseable responses scored as misses; stored runs re-scored. 3.0.0: a card without rows answered without rows scores 1, not 0; stored runs re-scored.
blacklist_cards 2.0.0: Montaner card's date and address corrected; empty information is null in the ground truth and [] counts as empty; unparseable responses scored as misses; dead code removed; stored runs re-scored.
magazine_pages: five runs from before unparseable pages counted as misses re-scored under that rule.
medieval_manuscripts 2.0.0: folio [69r] numbered 69; unparseable responses scored as misses; stored runs re-scored.
fraktur_adverts 3.0.0: unparseable responses scored as misses; image_5 adverts 4 (raw database format) and 5 (numbered 3) corrected; ground truth in one format; scorer simplified, behavior unchanged; stored runs re-scored.
general_meeting_minutes 3.0.0: unparseable responses scored as misses; degree signs in the ground truth corrected (U+00B0 → °); unused context/ removed; the four prompt-variant tests retired as legacy; stored runs re-scored.
bibliographic_data 2.0.0: ground truth checked against the scans and conformed to the response schema (transcription errors, six missing entries, nested entries, relation as id); entries aligned in page order instead of by position; wrapped responses unwrapped; unparseable responses scored as misses; prompt_v2.txt for new tests (printed page number, id as the printed number); scorer tests; stored runs re-scored.
book_advert_xml 2.0.0: scored by edits (F1 of the model's edits against the reference's, f1_macro) instead of fuzzy similarity, which unchanged input won at 99; ground truth curated to input text plus tag repairs, all 50 well-formed, unscored keys dropped; unparseable responses scored as misses; prompt_v2.txt for new tests; stored runs re-scored.
library_cards 3.0.0: year of 00575005 corrected; unparseable responses scored as misses; ground-truth files hold only the card record; the two files without an image and the annotation tools removed; stored runs re-scored.

Benchmark cards, a new result overview and a faster test-run search

Fri, 16 Oct 2026 00:00:00 +0000

Benchmark descriptions now live where the benchmarks live. Each benchmark's README in the repository is becoming a BenchmarkCard, the documentation standard proposed by Sokol et al. (NeurIPS 2025) and used by Evaluation Cards. A card covers the same ground for every benchmark: purpose and intended users, data and annotation, methodology and metrics, and ethical and legal considerations. It also carries a machine-readable benchmark_card.json. The website shows the card directly, so the page and the repository can no longer drift apart. Personnel Cards is the first; the other benchmarks follow.

Every benchmark also has a new result overview: a leaderboard of the best run per model with its cost, the cheapest model within two points of the top, score against cost with the cost frontier, and scores over time. Test runs are now searched and filtered directly in the browser, by benchmark, provider, model or test id. Each run card shows its score, rank and cost, and opens to the full scoring, the configuration, the prompt and the response-by-response comparison with the ground truth. The leaderboard has been restyled to match, and every provider now has one fixed colour across all pages.

Benchmark versions now follow semantic versioning: a new major version means scores before and after are not comparable.

Release v0.6.1

Tue, 29 Sep 2026 00:00:00 +0000

Added

6 new models: claude-opus-5-5, claude-sonnet-5-5 (Anthropic), gpt-6-sol, gpt-6-luna (OpenAI), grok-4.7 (x-ai) and Qwen3.8-Flash-Next-FP8 (sciCORE), with 90 benchmark test configurations (T1795-T1884); pricing.json metadata bumped to version 1.55.
Results for 2026-09-27: T1810-T1869 (60 tests, 2,596 requests) for claude-opus-5-5, gpt-6-sol, gpt-6-luna and grok-4.7; library_cards for the gpt-6 models (T1830, T1845) on prompt_year-string.txt. Results for 2026-09-29: T1870-T1884, T1675-T1689 and T1810-T1824 (45 tests, 1,947 requests) for claude-sonnet-5-5, claude-fable-5-1 and claude-opus-5-5 on generic-llm-api-client 0.5.2.

Changed

sciCORE moved to https://ai-api.scicore.unibas.ch/v1, set per test via base_url; qwen35-397b-a17b-fp8 marked legacy, no longer served.

Fixed

benchmark_base.py: gpt-6 sends no temperature, which it rejects, on the Responses path too; a run stops on Anthropic's credit-balance 400.
library_cards: new prompt_year-string.txt prompt variant asking for year as a string; prompt.txt asks for an integer the schema forbids, which loops constrained decoding on whitespace.
generic-llm-api-client 0.5.1 and 0.5.2: reasoning, failed and discarded attempts are recorded and priced, and priced on the requested model, a billed total keeps the provider's own input and output split, and a schema echoed back is a failed parse; resumed runs keep them and never reprice a stored or billed cost. backfill_costs.py applies the same split to stored results. Claude models that reject forced tool use get the schema as an optional tool instead of plain text.

Release v0.6.0

Wed, 23 Sep 2026 00:00:00 +0000

Added

scripts/backfill_costs.py: records reasoning tokens and fills missing input/output cost on stored results, priced at the rate in force on each run's date. Dry run by default.
generate_vars.py: total_reasoning_tokens and total_reasoning_cost_usd in vars.json; the energy estimate charges reasoning at the output rate.
README.md: Transkribus User Conference 2026 poster under Publications.
2 new models replacing retired ones: deepseek-flash (DeepSeek's V4.1-Flash, with vision) and qwen/qwen3.8-max-0902 (OpenRouter), with 30 benchmark test configurations (T1765-T1794, 15 each).
Pricing data for 2026-09-16: 85 priced models across 12 providers and for 2026-09-18; 2026-06-11 and 2026-06-23 buckets for cohere/command-r and command-r-plus, recovered from archived snapshots; pricing.json metadata bumped to version 1.52.
Results for the single-run and never-run backlog: 1,046 runs and 42,623 requests on 2026-09-16 (531 runs), 2026-09-17 (346) and 2026-09-18 (169).

Fixed

results/: 17,600 requests now record reasoning_tokens and reasoning_cost_usd; genai and x-ai bill reasoning outside output_tokens, so it was never costed.
results/: 17,398 requests that recorded tokens but no cost are now priced.
results/: 7,922 Hugging Face requests repriced.
pricing.json: cohere/command-r-08-2024 at 2026-03-02 corrected to $0.15/$0.60; it held command-r7b's price.
library_cards: Publication.year is optional and accepts a numeric year, coerced to string.
company_lists: Entry.location is optional.
business_letters: null and blank entries are dropped from the Metadata lists; the keys stay required.
magazine_pages: new prompt_explicit-json.txt prompt variant stating the JSON format and the box key. An unparseable answer scores as a miss instead of being skipped, via the new score_unparseable_as_miss opt-in.
publicai Apertus output ceiling lowered to 4,096.
benchmark_base.py: output ceilings for claude-haiku-4-5, claude-opus-4-5 and claude-sonnet-4-5 (64,000).
benchmark_base.py: sends max_completion_tokens on OpenAI chat endpoints, which gpt-5 and newer require in place of max_tokens.
update_pricing.py: reads every text block of the parser's reply, so a leading thinking block no longer raises KeyError: 'text' and blanks a provider's models.
update_pricing.py: Alibaba scraped from model-studio/model-pricing.
generate_compare_detail.py: a run whose detail comes back identical keeps its recorded rescored date, so a re-run no longer rewrites every detail file with no change in content.

Changed

Published totals over 3,417 runs: cost $940.84 → $1,796.42 and the energy estimate up four fifths, from the reasoning backfill, the cost fill and the runs since 2026-09-14.
collected_results/ is no longer committed, apart from compare_detail/; the deploy rebuilds it on the VM before it reads anything.
DEFAULT_MAX_OUTPUT_TOKENS lowered from 32,768 to 16,384.
deepseek-flash keeps 32,768 via the new MODEL_LONG_OUTPUT.
Output ceiling of 16,384 for the seven qwen/qwen3.5-* and both meta/muse-spark models, which binds when a benchmark raises its own cap.
legacy_test=true for six models their providers no longer serve: deepseek-chat, deepseek-reasoner, deepseek-v4-flash, deepseek-v4-flash-vision-exp (DeepSeek now serves deepseek-flash and deepseek-v4-pro only), qwen/qwen3.8-max (OpenRouter now lists qwen/qwen3.8-max-0902) and claude-opus-4-1-20250805 (Anthropic).
README.md: task-oriented sections, with the results dashboard up front, metrics rewritten around what each benchmark records, and reference material in collapsed blocks.
CITATION.cff: expanded keywords for Zenodo discovery.

Removed

3 scoring.json publishing 0.0 with no request files behind them (2026-09-16/T1480, 2026-09-17/T1322, 2026-09-17/T1335).
4 magazine_pages tests configured on text-only cohere models (T0798-T0801: command-r-08-2024, command-r-plus-08-2024, command-r7b-12-2024, command-a-03-2025); none could ever produce a result.

Release v0.5.5

Mon, 14 Sep 2026 00:00:00 +0000

Added

12 new models: gemini-3.7-flash and gemini-3.8-flash (GenAI), grok-4.6 (xAI), meta/muse-spark-1.2, meta/muse-spark-1.3, z-ai/glm-5v-turbo, z-ai/glm-5.3-flash, qwen/qwen3.8-flash and qwen/qwen3.8-27b (OpenRouter), gpt-6-astra (OpenAI; added to the benchmark_base.py hotfix list that forces temperature=1), claude-fable-5-1 (Anthropic; added to the benchmark_base.py no-temperature hotfix list) and deepseek-v4-flash-vision-exp (DeepSeek; first DeepSeek model with image input), with 180 benchmark test configurations (T1585-T1764, 15 each).
Pricing data for 2026-08-18 (gemini-3.7-flash, grok-4.6, meta/muse-spark-1.2, z-ai/glm-5v-turbo), 2026-09-02 (gemini-3.8-flash), 2026-09-03 (gpt-6-astra) and 2026-09-08 (claude-fable-5-1, meta/muse-spark-1.3, z-ai/glm-5.3-flash, qwen/qwen3.8-flash, qwen/qwen3.8-27b, deepseek-v4-flash-vision-exp); pricing.json metadata bumped to version 1.47.
Tests on 2026-08-18: T1585-T1644 (60 tests) for the 4 new models across all benchmarks — gemini-3.7-flash (T1585-T1599) (GenAI), grok-4.6 (T1600-T1614) (x-ai), meta/muse-spark-1.2 (T1615-T1629) and z-ai/glm-5v-turbo (T1630-T1644) (OpenRouter)
Tests on 2026-09-03: T1645-T1659 (15 tests) for gemini-3.8-flash (GenAI) across all benchmarks
Tests on 2026-09-04: T1660-T1674 (15 tests) for gpt-6-astra (OpenAI) across all benchmarks
Tests on 2026-09-08: T1690-T1739 and T1750-T1764 (65 tests) — meta/muse-spark-1.3 (T1690-T1704), z-ai/glm-5.3-flash (T1705-T1719), qwen/qwen3.8-flash (T1720-T1734) (OpenRouter), deepseek-v4-flash-vision-exp (T1750-T1764) (DeepSeek) across all benchmarks, and qwen/qwen3.8-27b on 5 of 15 (T1735-T1739) (OpenRouter)
Tests on 2026-09-09: T1740-T1749 (10 tests) completing qwen/qwen3.8-27b (OpenRouter).
tests/integrity/test_client_capability_integrity.py: fails when the installed ai_client would silently drop images from a DeepSeek vision model.
FatalProviderError: a 402, 401 or 403 now aborts the run instead of being retried per object for every remaining test.
field_scores in score_request_answer for the eight benchmarks that lacked it (bibliographic_data, blacklist_cards, book_advert_xml, business_letters, fraktur_adverts, general_meeting_minutes, magazine_pages, medieval_manuscripts), so all twelve now record what the scorer compared as {response, ground_truth, score}, with score set to null where a scorer counts true/false positives instead of assigning a similarity. Scores are unchanged: all 77,866 stored inputs re-score identically. Documented on Benchmark.score_request_answer and enforced by the new tests/integrity/test_field_scores_integrity.py, which also rejects a score that will not serialise, since save_answer writes it verbatim into the stored answer.
scripts/offline_scoring.py and scripts/rescore.py: re-run a benchmark's own scorer over stored results with no model call, to prove a scorer change leaves its numbers untouched. Each run's rules are applied, because personnel_cards selects which fields it scores from them.
scripts/ndr_export/generate_compare_detail.py, a new generate_all.py step (now 8), writing collected_results/compare_detail/<date>/<test_id>.json (1,792 files, ~13 MB in git). Only inputs whose stored answer has no field_scores are re-scored, so detail recorded at run time is always preferred and results/ is never rewritten; each file records its provenance and whether re-scoring still reproduces the stored score. compare_index.json marks such runs with detail: true.
timing on every test run in test_runs_export.json: total model time, mean and slowest per input, and inputs timed.

Fixed

general_meeting_minutes: score_benchmark no longer raises ZeroDivisionError when every request fails, matching the empty-score guard in the other benchmarks.
business_letters: a response that does not match the expected schema now scores as a complete failure instead of crashing the run.
business_letters: scoring no longer injects document_number into the saved response payload.
scripts/generate_test_report.py: a field with no per-field similarity no longer raises when the field-by-field table is rendered; score was formatted with :.2f, which fails on the null a counting scorer reports.
general_meeting_minutes: two print() calls in the scoring path now log instead of writing to stdout on every run.
benchmark_export.json: used_providers and used_models are now sorted instead of reordering on every run.

Changed

Requests now send a 32k max_tokens cap (duty_rosters 96k, general_meeting_minutes 40k, per-test override via the rules column); previously the provider default applied.
requirements.txt: relaxed all pins to the versions actually installed in the development environment; dropped the unused matplotlib and the redundant dotenv shim, and added pytest, which CI previously installed separately.

Test run cards now show every scored input, compared against the ground truth

Fri, 11 Sep 2026 00:00:00 +0000

Until now a test run card showed a single stored model response. A run scores many inputs, so that one response was an arbitrary sample: a Magazine Pages run scores 46 pages, and the card showed one of them.

Each card now carries the whole run. Open Response vs ground truth and you get every scored input, with the source image or text beside what the model returned and what the benchmark expects. The important part is what the comparison is made of. It is not a difference calculated in your browser; it is what the benchmark's own scorer recorded while it was scoring, field by field, with the similarity it assigned each one. That distinction is crucial because the benchmarks do not agree on what a match is:

Magazine Pages pairs predicted and expected bounding boxes by how far they overlap, so a box that is close but not identical still counts.
Medieval Manuscripts aligns folios by position, because the ground truth is keyed by folio reference while the model returns a list.
Business Letters compares normalised name sets and counts true and false positives rather than assigning a similarity at all, so those rows show the tally and leave the similarity column empty.

An independent comparison would contradict the score printed beside it. Showing the scorer's own judgement means the detail and the score always agree. Runs from before scorers recorded this detail have been re-scored so the view works for the full history, and those are labelled as such. Ground truths are occasionally corrected after a run, and where re-scoring no longer reproduces a run's stored score the card says so and shows both sets of numbers, rather than quietly presenting today's truth as the one the run was measured against.

Two smaller changes came with it. A new Speed panel reports the run's total model time, the mean per input and the slowest input, replacing a figure that was really just one input's response time. And the prompt now sits with the comparison it produced, rather than in a panel of its own.

Release v0.5.4

Mon, 17 Aug 2026 00:00:00 +0000

Added

9 new models: grok-4.5 (xAI), qwen/qwen3.8-max (OpenRouter), and seven on Hugging Face, namely swiss-ai/Apertus-v1.5-70B, swiss-ai/Apertus-v1.5-8B, Qwen/Qwen3-VL-235B-A22B-Instruct, MiniMaxAI/MiniMax-M3, thinkingmachines/Inkling, thinkingmachines/Inkling-Small and meta-models/Muse-Glimmer-30B
New huggingface provider: routes to third-party inference providers over the OpenAI-compatible Hugging Face router API.
Hugging Face models are recorded with their downstream inference provider pinned in the model name (<repo_id>:<provider>, e.g. MiniMaxAI/MiniMax-M3:deepinfra), which the router honors per request.
135 new benchmark test configurations (T1450-T1584) for the 9 new models across all benchmarks (15 tests each): grok-4.5 (T1450-T1464), qwen/qwen3.8-max (T1465-T1479), swiss-ai/Apertus-v1.5-70B:publicai (T1480-T1494), swiss-ai/Apertus-v1.5-8B:publicai (T1495-T1509), Qwen/Qwen3-VL-235B-A22B-Instruct:deepinfra (T1510-T1524), MiniMaxAI/MiniMax-M3:deepinfra (T1525-T1539), thinkingmachines/Inkling-Small:deepinfra (T1540-T1554), thinkingmachines/Inkling:together (T1555-T1569), meta-models/Muse-Glimmer-30B:together (T1570-T1584)
Tests on 2026-06-30: 4 book_advert_xml tests (Cohere) — command-r-08-2024 (T0483), command-r-plus-08-2024 (T0484), command-r7b-12-2024 (T0485) and command-a-03-2025 (T0486).
Tests on 2026-08-04: T1450-T1479 (30 tests) — grok-4.5 (T1450-T1464) (x-ai) and qwen/qwen3.8-max (T1465-T1479) (OpenRouter)
Tests on 2026-08-14: 21 tests (Hugging Face) with swiss-ai/Apertus-v1.5-70B:publicai (T1480-T1486) and swiss-ai/Apertus-v1.5-8B:publicai (T1495-T1508).
Tests on 2026-08-15: 66 tests (Hugging Face) with Qwen/Qwen3-VL-235B-A22B-Instruct:deepinfra (T1510-T1524), MiniMaxAI/MiniMax-M3:deepinfra (T1525-T1539), thinkingmachines/Inkling-Small:deepinfra (T1540-T1545), thinkingmachines/Inkling:together (T1555-T1560, T1562-T1569), meta-models/Muse-Glimmer-30B:together (T1570-T1575), swiss-ai/Apertus-v1.5-70B:publicai (T1485-T1494) and swiss-ai/Apertus-v1.5-8B:publicai (T1509).
Tests on 2026-08-16: 19 tests (Hugging Face) with thinkingmachines/Inkling-Small:deepinfra (T1546-T1554), thinkingmachines/Inkling:together (T1561) and meta-models/Muse-Glimmer-30B:together (T1576-T1584).
Pricing data for 2026-08-04 (grok-4.5, qwen/qwen3.8-max), 2026-08-10 (the two swiss-ai/Apertus-* models on Hugging Face) and 2026-08-15 (five new Hugging Face models, each priced from its pinned route); pricing.json metadata bumped to version 1.42.
scripts/data/model_aliases.json (v1.4): huggingface aliases for all seven pinned model names, mapping each to the model id its route writes into result files, plus an alibaba alias qwen3.5-plus-2026-02-15 → qwen3.5-plus.
scripts/ndr_export/copy_static_data.py: copies pricing.json and model_aliases.json verbatim into collected_results/ as a sixth generate_all.py step, making the export self-contained. The viz widget joins benchmark_export.json (model names from benchmarks_tests.csv) to test_runs_export.json (model names as result files report them) on provider + model, so without the alias map it silently lost cost and speed figures for every aliased model.
CITATION.cff: references block listing the two peer-reviewed Journal of Open Humanities Data articles describing the benchmark (10.5334/johd.470 and 10.5334/johd.481), so citation tools surface the papers alongside the software.

Fixed

Added the missing duty_rosters entry to the Available Benchmarks table in README.md
README.md consistency pass.
Tests with a dataclass whose completion yields no parseable JSON (truncated, empty or non-JSON) are now logged and skipped instead of being passed to score_request_answer with a None payload.

Release v0.5.3

Tue, 04 Aug 2026 00:00:00 +0000

Added

2 new models: claude-sonnet-5, claude-fable-5 (Anthropic); both added to the benchmark_base.py hotfix list that sends no temperature (the models reject the deprecated parameter)
7 new models: gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna (OpenAI; added to the benchmark_base.py hotfix list that forces temperature=1, as these models reject any other value), moonshotai/kimi-k3 (OpenRouter), claude-opus-5 (Anthropic; added to the benchmark_base.py no-temperature hotfix list), gemini-3.6-flash, gemini-3.5-flash-lite (GenAI)
26 new benchmark test configurations (T1177-T1202) for Anthropic claude-sonnet-5 (T1177-T1189) and claude-fable-5 (T1190-T1202) across all benchmarks
105 new benchmark test configurations (T1345-T1449) for the 7 new models across all benchmarks (15 tests each): gpt-5.6-sol (T1345-T1359), gpt-5.6-terra (T1360-T1374), gpt-5.6-luna (T1375-T1389), moonshotai/kimi-k3 (T1390-T1404), claude-opus-5 (T1405-T1419), gemini-3.6-flash (T1420-T1434), gemini-3.5-flash-lite (T1435-T1449)
Pricing data for 2026-07-01 (claude-sonnet-5), 2026-07-02 (claude-fable-5), 2026-07-03 (meta-llama/llama-4-maverick), and 2026-07-25 (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, moonshotai/kimi-k3, claude-opus-5, gemini-3.6-flash, gemini-3.5-flash-lite; pricing.json metadata bumped to version 1.39)
scripts/data/model_aliases.json (v1.1): added openrouter alias meta-llama/llama-4-maverick → meta-llama/llama-4-maverick-17b-128e-instruct
Tests on 2026-07-01: T1177-T1189 (13 tests) for Anthropic claude-sonnet-5 across all benchmarks
Tests on 2026-07-02: T1190-T1202 (13 tests) for Anthropic claude-fable-5 across all benchmarks
Tests on 2026-07-03: 57 general_meeting_minutes tests — T1209-T1218 (Anthropic), T1228-T1234 (GenAI), T1235-T1243 (Mistral), T1244-T1254 (OpenAI), T1255-T1271 (OpenRouter), T1273 (sciCORE), and T1274-T1275 (x-ai)
Tests on 2026-07-24: 37 duty_rosters tests — T1276-T1279, T1308-T1309, T1332, T1336, T1343-T1344 (Anthropic), T1281-T1284, T1311, T1334, T1342 (GenAI), T1287-T1289, T1337 (Mistral), T1293-T1303, T1310, T1312, T1333 (OpenAI), and T1313, T1338 (x-ai)
Tests on 2026-07-25/26: T1345-T1449 (105 tests) for the 7 new models across all benchmarks — gpt-5.6-sol (T1345-T1359), gpt-5.6-terra (T1360-T1374), gpt-5.6-luna (T1375-T1389) (OpenAI), moonshotai/kimi-k3 (T1390-T1404) (OpenRouter), claude-opus-5 (T1405-T1419) (Anthropic), gemini-3.6-flash (T1420-T1434), gemini-3.5-flash-lite (T1435-T1449) (GenAI), and T1341 (sciCORE)
65 new benchmark test configurations (T1203-T1275, excluding T1219 and T1221-T1227) extending general_meeting_minutes to all non-legacy multimodal models that lacked it, each using the standard prompt.txt (empirically the best-scoring of the five prompt variants) with the MinutesPage dataclass; OpenAI at temperature 1.0, all other providers at 0.0.
Unit test suite under tests/: test_scoring_helper.py and test_benchmark_base_helpers.py cover pure scoring/helper logic; tests/integrity/ adds data-integrity guards (integrity marker) over benchmarks_tests.csv and pricing.json. Run logic-only with pytest -m "not integrity".
Pricing-coverage guard: every non-legacy model in benchmarks_tests.csv must have an explicit exact-name entry in pricing.json with a usable numeric price (local providers, which have no per-token price, are exempt from the numeric requirement but still require an entry).
Model-addition guards: every non-legacy row's prompt_file must exist under benchmarks/<name>/prompts/; every non-legacy model must resolve to a source URL in update_pricing.py (scicore/local exempt) and appear in the README models table (local exempt).
Ground-truth guards: every ground_truths/*.json is valid JSON, and every logical object (image/text basename, collapsed with the benchmark's own basename pattern) has a matching ground-truth file (harness scaffold benchmarks excluded).
Per-benchmark scoring tests (run offline via __new__-bypassed instances, no AI client): field-level F1 (company_lists, library_cards, personnel_cards incl. rules-gated field selection), CER (medieval_manuscripts, fraktur_adverts), the simple averagers (bibliographic_data, blacklist_cards, book_advert_xml), and business_letters macro/micro F1 aggregation. The two complex scorers also have end-to-end tests with hand-verified expected scores: business_letters score_request_answer (person matching, inferred-person exclusion) and skip_object signature rules; fraktur_adverts compare_ads (section/number matching incl. the image_4 default-section path).
"Running the Tests" section in CONTRIBUTING.md documenting the pytest suite, the integrity marker, and CI enforcement
duty_rosters benchmark for extracting structured schedule data (shift icons, half-day splits, alternate-unit assignments, and staff metadata) from Swiss nursing staff duty rosters, scored with field-level F1 (micro/macro): 69 test configurations (T1276-T1344) covering all non-legacy multimodal models across all providers using the Schedule dataclass.
scripts/data/pricing_editor.html: browser-based editor for filling in missing pricing.json fields and deleting entries; and update_pricing.py --exclude-providers to skip providers in a run.

Fixed

update_pricing.py scraping: dropped br from Accept-Encoding (undecoded brotli garbled OpenAI/Cohere pages), routed Alibaba through the headless browser, tolerate prose around the LLM's JSON, switched Anthropic to its API docs pricing page, stopped partial scrapes from silently dropping models (null template instead), and preserved existing archived source_urls on --force re-runs.
blacklist_cards and bibliographic_data score_benchmark raised ZeroDivisionError on an empty score list (an all-failed run), crashing the run; both now return {"fuzzy": 0.0} like book_advert_xml.
pytest.ini section header corrected from [tool:pytest] to [pytest], so addopts, testpaths, and custom markers are actually applied (the [tool:pytest] form is only valid in setup.cfg/tox.ini).
general_meeting_minutes: Entry.signature_present had a trailing comma (bool = False,) making its default the tuple (False,) instead of the boolean False; removed unreachable return scores dead code in score_request_answer.
NDR export provider maps updated to the actual provider keys: PROVIDER_COLORS in generate_visualizations.py (previously only covered openai/anthropic and the nonexistent google/meta/huggingface, rendering most providers gray) replaced with a CVD-validated 11-provider palette plus a shared provider_display_name() helper, and provider_names in generate_multi_select_data.py gained genai ("Google/Gemini") and contour_local ("OpenCV (local)") instead of showing "Genai"/"Contour_local".
NDR export: benchmark folders without a meta.json are no longer exported or counted (get_benchmarks() skips them), fixing inflated dataset counts in vars.json and empty entries in benchmark_export.json, benchmarks.csv, and the heatmap/radar charts.

Changed

Moved dissemination materials (posters, talks, submissions; absorbing the former poster/ folder and benchmarks/duty_rosters/slides/) into a separate private repository, benchmark-dissemnination, with one subfolder per output, each with a README.md and meta.json.
Marked openrouter/x-ai/grok-4 as legacy_test=true (14 tests: T0265-T0270, T0304, T0336, T0401-T0402, T0481, T0620, T0793, T1272).
Marked 5 provider-deprecated Mistral models as legacy_test=true (75 tests): magistral-medium-2509, magistral-small-2509, mistral-medium-2505, mistral-medium-2508, mistral-small-2506.
general_meeting_minutes: disabled shared context (use_shared_context was True but no context files were configured, so the harness forced single-threaded runs and sent a context prompt with nothing attached); removed the empty get_shared_context_files/get_shared_context_prompt overrides and updated the benchmark README to match. Signatures are still scored as ordinary fields.

Release v0.5.2

Mon, 29 Jun 2026 00:00:00 +0000

Added

9 new models: gemini-3.5-flash, gemini-3.1-flash-lite (GenAI), qwen/qwen3.7-plus, meta-llama/llama-4-scout, stepfun/step-3.7-flash (OpenRouter), claude-opus-4-8 (Anthropic), mistral-medium-3.5 (Mistral), grok-4.3 (x-ai), qwen35-397b-a17b-fp8 (sciCORE)
91 new benchmark test configurations (T1050-T1140) for GenAI gemini-3.5-flash (T1050-T1062), OpenRouter qwen/qwen3.7-plus (T1063-T1075), Anthropic claude-opus-4-8 (T1076-T1088), Mistral mistral-medium-3.5 (T1089-T1101), x-ai grok-4.3 (T1102-T1114), OpenRouter meta-llama/llama-4-scout (T1115-T1127), and OpenRouter stepfun/step-3.7-flash (T1128-T1140) across all benchmarks
10 new benchmark test configurations (T1141-T1150) for Cohere command-a-vision-07-2025, completing its coverage of all benchmarks (bibliographic_data, business_letters, fraktur_adverts, library_cards, medieval_manuscripts, blacklist_cards, company_lists)
13 new benchmark test configurations (T1151-T1163) for sciCORE qwen35-397b-a17b-fp8 across all benchmarks
13 new benchmark test configurations (T1164-T1176) for GenAI gemini-3.1-flash-lite across all benchmarks
Tests on 2026-05-22: T1050-T1062 (13 tests) for gemini-3.5-flash across all benchmarks
Tests on 2026-06-04: T1063-T1101 (39 tests) for qwen/qwen3.7-plus (T1063-T1075), claude-opus-4-8 (T1076-T1088), and mistral-medium-3.5 (T1089-T1101) across all benchmarks; 117 re-run tests for pixtral-large-2411 (T0023, T0035, T0060-T0061, T0095, T0159, T0294, T0326, T0381-T0382, T0469, T0604, T0771), mistral-medium-2508 (T0169, T0171, T0173, T0175, T0177, T0179, T0295, T0327, T0383-T0384, T0470, T0602, T0781), mistral-medium-2505 (T0170, T0172, T0174, T0176, T0178, T0180, T0296, T0328, T0385-T0386, T0471, T0601, T0782), magistral-medium-2509 (T0425-T0434, T0474, T0595, T0795), mistral-small-2506 (T0435-T0444, T0475, T0603, T0796), mistral-large-2512 (T0532-T0542, T0600, T0807), ministral-14b-2512 (T0543-T0553, T0597, T0808), ministral-8b-2512 (T0554-T0564, T0598, T0809), and magistral-small-2509 (T0565-T0575, T0596, T0810) across all benchmarks
Tests on 2026-06-05: T1115-T1127 (13 tests) for meta-llama/llama-4-scout and T1128-T1140 (13 tests) for stepfun/step-3.7-flash across all benchmarks
Tests on 2026-06-08: T1102-T1114 (13 tests) for grok-4.3 across all benchmarks
Tests on 2026-06-22: T1151-T1163 (13 tests) for sciCORE qwen35-397b-a17b-fp8 across all benchmarks
Tests on 2026-06-29: T1164-T1176 (13 tests) for GenAI gemini-3.1-flash-lite across all benchmarks
Pricing data for 2026-05-22, 2026-06-04, 2026-06-05, and 2026-06-29 (gemini-3.1-flash-lite)
scripts/data/model_aliases.json for tracking model name aliases between benchmarks_tests.csv and result files, organised by provider

Changed

inject_costs.py now resolves model names via scripts/data/model_aliases.json, so tests whose result files use alias model names get correct cost calculations
Marked 8 provider-retired models as legacy_test=true: claude-opus-4-20250514, claude-sonnet-4-20250514 (Anthropic), gemini-2.0-flash, gemini-2.0-flash-lite, gemini-2.5-flash-lite-preview-09-2025, gemini-3.1-flash-lite-preview (GenAI), mistral-large-2411, pixtral-large-2411 (Mistral)
Marked all 16 sciCORE tests as legacy_test=true (GLM-4.5V-FP8 and qwen3-235b-fp8): T0237-T0242, T0299, T0331, T0391-T0392, T0476, T0482, T0621-T0622, T0790, T0797

New "Which Model for Me?" tab: a guided model recommender

Wed, 17 Jun 2026 00:00:00 +0000

The benchmark dashboard now includes a "Which Model for Me?" tab that turns our benchmark results into a concrete recommendation for your own project, so you don't have to read every score table yourself.

Instead of asking you to know which benchmark matches your material, it asks about your data and task. You describe what you're working with (document type, handwriting vs. print, language, the task such as transcription, extraction or layout analysis, and century), and the tool maps your answers to the closest benchmarks in the suite. You can fine-tune the matched benchmarks, or switch to expert mode and pick them directly.

Three further inputs shape the result:

Scale: from a one-off run to a 100k+ document pipeline. The larger your scale, the more score the tool is willing to trade for lower cost or faster throughput.
Optimize for: cost or time. With run data loaded, recommendations are ranked by price-per-run or seconds-per-run within a score band, rather than by raw score alone.
Data sensitivity: a simple choice between non-sensitive and sensitive data. Non-sensitive data may use cloud-hosted models; sensitive data restricts the recommendation to on-premise models (e.g., hosted by sciCORE) that keep your documents on site. Always follow your own institution's data-classification directive.

The output is a single recommended model with its per-benchmark scores, cost and speed where available, and a clear privacy note, plus a short list of alternatives. Each score is a normalized 0–100 value that puts every benchmark's native metric (F1, character error rate, and so on) on a common scale, so results from very different tasks stay comparable. Scores always reflect the latest non-legacy run for each model, so recommendations track the current generation rather than retired configurations.

As always, the underlying data is open and the methodology is documented in the RISE Humanities Data Benchmark repository.

Release v0.5.1

Wed, 20 May 2026 00:00:00 +0000

Added

6 new models: qwen3.5-plus, qwen3.5-35b-a3b, qwen3.5-27b, qwen3.5-122b-a10b, qwen3.5-397b-a17b, qwen3.5-flash-2026-02-23 (Alibaba); 10 new models: qwen/qwen3.6-plus, qwen/qwen3.5-122b-a10b, qwen/qwen3.5-27b, qwen/qwen3.5-35b-a3b, qwen/qwen3.5-397b-a17b, qwen/qwen3.5-plus-02-15, qwen/qwen3.5-flash-02-23, qwen/qwen3.5-9b, google/gemma-4-26b-a4b-it, google/gemma-4-31b-it (OpenRouter); 1 new model: claude-opus-4-7 (Anthropic); 2 new models: deepseek-v4-flash, deepseek-v4-pro (DeepSeek); 1 new model: gpt-5.5-2026-04-23 (OpenAI)
78 new benchmark test configurations (T0814-T0891) for Alibaba Qwen 3.5 models across all benchmarks; 130 new benchmark test configurations (T0892-T1021) for OpenRouter Qwen and Gemma models across all benchmarks; 13 new benchmark test configurations (T1022-T1034) for Anthropic claude-opus-4-7 across all benchmarks; 2 new benchmark test configurations (T1035-T1036) for DeepSeek deepseek-v4-flash and deepseek-v4-pro on the book_advert_xml benchmark (text-only, as these models do not support images); 13 new benchmark test configurations (T1037-T1048, T1054) for OpenAI gpt-5.5-2026-04-23 across all benchmarks
Alibaba provider support with direct API integration and pricing scraper
Test results on 2026-03-25 for all 65 Alibaba Qwen 3.5 tests (T0827-T0891) and 2 DeepSeek book_advert_xml tests (T0752, T0764)
Tests on 2026-04-21: T0892-T1034 (142 tests)
Tests on 2026-04-22: T0983-T0994 (12 tests)
Pricing data for 2026-04-21, 2026-04-24, and 2026-04-27

Changed

Deduplicated grok-4.20-0309-reasoning test configurations (removed duplicate T0730-T0741 and corresponding results)

Two papers on the RISE Humanities Data Benchmark in the Journal of Open Humanities Data

Fri, 27 Mar 2026 00:00:00 +0000

The Journal of Open Humanities Data (JOHD) published two papers by Maximilian Hindermann, Sorin Marti, Lea Katharina Kasper and Arno Bosse on the RISE Humanities Data Benchmark. Which large language models perform best on humanities research tasks, and how can we systematically compare their capabilities?

The data paper "The RISE Humanities Data Benchmark: A Framework for Evaluating Large Language Models for Humanities Tasks" presents a framework for assessing the performance of large language models on humanities-related tasks. The benchmark suite (available on GitHub) includes text and image datasets, prompts, ground truths, and evaluation scripts and addresses tasks essential to digital humanities work including document analysis, transcription, and metadata extraction from historical materials.

The discussion paper "From Experiments to Epistemic Practice: The RISE Humanities Data Benchmark" traces how the suite emerged from RISE's consulting practice and reflects on the methodological challenges of applying benchmarking to humanities contexts. It argues that ground truth in humanities benchmarking is not a matter of objective correctness but of explicit, scholar-defined interpretive choices, and that benchmarking should therefore be understood as an epistemic practice rather than a neutral measurement.

Both papers contribute to the JOHD special collection "Benchmarking in Digital Humanities", which aims to establish benchmarking as common practice in the humanities. The framework promotes evidence-based decision making on which models to use for specific tasks and provides quantifiable comparisons between different LLMs, via an interactive dashboard.

Researchers interested in using the benchmark with their own materials are welcome to get in touch. In their roles at RISE, Maximilian Hindermann, Sorin Marti and Arno Bosse advise researchers on the use of computational methods and large language models for humanities research projects and are happy to discuss how the framework can be applied to new data and research contexts.

Citations

Hindermann, M., Marti, S., Kasper, L. K., & Bosse, A. (2026). The RISE Humanities Data Benchmark: A Framework for Evaluating Large Language Models for Humanities Tasks. Journal of Open Humanities Data, 12(1), 24. https://doi.org/10.5334/johd.481
Hindermann, M., Kasper, L. K., Marti, S., & Bosse, A. (2026). From Experiments to Epistemic Practice: The RISE Humanities Data Benchmark. Journal of Open Humanities Data, 12(1), 38. https://doi.org/10.5334/johd.470

Source: rise.unibas.ch

Release v0.5.0

Tue, 24 Mar 2026 00:00:00 +0000

Added

17 new models: gpt-5.2, gpt-5.3-codex, gpt-5.4-2026-03-05 (OpenAI), gemini-3-flash-preview, gemini-3.1-flash-lite-preview, gemini-3.1-pro-preview (GenAI), claude-opus-4-6, claude-sonnet-4-6, claude-opus-4-5-20251101, claude-haiku-4-5-20251001 (Anthropic), mistral-large-2512, ministral-14b-2512, ministral-8b-2512, magistral-small-2509 (Mistral), grok-4.20-0309-reasoning (x-ai), deepseek-chat, deepseek-reasoner (DeepSeek)
170 new benchmark test configurations (T0488-T0709) across all benchmarks for new models
47 new benchmark test configurations (T0576-T0622) for personnel_cards benchmark across all non-legacy models
32 new benchmark test configurations (T0710-T0741) for magazine_pages benchmark and x-ai/grok-4.20-0309-reasoning across all benchmarks
26 new benchmark test configurations (T0742-T0767) for deepseek-chat and deepseek-reasoner across all benchmarks
46 new benchmark test configurations (T0768-T0813) for magazine_pages benchmark across all remaining non-legacy models
update_pricing.py: semi-automated pricing updater that scrapes provider websites, extracts prices via LLM, archives source URLs on Wayback Machine in parallel, and updates pricing.json
Pricing data for 2026-03-02
Pricing data for 2026-01-23
Tests on 2026-01-24 to 2026-01-26: all 484 non-legacy tests across 7 providers (Anthropic, Cohere, GenAI, Mistral, OpenAI, OpenRouter, sciCORE)
Tests on 2026-02-09: T0576-T0620 (33 tests)
Tests on 2026-02-10: T0109-T0615 (17 tests)
Tests on 2026-02-17: T0061-T0534 (23 tests)
Tests on 2026-02-18: T0535-T0538 (2 tests)
Tests on 2026-03-04: T0023-T0245 (3 tests)
Tests on 2026-03-12: T0504-T0627 (6 tests)
Tests on 2026-03-16: T0628-T0692 (65 tests)
Tests on 2026-03-21: T0710-T0717 (8 tests)
Tests on 2026-03-23: T0060-T0813 (85 tests)
personnel_cards benchmark for extracting structured data from 20th century personnel card tables
magazine_pages benchmark for extracting structured data from historical magazine pages
x-ai provider support with direct API integration
DeepSeek provider support with direct API integration
Local model support via contour_local provider (e.g. opencv-contour); tokens are estimated and all downstream scoring/saving/aggregating remains unchanged
Published paper citation in README: Hindermann et al. (2026) in Journal of Open Humanities Data (DOI: 10.5334/johd.481)

Changed

inject_costs.py: added --date mode to process all tests for a specific date (defaults to results/ only, use --test-runs to target test_runs/)
Marked all gemini-3-pro-preview, claude-3-7-sonnet-20250219, claude-3-opus-20240229, and pixtral-12b benchmark tests as legacy
Updated generate_date_report.py: outputs to reports/ folder, filename format {date}_report, added warnings system with 5 checks (ZERO_COST, ALL_NA, ZERO_SCORE, ZERO_ITEMS, ZERO_DURATION), generates {date}_report_warnings.md for manual review, added progress reporting, added --no-warnings flag, added date range support with --start-date and --end-date arguments
OpenAI models now support selectable API endpoints

Fixed

Converted library_cards images from PNG to JPEG format (263 files had .jpg extension but were actually PNG)
Converted fraktur_adverts images from PNG to JPEG format (3 files had .jpg extension but were actually PNG)
Missing name variants in business_letters

Release v0.4.1

Sun, 18 Jan 2026 00:00:00 +0000

Added

Repeat adhoc test functionality
Docstrings to fraktur_adverts codebase

Fixed

Typos in READMEs

Removed

ntokens from fraktur_adverts ground truth

Release v0.4.0

Tue, 09 Dec 2025 00:00:00 +0000

Added

4 new models: gpt-5.1 (OpenAI), gemini-3-pro-preview (GenAI), magistral-medium-2509 (Mistral), mistral-small-2506 (Mistral)
42 new benchmark test configurations (T0403-T0444) across all benchmarks for new models
Pricing data for 2025-11-24 with updated model prices and source URLs
Cohere provider support with 5 models: command-r-08-2024, command-r-plus-08-2024, command-r7b-12-2024, command-a-03-2025, command-a-vision-07-2025
book_advert_xml benchmark for correcting malformed XML from 18th century book advertisements
43 new benchmark test configurations (T0445-T0487) for book_advert_xml across all providers
Pricing data for Cohere models (2025-12-09)

Changed

All requests are now handled by https://pypi.org/project/generic-llm-api-client/
Suite name is now "RISE Humanities Data Benchmark"
Remap "latest" suffix to actual model used

Removed

All tests with claude-3-5-sonnet-20241022 (now legacy)
All renders and related docs (now handled by dedicated frontend)

The RISE Humanities Data Benchmark platform is now online

Mon, 08 Dec 2025 00:00:00 +0000

We are pleased to announce the public launch of the RISE Humanities Data Benchmark, a new research infrastructure designed to support systematic, transparent, and reproducible evaluation of large language models on humanities-oriented tasks.

The platform brings together a growing suite of benchmark datasets derived from historical documents, bibliographic sources, index cards, and other forms of cultural heritage material. Each benchmark includes detailed contextual information, clearly defined ground truth, and openly documented evaluation procedures. Together, these resources provide an evidence-based foundation for assessing how well current models handle the complex, data-rich challenges commonly encountered in humanities research.

In addition to offering full access to all benchmark descriptions, the platform provides:

A comprehensive leaderboard, enabling cross-model and cross-provider comparison
Interactive visualisations of accuracy, consistency, efficiency, and cost
Searchable test-run archives with detailed outputs, scoring metrics, and configurations
Guidance and tooling for creating and contributing new benchmarks
Documentation to support transparency and reuse

With this launch, our goal is to create a shared, extensible environment that facilitates rigorous evaluation, fosters methodological discussion, and encourages community contributions. We invite researchers, practitioners, and institutions to explore the platform, reuse our benchmark setups, and experiment with their own datasets.

The RISE Humanities Data Benchmark will continue to evolve as new benchmarks, evaluation methods, and model providers are added. We welcome your feedback and look forward to collaborative development in the coming months.

Release v0.3.1

Wed, 29 Oct 2025 00:00:00 +0000

Added

OpenRouter provider support with fallback for models not supporting structured outputs
sciCORE provider support (LiteLLM-based OpenAI-compatible API)
blacklist benchmark on extracting structured company information from historical index cards
company_lists benchmark on extracting structured company information (company name and location) from historical trade indexes
medieval_manuscripts benchmark for 15th century page segmentation and handwritten text extraction with CER and fuzzy matching
6 new models: qwen/qwen3-vl-8b-thinking, qwen/qwen3-vl-30b-a3b-instruct, qwen/qwen3-vl-8b-instruct, meta-llama/llama-4-maverick, x-ai/grok-4, GLM-4.5V-FP8
170 new benchmark test configurations (T0233-T0402) across blacklist, medieval_manuscripts, and company_lists benchmarks
Tests on 2025-10-03: T0164
Tests on 2025-10-17: T0233-T0234, T0237-T0252
Tests on 2025-10-20: T0253-T0270
Tests on 2025-10-24: T0271-T0336
Tests on 2025-10-28: T0336-T0402
Update pricing data to 2025-10-28

Fixed

metadata_extraction scoring now correctly counts failed requests as complete failures (0 TP, all FN) instead of excluding them
Fuzzy score matching now handles type mismatches between strings and integers (e.g., "1965" vs 1965 for year fields)
Pricing lookup now searches through all available dates to find provider/model pricing instead of only checking the most recent date

Release v0.3.0

Fri, 03 Oct 2025 00:00:00 +0000

Added

Cost tracking system with pricing database and automatic cost calculation
- Token usage extraction for all providers (OpenAI, GenAI, Anthropic, Mistral)
- Automatic cost calculation based on token usage and date-based pricing data
- Cost summary in benchmark scoring files with detailed token and cost breakdowns
- tables showing test execution costs
- Cost per Point metric in global leaderboard showing normalized cost efficiency ($/performance point)
7 new models: pixtral-12b, mistral-large-latest, gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-flash-lite-preview-09-2025, gemini-2.5-flash-preview-09-2025, claude-sonnet-4-5-20250929
50 new benchmark test configurations (T0181-T0230) for new models across all benchmark variants
Test Time tracking and metrics
- Test Time (s) column in benchmark tables showing total execution time per test
- Time per Point column in benchmark tables showing time efficiency (seconds/point per item)
- Time/Point metric in global leaderboard showing normalized time efficiency
- Multi-level normalized calculation: per-test (average time per item / score), per-benchmark (average of test ratios), global (average of benchmark ratios)
- Analogous to cost calculation methodology for consistency
Structured outputs for Google Gen AI, Anthropic (native tool calling), and Mistral models
Automatic regeneration of empty or invalid JSON result files
Tests on 2025-09-24: T0099, T0107, T0117, T0120, T0125, T0130, T0132, T0134, T0145, T0151, T0162
Tests on 2025-09-25: T0023, T0035, T0095, T0159
Tests on 2025-09-26: T0169-T0180
Tests on 2025-09-30: T0001-T0010, T0012-T0013, T0017-T0018, T0020, T0023-T0025, T0027, T0031, T0033, T0035-T0036, T0038-T0039, T0042-T0045, T0052-T0053, T0056-T0057, T0060-T0063, T0066-T0079, T0082-T0086, T0090, T0092-T0095, T0098-T0107, T0108-T0133, T0160, T0193-T0200, T0230
Tests on 2025-10-01: T0129-T0148, T0151-T0152, T0155, T0159-T0161, T0165-T0192, T0201-T0229
Tests on 2025-10-02: T0161-T0162, T0164

Changed

All provider response objects now converted to JSON-serializable format before storage
Schema default values automatically removed for GenAI API compatibility

Fixed

Empty log files (FileHandler now properly configured)
JSON serialization errors for OpenAI, GenAI, Anthropic, and Mistral response objects
Pydantic validation errors now handled gracefully with fallback to raw tool input
Letter dataclass normalization for list-formatted fields (letter_title, send_date)
bibliographic_data attributions
Pydantic dataclass models

Release v0.2.2

Fri, 19 Sep 2025 00:00:00 +0000

Added

bibliographic_data README.md sections
CONTRIBUTING.md
CONTRIBUTORS.md

Release v0.2.1

Wed, 10 Sep 2025 00:00:00 +0000

Added

Radar chart top 10 models
Zettelkatalog benchmark
Tests on 2025-09-02: T0066, T0143-T0148, T0151-T0152, T0155, T0159-T0162, T0164-T0168

Release v0.2.0

Sun, 31 Aug 2025 00:00:00 +0000

Added

Global model performance leaderboard to docs
Tests on 2025-08-27: T0106, T0136-T0141

Fixed

Broken link patterns in docs

Changed

Standardize test-IDs to 4-digit zero-padded format (T0001)

Release v0.1.0

Mon, 25 Aug 2025 00:00:00 +0000

Added

Changelog