Benchmark Results

Bibliographic Data

Evaluates models' ability to extract bibliographic information from historical documents such as publication details, authors, dates, and other metadata from digitized sources.

See Benchmark Description See Test Results Open on Github

• Type(s): book-page
• Benchmark task(s):  information-extraction
• Writing: printed
• Source creation (century): 20
• Source Layout: list
• Language(s): en

Blacklist Cards

Index cards of companies on a British 'black list' in the 1940s. Assesses models' capability to recognize typed and handwritten information from index cards.

See Benchmark Description See Test Results Open on Github

• Type(s): index-card
• Benchmark task(s):  information-extraction
• Writing: typed, handwritten
• Source creation (century): 20
• Source Layout: n/a
• Language(s): de

Book Advert XML files (malformed) from Avisblatt

See Benchmark Description See Test Results Open on Github

• Type(s): newspaper-page
• Benchmark task(s):  data-correction
• Writing: n/a
• Source creation (century): 18
• Source Layout: n/a
• Language(s): en

Business Letters

Tests models on extracting structured metadata from historical correspondence, including person names, organizations, dates, locations, and other contextual information from 20th century Swiss historical letters.

See Benchmark Description See Test Results Open on Github

• Type(s): letter
• Benchmark task(s):  information-extraction
• Writing: typed, handwritten
• Source creation (century): 20
• Source Layout: prose
• Language(s): de

Company Lists

See Benchmark Description See Test Results Open on Github

• Type(s): book-page
• Benchmark task(s):  information-extraction
• Writing: printed
• Source creation (century): 20
• Source Layout: list
• Language(s): en, de

Fraktur Adverts

Assesses models' capability to recognize and transcribe historical German Fraktur script, a Gothic typeface commonly used in German-language documents.

See Benchmark Description See Test Results Open on Github

• Type(s): book-page
• Benchmark task(s):  transcription
• Writing: printed
• Source creation (century): 18, 19
• Source Layout: prose
• Language(s): de

General Meeting Minutes

Extract names, locations, signatures from table-like meeting minutes of Mines de Costano S.A., 1930s - 1960s

See Benchmark Description See Test Results Open on Github

• Type(s): minutes
• Benchmark task(s):  information-extraction
• Writing: typed, handwritten
• Source creation (century): 20
• Source Layout: table
• Language(s): it, fr, de

Library Cards

A comprehensive benchmark focused on catalog card analysis and information extraction from historical library catalog systems. This benchmark evaluates models on structured data extraction from digitized catalog cards, testing their ability to parse complex bibliographic information, author names, dates, and hierarchical catalog structures from historical Swiss library records.

See Benchmark Description See Test Results Open on Github

• Type(s): index-card
• Benchmark task(s):  information-extraction
• Writing: typed, printed, handwritten
• Source creation (century): 20, 19
• Source Layout: index
• Language(s): de, fr, en, la, el, fi, sv, pl

Magazine Pages - Anglo Swiss Trade Review

Examines a model's ability to extract bounding boxes of advertisements from magazine pages.

See Benchmark Description See Test Results Open on Github

• Type(s): newspaper-page
• Benchmark task(s):  document-understanding
• Writing: printed
• Source creation (century): 20
• Source Layout: prose, columns
• Language(s): en

Medieval Manuscripts

Evaluates models on page segmentation and handwritten text extraction from 15th century medieval manuscripts written in late medieval German. Tests the ability to transcribe historical handwriting, identify folio numbers, distinguish main text from marginal additions, and maintain historical spelling and formatting. Performance is measured using fuzzy string matching and Character Error Rate (CER).

See Benchmark Description See Test Results Open on Github

• Type(s): manuscript
• Benchmark task(s):  transcription
• Writing: handwritten
• Source creation (century): 15
• Source Layout: prose
• Language(s): de