A test run is a single execution of a benchmark test using a defined model configuration.
Each run represents how a particular large language model (LLM) — such as GPT-4, Claude-3, or Gemini — performed on a given task at a specific time, with specific settings.
A test run includes:
Together, test runs make it possible to compare models, providers, and configurations across benchmarks in a transparent and reproducible way.
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [18, 19], 'language': ['de'], 'layout': ['prose'], 'script-style': ['fraktur'], 'task': ['transcription']}
| Provider | openai |
| Model | gpt-4.1-mini |
| Temperature | 0.0 |
| Dataclass | Document |
| Normalized Score | 43.20 % |
| Test time | unknown seconds |
## IDENTITY AND PURPOSE
You are an OCR and information extraction system trained to process historical newspaper pages printed in 18th-century German using Fraktur type. The pages contain mostly classified advertisements. Your task is to identify and extract each advertisement *exactly as printed*, including historical spellings, typographic errors, punctuation, and formatting.
## INSTRUCTIONS
- Extract **all advertisements** from the input image, one after the other, following the sequence on the page.
- Maintain the **original spelling**, capitalization, and any **typos or non-standard forms**.
- Follow these transcription rules:
- the long s (ſ) is transcribed as "s"
- "/" is transcribed as ","
- Use the masthead of the newspaper only to extract the date, ignore other content.
- The layout is typically **two-column**; extract ads from both columns, including the ad number.
- Return the result as a **JSON object** in the specified format and **nothing else** (no explanations, summaries, or additional text).
- For each advertisement, include:
- `"date"`: the publication date of the page in ISO 8061 format (YYYY-MM-DD)
- `"tags_section"`: the heading under which the advertisement appears
- `"text"`: the full advertisement text
## EXAMPLE OUTPUT
{
"advertisements": [
{
"date": "1731-01-02",
"tags_section": "Es werden zum Verkauff offeriert",
"text": "5. Ein kleines, jedoch listiges Lehrbuch der Zauberkunst, lange im Gebrauche des jungen Bartolomeus Simpson."
},
{
"date": "1731-01-02",
"tags_section": "Es werden zum Verkauff offeriert",
"text": "6. Ein rarer, mit Edelsteinen besetzter Saxophon-Kasten, aus dem Besitze der Jungfer Lisa Simpson."
},
{
"date": "1731-01-02",
"tags_section": "Es werden zu Entleihen begehrt",
"text": "7. Ein gar prachtvoller, jedoch etwas zerlesener Band mit Rezepten von Margaretha Simpsonin."
}
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| 0.45 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: n/a IT + n/a OT = n/a TT | Cost: n/a$ + n/a$ = n/a$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [18, 19], 'language': ['de'], 'layout': ['prose'], 'script-style': ['fraktur'], 'task': ['transcription']}
| Provider | openai |
| Model | gpt-4o-mini |
| Temperature | 0.0 |
| Dataclass | Document |
| Normalized Score | 0.70 % |
| Test time | unknown seconds |
## IDENTITY AND PURPOSE
You are an OCR and information extraction system trained to process historical newspaper pages printed in 18th-century German using Fraktur type. The pages contain mostly classified advertisements. Your task is to identify and extract each advertisement *exactly as printed*, including historical spellings, typographic errors, punctuation, and formatting.
## INSTRUCTIONS
- Extract **all advertisements** from the input image, one after the other, following the sequence on the page.
- Maintain the **original spelling**, capitalization, and any **typos or non-standard forms**.
- Follow these transcription rules:
- the long s (ſ) is transcribed as "s"
- "/" is transcribed as ","
- Use the masthead of the newspaper only to extract the date, ignore other content.
- The layout is typically **two-column**; extract ads from both columns, including the ad number.
- Return the result as a **JSON object** in the specified format and **nothing else** (no explanations, summaries, or additional text).
- For each advertisement, include:
- `"date"`: the publication date of the page in ISO 8061 format (YYYY-MM-DD)
- `"tags_section"`: the heading under which the advertisement appears
- `"text"`: the full advertisement text
## EXAMPLE OUTPUT
{
"advertisements": [
{
"date": "1731-01-02",
"tags_section": "Es werden zum Verkauff offeriert",
"text": "5. Ein kleines, jedoch listiges Lehrbuch der Zauberkunst, lange im Gebrauche des jungen Bartolomeus Simpson."
},
{
"date": "1731-01-02",
"tags_section": "Es werden zum Verkauff offeriert",
"text": "6. Ein rarer, mit Edelsteinen besetzter Saxophon-Kasten, aus dem Besitze der Jungfer Lisa Simpson."
},
{
"date": "1731-01-02",
"tags_section": "Es werden zu Entleihen begehrt",
"text": "7. Ein gar prachtvoller, jedoch etwas zerlesener Band mit Rezepten von Margaretha Simpsonin."
}
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| 0.01 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: n/a IT + n/a OT = n/a TT | Cost: n/a$ + n/a$ = n/a$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [18, 19], 'language': ['de'], 'layout': ['prose'], 'script-style': ['fraktur'], 'task': ['transcription']}
| Provider | genai |
| Model | gemini-2.0-flash-lite |
| Temperature | 0.0 |
| Dataclass | Document |
| Normalized Score | 77.00 % |
| Test time | unknown seconds |
## IDENTITY AND PURPOSE
You are an OCR and information extraction system trained to process historical newspaper pages printed in 18th-century German using Fraktur type. The pages contain mostly classified advertisements. Your task is to identify and extract each advertisement *exactly as printed*, including historical spellings, typographic errors, punctuation, and formatting.
## INSTRUCTIONS
- Extract **all advertisements** from the input image, one after the other, following the sequence on the page.
- Maintain the **original spelling**, capitalization, and any **typos or non-standard forms**.
- Follow these transcription rules:
- the long s (ſ) is transcribed as "s"
- "/" is transcribed as ","
- Use the masthead of the newspaper only to extract the date, ignore other content.
- The layout is typically **two-column**; extract ads from both columns, including the ad number.
- Return the result as a **JSON object** in the specified format and **nothing else** (no explanations, summaries, or additional text).
- For each advertisement, include:
- `"date"`: the publication date of the page in ISO 8061 format (YYYY-MM-DD)
- `"tags_section"`: the heading under which the advertisement appears
- `"text"`: the full advertisement text
## EXAMPLE OUTPUT
{
"advertisements": [
{
"date": "1731-01-02",
"tags_section": "Es werden zum Verkauff offeriert",
"text": "5. Ein kleines, jedoch listiges Lehrbuch der Zauberkunst, lange im Gebrauche des jungen Bartolomeus Simpson."
},
{
"date": "1731-01-02",
"tags_section": "Es werden zum Verkauff offeriert",
"text": "6. Ein rarer, mit Edelsteinen besetzter Saxophon-Kasten, aus dem Besitze der Jungfer Lisa Simpson."
},
{
"date": "1731-01-02",
"tags_section": "Es werden zu Entleihen begehrt",
"text": "7. Ein gar prachtvoller, jedoch etwas zerlesener Band mit Rezepten von Margaretha Simpsonin."
}
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| 0.80 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: n/a IT + n/a OT = n/a TT | Cost: n/a$ + n/a$ = n/a$ |