Benchmark Results

Fraktur Adverts

Assesses models' capability to recognize and transcribe historical German Fraktur script, a Gothic typeface commonly used in German-language documents.

Dataset Description Result Overview Test Runs

This benchmark has been run 280 times. It uses cer metric.

Tested providers: alibaba, anthropic, cohere, deepseek, genai, huggingface, mistral, openai, openrouter, scicore, x-ai

Tested models: GLM-4.5V-FP8, MiniMaxAI/MiniMax-M3:deepinfra, Qwen/Qwen3-VL-235B-A22B-Instruct:deepinfra, Qwen3.8-Flash-Next-FP8, claude-3-5-sonnet-20241022, claude-3-7-sonnet-20250219, claude-3-opus-20240229, claude-fable-5, claude-fable-5-1, claude-haiku-4-5-20251001, claude-opus-4-1-20250805, claude-opus-4-20250514, claude-opus-4-5-20251101, claude-opus-4-6, claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-opus-5-5, claude-sonnet-4-20250514, claude-sonnet-4-5-20250929, claude-sonnet-4-6, claude-sonnet-5, claude-sonnet-5-5, command-a-vision-07-2025, deepseek-flash, deepseek-v4-flash-vision-exp, gemini-1.5-flash, gemini-1.5-pro, gemini-2.0-flash, gemini-2.0-flash-lite, gemini-2.0-pro-exp-02-05, gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-flash-lite-preview-09-2025, gemini-2.5-flash-preview-04-17, gemini-2.5-flash-preview-09-2025, gemini-2.5-pro, gemini-2.5-pro-exp-03-25, gemini-2.5-pro-preview-05-06, gemini-3-flash-preview, gemini-3-pro-preview, gemini-3.1-flash-lite, gemini-3.1-flash-lite-preview, gemini-3.1-pro-preview, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.6-flash, gemini-3.7-flash, gemini-3.8-flash, gemini-exp-1206, google/gemma-4-26b-a4b-it, google/gemma-4-31b-it, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4.5-preview, gpt-4o, gpt-4o-mini, gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.1-2025-11-13, gpt-5.2-2025-12-11, gpt-5.3-codex, gpt-5.4-2026-03-05, gpt-5.5-2026-04-23, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra, gpt-6-luna, gpt-6-sol, grok-4.20-0309-reasoning, grok-4.3, grok-4.5, grok-4.6, grok-4.7, magistral-medium-2509, magistral-small-2509, meta-llama/llama-4-maverick, meta-llama/llama-4-scout, meta-models/Muse-Glimmer-30B:together, meta/muse-spark-1.2, meta/muse-spark-1.3, ministral-14b-2512, ministral-8b-2512, mistral-large-2411, mistral-large-2512, mistral-medium-2505, mistral-medium-2508, mistral-medium-3.5, mistral-small-2506, moonshotai/kimi-k3, o3, pixtral-12b, pixtral-large-2411, qwen/qwen3-vl-30b-a3b-instruct, qwen/qwen3-vl-8b-instruct, qwen/qwen3-vl-8b-thinking, qwen/qwen3.5-122b-a10b, qwen/qwen3.5-27b, qwen/qwen3.5-35b-a3b, qwen/qwen3.5-397b-a17b, qwen/qwen3.5-9b, qwen/qwen3.5-flash-02-23, qwen/qwen3.5-plus-02-15, qwen/qwen3.6-plus, qwen/qwen3.7-plus, qwen/qwen3.8-27b, qwen/qwen3.8-flash, qwen/qwen3.8-max, qwen/qwen3.8-max-0902, qwen3.5-122b-a10b, qwen3.5-27b, qwen3.5-35b-a3b, qwen3.5-397b-a17b, qwen3.5-flash-2026-02-23, qwen3.5-plus-2026-02-15, qwen35-397b-a17b-fp8, stepfun/step-3.7-flash, swiss-ai/Apertus-v1.5-70B:publicai, swiss-ai/Apertus-v1.5-8B:publicai, thinkingmachines/Inkling-Small:deepinfra, thinkingmachines/Inkling:together, x-ai/grok-4, z-ai/glm-5.3-flash, z-ai/glm-5v-turbo

• Type(s): book-page
• Benchmark task(s):  transcription
• Writing: printed
• Source creation (century): 18, 19
• Source Layout: prose
• Language(s): de