Benchmark Results

Blacklist Cards

Index cards of companies on a British 'black list' in the 1940s. Assesses models' capability to recognize typed and handwritten information from index cards.

Dataset Description Result Overview Test Runs

This benchmark has been run 146 times. It uses fuzzy metric.

Tested providers: huggingface, openrouter, scicore, genai, openai, x-ai, alibaba, mistral, anthropic

Tested models: magistral-medium-2509, gpt-5.6-sol, claude-fable-5, pixtral-large-2411, gpt-5, mistral-small-2506, gemini-3.1-flash-lite-preview, meta/muse-spark-1.2, claude-3-7-sonnet-20250219, o3, pixtral-12b, gemini-3-pro-preview, qwen3.5-27b, claude-opus-4-6, grok-4.5, qwen/qwen3.5-27b, qwen/qwen3.5-9b, qwen35-397b-a17b-fp8, gemini-2.5-flash-lite, gpt-4o, gpt-5.1-2025-11-13, thinkingmachines/Inkling-Small:deepinfra, gemini-3.5-flash-lite, qwen/qwen3.7-plus, google/gemma-4-31b-it, google/gemma-4-26b-a4b-it, claude-sonnet-5, qwen/qwen3-vl-30b-a3b-instruct, qwen3.5-flash-2026-02-23, gpt-4.1-nano, meta-llama/llama-4-maverick, ministral-14b-2512, claude-opus-4-1-20250805, gpt-5.3-codex, qwen/qwen3.5-plus-02-15, qwen/qwen3.5-397b-a17b, mistral-large-2411, qwen/qwen3.5-122b-a10b, gpt-4o-mini, GLM-4.5V-FP8, gpt-5.2-2025-12-11, mistral-medium-2505, thinkingmachines/Inkling:together, claude-3-opus-20240229, qwen/qwen3.5-flash-02-23, Qwen/Qwen3-VL-235B-A22B-Instruct:deepinfra, qwen/qwen3-vl-8b-thinking, claude-opus-4-20250514, claude-opus-4-8, meta-models/Muse-Glimmer-30B:together, claude-haiku-4-5-20251001, claude-3-5-sonnet-20241022, qwen3.5-plus-2026-02-15, qwen3.5-35b-a3b, claude-opus-5, swiss-ai/Apertus-v1.5-70B:publicai, claude-sonnet-4-6, z-ai/glm-5v-turbo, gpt-5.6-terra, claude-sonnet-4-20250514, gpt-5-mini, gemini-3.5-flash, mistral-medium-3.5, grok-4.20-0309-reasoning, qwen3.5-397b-a17b, gpt-5.4-2026-03-05, gpt-4.1, gemini-3-flash-preview, gemini-2.5-pro, ministral-8b-2512, gpt-5.5-2026-04-23, qwen/qwen3.6-plus, gpt-5.6-luna, gpt-4.1-mini, claude-opus-4-5-20251101, mistral-large-2512, gemini-3.1-flash-lite, gpt-5-nano, gemini-2.5-flash-lite-preview-09-2025, gemini-2.0-flash-lite, x-ai/grok-4, gemini-2.0-flash, qwen/qwen3-vl-8b-instruct, gemini-3.1-pro-preview, grok-4.6, gemini-3.6-flash, claude-opus-4-7, grok-4.3, mistral-medium-2508, qwen/qwen3.5-35b-a3b, claude-sonnet-4-5-20250929, gemini-2.5-flash, stepfun/step-3.7-flash, qwen/qwen3.8-max, qwen3.5-122b-a10b, magistral-small-2509, gemini-2.5-flash-preview-09-2025, moonshotai/kimi-k3, meta-llama/llama-4-scout, gemini-3.7-flash, MiniMaxAI/MiniMax-M3:deepinfra, swiss-ai/Apertus-v1.5-8B:publicai

• Type(s): index-card
• Benchmark task(s):  information-extraction
• Writing: typed, handwritten
• Source creation (century): 20
• Source Layout: n/a
• Language(s): de