A test run is a single execution of a benchmark test using a defined model configuration.
Each run represents how a particular large language model (LLM) — such as GPT-4, Claude-3, or Gemini — performed on a given task at a specific time, with specific settings.
A test run includes:
Together, test runs make it possible to compare models, providers, and configurations across benchmarks in a transparent and reproducible way.
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | x-ai |
| Model | grok-4.3 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 55.40 % |
| Test time | unknown seconds |
The image you are presented with stems from a digitized book containing lists of companies.
Your task is to extract structured information about each company listed on the page.
About the source:
- The image stems from a trade index of the British Swiss Chamber of Commerce.
- The image can show an alphabetical or a thematic list of companies.
- The companies are mostly located in Switzerland and the UK.
- The image stems from a trade index between 1925 and 1958.
- Most pages have one column but some years have two columns.
- The source itself is in English and German but the company names can be in English, German, French or Italian.
About the entries:
- Each entry describes a single company or person.
- Alphabetical entries have filling dots between the company name and the page number. Dots and page numbers are not part of the data and should be ignored.
- Alphabetical entries seldom to never have locations.
- Thematic entries often have locations.
- Thematic entries are listed under headings that describe the type of business.
- Some thematic headings are only references to other headings, e.g. "X, s. Y".
About the output:
- Answer in valid JSON. The JSON should be an array of objects with the following fields:
- The page ID is given as {page_id}.
- Do not add country information, if it is not directly written with the location.
{
"entry_id": "A unique identifier for the entry, e.g. '{page_id}-1'",
"company_name": "The name of the company or person",
"location": "The location of the company, e.g. 'Zurich' or 'London, UK'. If no location is given, set to null."
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.57 | 0.55 | 0.58 | 0.56 | 15 | 558 | 408 | 434 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 45.5K IT + 11.2K OT = 56.6K TT | Cost: 0.057$ + 0.028$ = 0.085$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | x-ai |
| Model | grok-4.3 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 41.47 % |
| Test time | unknown seconds |
- Answer in valid JSON.
- The page ID is given as {page_id}.
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.43 | 0.41 | 0.45 | 0.42 | 15 | 413 | 514 | 579 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 40.5K IT + 10.1K OT = 50.6K TT | Cost: 0.051$ + 0.025$ = 0.076$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | openrouter |
| Model | stepfun/step-3.7-flash-20260528 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 37.87 % |
| Test time | unknown seconds |
- Answer in valid JSON.
- The page ID is given as {page_id}.
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.38 | 0.38 | 0.37 | 0.40 | 15 | 398 | 685 | 594 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 18.3K IT + 59.0K OT = 77.4K TT | Cost: 0.004$ + 0.068$ = 0.072$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | openrouter |
| Model | meta-llama/llama-4-scout-17b-16e-instruct |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 31.33 % |
| Test time | unknown seconds |
- Answer in valid JSON.
- The page ID is given as {page_id}.
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.30 | 0.31 | 0.30 | 0.30 | 15 | 299 | 682 | 693 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 35.6K IT + 14.9K OT = 50.5K TT | Cost: 0.003$ + 0.004$ = 0.007$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | openrouter |
| Model | stepfun/step-3.7-flash-20260528 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 49.80 % |
| Test time | unknown seconds |
The image you are presented with stems from a digitized book containing lists of companies.
Your task is to extract structured information about each company listed on the page.
About the source:
- The image stems from a trade index of the British Swiss Chamber of Commerce.
- The image can show an alphabetical or a thematic list of companies.
- The companies are mostly located in Switzerland and the UK.
- The image stems from a trade index between 1925 and 1958.
- Most pages have one column but some years have two columns.
- The source itself is in English and German but the company names can be in English, German, French or Italian.
About the entries:
- Each entry describes a single company or person.
- Alphabetical entries have filling dots between the company name and the page number. Dots and page numbers are not part of the data and should be ignored.
- Alphabetical entries seldom to never have locations.
- Thematic entries often have locations.
- Thematic entries are listed under headings that describe the type of business.
- Some thematic headings are only references to other headings, e.g. "X, s. Y".
About the output:
- Answer in valid JSON. The JSON should be an array of objects with the following fields:
- The page ID is given as {page_id}.
- Do not add country information, if it is not directly written with the location.
{
"entry_id": "A unique identifier for the entry, e.g. '{page_id}-1'",
"company_name": "The name of the company or person",
"location": "The location of the company, e.g. 'Zurich' or 'London, UK'. If no location is given, set to null."
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.51 | 0.50 | 0.51 | 0.52 | 15 | 517 | 505 | 475 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 23.2K IT + 109.5K OT = 132.7K TT | Cost: 0.005$ + 0.126$ = 0.131$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | openrouter |
| Model | meta-llama/llama-4-scout-17b-16e-instruct |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 44.60 % |
| Test time | unknown seconds |
The image you are presented with stems from a digitized book containing lists of companies.
Your task is to extract structured information about each company listed on the page.
About the source:
- The image stems from a trade index of the British Swiss Chamber of Commerce.
- The image can show an alphabetical or a thematic list of companies.
- The companies are mostly located in Switzerland and the UK.
- The image stems from a trade index between 1925 and 1958.
- Most pages have one column but some years have two columns.
- The source itself is in English and German but the company names can be in English, German, French or Italian.
About the entries:
- Each entry describes a single company or person.
- Alphabetical entries have filling dots between the company name and the page number. Dots and page numbers are not part of the data and should be ignored.
- Alphabetical entries seldom to never have locations.
- Thematic entries often have locations.
- Thematic entries are listed under headings that describe the type of business.
- Some thematic headings are only references to other headings, e.g. "X, s. Y".
About the output:
- Answer in valid JSON. The JSON should be an array of objects with the following fields:
- The page ID is given as {page_id}.
- Do not add country information, if it is not directly written with the location.
{
"entry_id": "A unique identifier for the entry, e.g. '{page_id}-1'",
"company_name": "The name of the company or person",
"location": "The location of the company, e.g. 'Zurich' or 'London, UK'. If no location is given, set to null."
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.46 | 0.45 | 0.47 | 0.44 | 15 | 439 | 488 | 553 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 40.8K IT + 12.5K OT = 53.3K TT | Cost: 0.003$ + 0.004$ = 0.007$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | mistral |
| Model | mistral-medium-2505 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 36.33 % |
| Test time | unknown seconds |
- Answer in valid JSON.
- The page ID is given as {page_id}.
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.41 | 0.36 | 0.48 | 0.36 | 15 | 361 | 391 | 631 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 27.8K IT + 9.9K OT = 37.7K TT | Cost: 0.011$ + 0.020$ = 0.031$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | anthropic |
| Model | claude-opus-4-8 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 41.60 % |
| Test time | unknown seconds |
- Answer in valid JSON.
- The page ID is given as {page_id}.
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.42 | 0.42 | 0.41 | 0.43 | 15 | 424 | 603 | 568 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 65.8K IT + 18.4K OT = 84.3K TT | Cost: 0.329$ + 0.460$ = 0.790$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | openrouter |
| Model | qwen/qwen3.7-plus-20260602 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 52.47 % |
| Test time | unknown seconds |
The image you are presented with stems from a digitized book containing lists of companies.
Your task is to extract structured information about each company listed on the page.
About the source:
- The image stems from a trade index of the British Swiss Chamber of Commerce.
- The image can show an alphabetical or a thematic list of companies.
- The companies are mostly located in Switzerland and the UK.
- The image stems from a trade index between 1925 and 1958.
- Most pages have one column but some years have two columns.
- The source itself is in English and German but the company names can be in English, German, French or Italian.
About the entries:
- Each entry describes a single company or person.
- Alphabetical entries have filling dots between the company name and the page number. Dots and page numbers are not part of the data and should be ignored.
- Alphabetical entries seldom to never have locations.
- Thematic entries often have locations.
- Thematic entries are listed under headings that describe the type of business.
- Some thematic headings are only references to other headings, e.g. "X, s. Y".
About the output:
- Answer in valid JSON. The JSON should be an array of objects with the following fields:
- The page ID is given as {page_id}.
- Do not add country information, if it is not directly written with the location.
{
"entry_id": "A unique identifier for the entry, e.g. '{page_id}-1'",
"company_name": "The name of the company or person",
"location": "The location of the company, e.g. 'Zurich' or 'London, UK'. If no location is given, set to null."
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.51 | 0.52 | 0.51 | 0.51 | 15 | 509 | 481 | 483 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 46.8K IT + 31.7K OT = 78.5K TT | Cost: 0.019$ + 0.051$ = 0.069$ |
{'document-type': ['book-page'], 'writing': ['printed'], 'century': [20], 'language': ['en', 'de'], 'layout': ['list'], 'entry-type': ['company'], 'task': ['information-extraction']}
| Provider | mistral |
| Model | mistral-large-2512 |
| Temperature | 0.5 |
| Dataclass | ListPage |
| Normalized Score | 17.40 % |
| Test time | unknown seconds |
The image you are presented with stems from a digitized book containing lists of companies.
Your task is to extract structured information about each company listed on the page.
About the source:
- The image stems from a trade index of the British Swiss Chamber of Commerce.
- The image can show an alphabetical or a thematic list of companies.
- The companies are mostly located in Switzerland and the UK.
- The image stems from a trade index between 1925 and 1958.
- Most pages have one column but some years have two columns.
- The source itself is in English and German but the company names can be in English, German, French or Italian.
About the entries:
- Each entry describes a single company or person.
- Alphabetical entries have filling dots between the company name and the page number. Dots and page numbers are not part of the data and should be ignored.
- Alphabetical entries seldom to never have locations.
- Thematic entries often have locations.
- Thematic entries are listed under headings that describe the type of business.
- Some thematic headings are only references to other headings, e.g. "X, s. Y".
About the output:
- Answer in valid JSON. The JSON should be an array of objects with the following fields:
- The page ID is given as {page_id}.
- Do not add country information, if it is not directly written with the location.
{
"entry_id": "A unique identifier for the entry, e.g. '{page_id}-1'",
"company_name": "The name of the company or person",
"location": "The location of the company, e.g. 'Zurich' or 'London, UK'. If no location is given, set to null."
]
}
no valid result| Fuzzy Score | F1 micro / macro | Micro precision/recall | Tue/False Positives | |||||
| n/a | 0.23 | 0.17 | 0.44 | 0.16 | 15 | 158 | 204 | 834 |
| Micro Precision | Micro Recall | Instances | TP | FP | FN | |||
| Pricing Date: n/a, n/a. | Tokens: 15.7K IT + 5.4K OT = 21.1K TT | Cost: 0.008$ + 0.008$ = 0.016$ |