{"_id": {"$oid": "6a26c67c3572c06bc1455dfc"}, "test_id": "T0945", "benchmark": "business_letters", "date": "2026-04-21", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "openrouter", "model": "qwen/qwen3.5-397b-a17b", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1922-12-16\"],\"letter_title\":[\"null\"],\"sender_persons\":[\"Basler Personenschiffahrtsgesellschaft\"],\"receiver_persons\":[\"Basler Rheinschiffahrt A.-G.\"]}}", "model": "qwen/qwen3.5-397b-a17b-20260216", "provider": "openrouter", "finish_reason": "stop", "usage": {"input_tokens": 3223, "output_tokens": 1744, "total_tokens": 4967, "input_cost_usd": 0.00125697, "output_cost_usd": 0.00408096, "estimated_cost_usd": 0.005337929999999999}, "duration": 55.720293045043945, "timestamp": "2026-04-21T13:56:08.830178", "parsed": {"metadata": {"send_date": ["1922-12-16"], "letter_title": ["null"], "sender_persons": ["Basler Personenschiffahrtsgesellschaft"], "receiver_persons": ["Basler Rheinschiffahrt A.-G."], "document_number": "letter59"}}, "conversation_id": "c6cabfd5-73b4-45e3-b3b3-3e85f522ef84", "score": {"send_date_tp": 0, "send_date_fp": 1, "send_date_fn": 1, "sender_persons_tp": 0, "sender_persons_fp": 1, "sender_persons_fn": 0, "receiver_persons_tp": 0, "receiver_persons_fp": 1, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"gen-1776772513-aE5RNXazfVYmtzAZHQUk\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n    \\\"metadata\\\": {\\n        \\\"send_date\\\": [\\n            \\\"1922-12-16\\\"\\n        ]\\n    ,\\n    \\\"letter_title\\\": [\\n        \\\"null\\\"\\n    ],\\n    \\\"sender_persons\\\": [\\n        \\\"Basler Personenschiffahrtsgesellschaft\\\"\\n    ],\\n    \\\"receiver_persons\\\": [\\n        \\\"Basler Rheinschiffahrt A.-G.\\\"\\n    ]\\n}\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1922-12-16\"],\"letter_title\":[\"null\"],\"sender_persons\":[\"Basler Personenschiffahrtsgesellschaft\"],\"receiver_persons\":[\"Basler Rheinschiffahrt A.-G.\"]}},\"reasoning\":\"The user wants me to extract specific information from a historical letter image and format it as a JSON object.\\nI need to identify:\\n1.  **letter_title**: The title of the letter. Looking at the document, there isn't a specific \\\"Title\\\" like \\\"Re: Contract\\\". It's a formal business letter. Often, if there's no explicit title, I might use the letterhead or just leave it null, but the instructions say \\\"Title of the letter\\\". In business correspondence of this era, the letterhead often serves as the identifier, or there is no title. However, looking closely at the top, it says \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\". This is the organization sending it. There is no specific subject line like \\\"Betreff: ...\\\". I will check if there is a more appropriate title. Since there is no explicit subject line, I will set this to \\\"null\\\" or perhaps use the organization name if it functions as a header, but strictly speaking, it's the sender. Let's look for a subject line. There isn't one. I will set `letter_title` to `null` as there is no specific title given in the text body or header distinct from the sender. Wait, looking at the example, \\\"Petition for Environmental Protection\\\" is a title. This document is a business letter. I will set it to `null` because there is no explicit title.\\n2.  **sender_persons**: Who wrote the letter? The letterhead says \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\". At the bottom, there are faint stamps/signatures. One says \\\"Basler Personenschiffahrtsgesellschaft\\\" again, and below that \\\"Der Pr\u00e4sident\\\" (The President) and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" (The Managing Director). The names are not clearly legible/signed by hand, it looks like a typed form letter or the signatures are too faint to read specific names. However, the entity sending it is the \\\"Basler Personenschiffahrtsgesellschaft\\\". The prompt asks for \\\"Name(s) of the person(s)\\\". Since specific names aren't legible (just titles \\\"Der Pr\u00e4sident\\\", \\\"Der Gesch\u00e4ftsf\u00fchrer\\\"), I should probably list the organization or null. But usually, in these tasks, the organization acting as the sender is acceptable if individual names aren't clear. Let's look really closely at the bottom. It's very faint. It looks like a stamp. I cannot read specific personal names. I will list the organization \\\"Basler Personenschiffahrtsgesellschaft\\\" as the sender entity, or perhaps just null if strict person names are required. The prompt says \\\"sender_persons: Name(s) of the person(s)\\\". If I can't read the names, I might have to put null. However, the letterhead is the sender. Let's assume the organization represents the senders. But wait, the key is `sender_persons`. If I put the company name, is that a person? In historical archiving, often the corporate body is treated as the sender. Let's look at the bottom again. \\\"Der Pr\u00e4sident\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". No names are visible next to them. I will use the organization name as the sender, as is common in business correspondence extraction when individuals aren't named. Or I can list \\\"Basler Personenschiffahrtsgesellschaft\\\" under sender. Let's check the receiver.\\n3.  **send_date**: The date is clearly typed at the top right: \\\"Basel, den 16. Dezember 1922.\\\" So the date is 1922-12-16.\\n4.  **receiver_persons**: The letter is addressed to \\\"An die Basler Rheinschiffahrt A.-G.-\\\". This is a company (Aktiengesellschaft). The address is \\\"Marktgasse 5, Basel\\\". So the receiver is the organization \\\"Basler Rheinschiffahrt A.-G.\\\". Again, the key is `receiver_persons`. I will list the organization name.\\n\\nRefining the extraction:\\n- **letter_title**: There is no subject line. I will return `null`.\\n- **sender_persons**: The letterhead is \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\". The bottom signature block is for \\\"Der Pr\u00e4sident\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" of that company. Since no specific human names are legible, I will use the organization name as the sender entity.\\n- **send_date**: \\\"16. Dezember 1922\\\" -> \\\"1922-12-16\\\".\\n- **receiver_persons**: \\\"An die Basler Rheinschiffahrt A.-G.-\\\". I will extract \\\"Basler Rheinschiffahrt A.-G.\\\".\\n\\nLet's double check the bottom signature area. It is extremely faint. It looks like a stamp of the company name again. I cannot decipher individual names like \\\"Hans Muller\\\". So I will stick with the organization.\\n\\nWait, looking at the instruction: \\\"Name(s) of the person(s)\\\". If it's an organization, is it a person? In the context of entity extraction from letters, usually, the named entity sending/receiving is captured. I will capture the organization names.\\n\\nLet's re-evaluate `letter_title`. Sometimes the letterhead is considered the title if nothing else exists, but usually `null` is safer if there's no \\\"Subject:\\\" line. I'll go with `null`.\\n\\nLet's re-evaluate `sender_persons`. The text \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\" is at the very top. This is the sender.\\nLet's re-evaluate `receiver_persons`. \\\"An die Basler Rheinschiffahrt A.-G.-\\\". This is the receiver.\\n\\nDate format: YYYY-MM-DD.\\n16. Dezember 1922 -> 1922-12-16.\\n\\nFinal check of the text:\\nTop: BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\nDate: Basel, den 16. Dezember 1922.\\nTo: An die Basler Rheinschiffahrt A.-G.-\\nBody: Acknowledges receipt of letter from 15. December. Contract accepted in general assembly. Discussion about frequency numbers for propaganda purposes.\\nSign-off: mit vorz\u00fcglicher Hochachtung (With highest regards).\\nSignatures: Faint stamp \\\"Basler Personenschiffahrtsgesellschaft\\\", \\\"Der Pr\u00e4sident\\\", \\\"Der Gesch\u00e4ftsf\u00fchrer\\\".\\n\\nSo:\\nletter_title: null (No subject line)\\nsender_persons: [\\\"Basler Personenschiffahrtsgesellschaft\\\"]\\nsend_date: [\\\"1922-12-16\\\"]\\nreceiver_persons: [\\\"Basler Rheinschiffahrt A.-G.\\\"]\\n\\nOne detail: The prompt asks for \\\"sender_persons\\\". If I put a company name, is it correct? In many datasets, yes. If I put `null`, it might be more strictly accurate to \\\"persons\\\", but less useful. Given the context of \\\"historical letter\\\", the entity is the important part. I will include the company name.\\n\\nWait, is there a title? \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\" acts as the letterhead. I will leave title as null.\\n\\nLet's verify the receiver name. \\\"Basler Rheinschiffahrt A.-G.-\\\". The \\\"A.-G.-\\\" stands for Aktiengesellschaft. I will include it.\\n\\nLet's verify the sender name. \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\".\\n\\nOkay, constructing the JSON.\\n\",\"reasoning_details\":[{\"type\":\"reasoning.text\",\"text\":\"The user wants me to extract specific information from a historical letter image and format it as a JSON object.\\nI need to identify:\\n1.  **letter_title**: The title of the letter. Looking at the document, there isn't a specific \\\"Title\\\" like \\\"Re: Contract\\\". It's a formal business letter. Often, if there's no explicit title, I might use the letterhead or just leave it null, but the instructions say \\\"Title of the letter\\\". In business correspondence of this era, the letterhead often serves as the identifier, or there is no title. However, looking closely at the top, it says \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\". This is the organization sending it. There is no specific subject line like \\\"Betreff: ...\\\". I will check if there is a more appropriate title. Since there is no explicit subject line, I will set this to \\\"null\\\" or perhaps use the organization name if it functions as a header, but strictly speaking, it's the sender. Let's look for a subject line. There isn't one. I will set `letter_title` to `null` as there is no specific title given in the text body or header distinct from the sender. Wait, looking at the example, \\\"Petition for Environmental Protection\\\" is a title. This document is a business letter. I will set it to `null` because there is no explicit title.\\n2.  **sender_persons**: Who wrote the letter? The letterhead says \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\". At the bottom, there are faint stamps/signatures. One says \\\"Basler Personenschiffahrtsgesellschaft\\\" again, and below that \\\"Der Pr\u00e4sident\\\" (The President) and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" (The Managing Director). The names are not clearly legible/signed by hand, it looks like a typed form letter or the signatures are too faint to read specific names. However, the entity sending it is the \\\"Basler Personenschiffahrtsgesellschaft\\\". The prompt asks for \\\"Name(s) of the person(s)\\\". Since specific names aren't legible (just titles \\\"Der Pr\u00e4sident\\\", \\\"Der Gesch\u00e4ftsf\u00fchrer\\\"), I should probably list the organization or null. But usually, in these tasks, the organization acting as the sender is acceptable if individual names aren't clear. Let's look really closely at the bottom. It's very faint. It looks like a stamp. I cannot read specific personal names. I will list the organization \\\"Basler Personenschiffahrtsgesellschaft\\\" as the sender entity, or perhaps just null if strict person names are required. The prompt says \\\"sender_persons: Name(s) of the person(s)\\\". If I can't read the names, I might have to put null. However, the letterhead is the sender. Let's assume the organization represents the senders. But wait, the key is `sender_persons`. If I put the company name, is that a person? In historical archiving, often the corporate body is treated as the sender. Let's look at the bottom again. \\\"Der Pr\u00e4sident\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". No names are visible next to them. I will use the organization name as the sender, as is common in business correspondence extraction when individuals aren't named. Or I can list \\\"Basler Personenschiffahrtsgesellschaft\\\" under sender. Let's check the receiver.\\n3.  **send_date**: The date is clearly typed at the top right: \\\"Basel, den 16. Dezember 1922.\\\" So the date is 1922-12-16.\\n4.  **receiver_persons**: The letter is addressed to \\\"An die Basler Rheinschiffahrt A.-G.-\\\". This is a company (Aktiengesellschaft). The address is \\\"Marktgasse 5, Basel\\\". So the receiver is the organization \\\"Basler Rheinschiffahrt A.-G.\\\". Again, the key is `receiver_persons`. I will list the organization name.\\n\\nRefining the extraction:\\n- **letter_title**: There is no subject line. I will return `null`.\\n- **sender_persons**: The letterhead is \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\". The bottom signature block is for \\\"Der Pr\u00e4sident\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" of that company. Since no specific human names are legible, I will use the organization name as the sender entity.\\n- **send_date**: \\\"16. Dezember 1922\\\" -> \\\"1922-12-16\\\".\\n- **receiver_persons**: \\\"An die Basler Rheinschiffahrt A.-G.-\\\". I will extract \\\"Basler Rheinschiffahrt A.-G.\\\".\\n\\nLet's double check the bottom signature area. It is extremely faint. It looks like a stamp of the company name again. I cannot decipher individual names like \\\"Hans Muller\\\". So I will stick with the organization.\\n\\nWait, looking at the instruction: \\\"Name(s) of the person(s)\\\". If it's an organization, is it a person? In the context of entity extraction from letters, usually, the named entity sending/receiving is captured. I will capture the organization names.\\n\\nLet's re-evaluate `letter_title`. Sometimes the letterhead is considered the title if nothing else exists, but usually `null` is safer if there's no \\\"Subject:\\\" line. I'll go with `null`.\\n\\nLet's re-evaluate `sender_persons`. The text \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\" is at the very top. This is the sender.\\nLet's re-evaluate `receiver_persons`. \\\"An die Basler Rheinschiffahrt A.-G.-\\\". This is the receiver.\\n\\nDate format: YYYY-MM-DD.\\n16. Dezember 1922 -> 1922-12-16.\\n\\nFinal check of the text:\\nTop: BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\nDate: Basel, den 16. Dezember 1922.\\nTo: An die Basler Rheinschiffahrt A.-G.-\\nBody: Acknowledges receipt of letter from 15. December. Contract accepted in general assembly. Discussion about frequency numbers for propaganda purposes.\\nSign-off: mit vorz\u00fcglicher Hochachtung (With highest regards).\\nSignatures: Faint stamp \\\"Basler Personenschiffahrtsgesellschaft\\\", \\\"Der Pr\u00e4sident\\\", \\\"Der Gesch\u00e4ftsf\u00fchrer\\\".\\n\\nSo:\\nletter_title: null (No subject line)\\nsender_persons: [\\\"Basler Personenschiffahrtsgesellschaft\\\"]\\nsend_date: [\\\"1922-12-16\\\"]\\nreceiver_persons: [\\\"Basler Rheinschiffahrt A.-G.\\\"]\\n\\nOne detail: The prompt asks for \\\"sender_persons\\\". If I put a company name, is it correct? In many datasets, yes. If I put `null`, it might be more strictly accurate to \\\"persons\\\", but less useful. Given the context of \\\"historical letter\\\", the entity is the important part. I will include the company name.\\n\\nWait, is there a title? \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\" acts as the letterhead. I will leave title as null.\\n\\nLet's verify the receiver name. \\\"Basler Rheinschiffahrt A.-G.-\\\". The \\\"A.-G.-\\\" stands for Aktiengesellschaft. I will include it.\\n\\nLet's verify the sender name. \\\"BASLER PERSONENSCHIFFAHRTSGESELLSCHAFT\\\".\\n\\nOkay, constructing the JSON.\\n\",\"format\":\"unknown\",\"index\":0}]},\"native_finish_reason\":\"stop\"}],\"created\":1776772513,\"model\":\"qwen/qwen3.5-397b-a17b-20260216\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":1744,\"prompt_tokens\":3223,\"total_tokens\":4967,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":0,\"reasoning_tokens\":1639,\"rejected_prediction_tokens\":null,\"image_tokens\":0},\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":0,\"cache_write_tokens\":0,\"video_tokens\":0},\"cost\":0.00533793,\"is_byok\":false,\"cost_details\":{\"upstream_inference_cost\":0.00533793,\"upstream_inference_prompt_cost\":0.00125697,\"upstream_inference_completions_cost\":0.00408096}},\"provider\":\"Chutes\"}"}, "scoring": {"f1_macro": 0.56, "f1_micro": 0.56, "cost_summary": {"total_input_tokens": 297006, "total_output_tokens": 195753, "total_tokens": 492759, "input_cost_usd": 0.11583233999999999, "output_cost_usd": 0.45806201999999996, "total_cost_usd": 0.5738943599999999}}, "normalized_score": 56.00000000000001}