{"_id": {"$oid": "6a671d25338dadb0be4e82f5"}, "test_id": "T0894", "benchmark": "business_letters", "date": "2026-04-21", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "openrouter", "model": "qwen/qwen3.6-plus", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": true, \"skip_non_signatures\": false}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1928-03-12\"],\"letter_title\":null,\"sender_persons\":[\"Basler Personenschiffahrtsgesellschaft\"],\"receiver_persons\":[\"Basler Rheinschiffahrt A.G.\"]}}", "model": "qwen/qwen3.6-plus-04-02", "provider": "openrouter", "finish_reason": "stop", "usage": {"input_tokens": 3124, "output_tokens": 1526, "total_tokens": 4650, "input_cost_usd": 0.0010153, "output_cost_usd": 0.0029757, "estimated_cost_usd": 0.003991}, "duration": 27.117142915725708, "timestamp": "2026-04-21T11:07:46.833047", "parsed": {"metadata": {"send_date": ["1928-03-12"], "letter_title": null, "sender_persons": ["Basler Personenschiffahrtsgesellschaft"], "receiver_persons": ["Basler Rheinschiffahrt A.G."], "document_number": "letter18"}}, "conversation_id": "773b37de-a073-4467-a40e-1893ec4cfb01", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 1, "sender_persons_fn": 0, "receiver_persons_tp": 0, "receiver_persons_fp": 1, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"gen-1776762440-6SnuC3V7Iiz5GOmB9r5s\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n    \\\"metadata\\\": {\\n        \\\"send_date\\\": [\\n            \\\"1928-03-12\\\"\\n        ],\\n        \\\"letter_title\\\": null,\\n        \\\"sender_persons\\\": [\\n            \\\"Basler Personenschiffahrtsgesellschaft\\\"\\n        ],\\n        \\\"receiver_persons\\\": [\\n            \\\"Basler Rheinschiffahrt A.G.\\\"\\n        ]\\n    }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1928-03-12\"],\"letter_title\":null,\"sender_persons\":[\"Basler Personenschiffahrtsgesellschaft\"],\"receiver_persons\":[\"Basler Rheinschiffahrt A.G.\"]}},\"reasoning\":\"user wants me to extract specific metadata from a historical letter image.\\n\\n**1. Analyze the image:**\\n*   **Header:** \\\"Basel, den 12. M\u00e4rz 1928.\\\" -> This is the date and location.\\n*   **Recipient:** \\\"An die Basler Rheinschiffahrt A.G., Basel.\\\" -> This is the receiver.\\n*   **Body:** \\\"Wir bekennen uns zum Empfang Ihres Schreibens vom 1.ds. und teilen Ihnen h\u00f6flich mit, dass dessen Behandlung auf der Traktandenliste der Vorstands-Sitzung vom 13. M\u00e4rz steht.\\\" (We acknowledge receipt of your letter from the 1st of this month and inform you politely that its treatment is on the agenda of the board meeting on March 13th.)\\n*   **Closing:** \\\"Wir werden nicht verfehlen Ihnen in K\u00fcrze weitere Mitteilungen zukommen zu lassen und zeichnen hochachtungsvoll:\\\" (We will not fail to send you further communications shortly and sign with high respect:)\\n*   **Signature/Stamp:** There is a faint stamp at the bottom right. It reads \\\"Basler Personenschiffahrtsgesellschaft\\\" and below that \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". This indicates the sender is the \\\"Basler Personenschiffahrtsgesellschaft\\\" (Basel Passenger Shipping Company), specifically the Managing Director (\\\"Der Gesch\u00e4ftsf\u00fchrer\\\"). Since no specific person's name is signed, the entity is the sender.\\n\\n**2. Extract the required fields:**\\n\\n*   **letter_title:** There is no explicit title like \\\"Subject: ...\\\". It's a standard business letter acknowledging receipt. I will set this to null as there is no specific title provided in the text.\\n*   **sender_persons:** The stamp says \\\"Basler Personenschiffahrtsgesellschaft\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". So the sender is the company or the managing director of that company. I will use the company name as the sender entity.\\n*   **send_date:** \\\"12. M\u00e4rz 1928\\\". In ISO format: 1928-03-12.\\n*   **receiver_persons:** \\\"An die Basler Rheinschiffahrt A.G.\\\". This is the receiver.\\n\\n**3. Format the output:**\\n*   Construct the JSON according to the schema.\\n*   `letter_title`: null (or empty list if strictly following \\\"list\\\" instruction, but schema says `anyOf: [array, null]`. The prompt says \\\"set the value... to 'null'\\\". Wait, the prompt says \\\"values corresponding to each key should be stored as a list... If a piece of information is not included... set the value... to 'null'\\\". This is slightly contradictory. The example shows lists. The schema allows null. I will follow the schema which allows null for the whole field or a list. Let's look at the example: `\\\"letter_title\\\": [\\\"Petition...\\\"]`. If not found, usually `null` is safer or `[]`. The prompt says \\\"set the value for the corresponding key to 'null'\\\". So I will use `null` for missing info.\\n*   `sender_persons`: [\\\"Basler Personenschiffahrtsgesellschaft\\\"] (or \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" but the company is the entity). Let's include the company name.\\n*   `send_date`: [\\\"1928-03-12\\\"]\\n*   `receiver_persons`: [\\\"Basler Rheinschiffahrt A.G.\\\"]\\n\\n**Refining Sender/Receiver:**\\n*   Sender: The stamp is \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n*   Receiver: \\\"Basler Rheinschiffahrt A.G.\\\"\\n\\n**Final JSON Construction:**\\n```json\\n{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1928-03-12\\\"],\\n    \\\"letter_title\\\": null,\\n    \\\"sender_persons\\\": [\\\"Basler Personenschiffahrtsgesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Basler Rheinschiffahrt A.G.\\\"]\\n  }\\n}\\n```\\nWait, looking closely at the stamp, it's very faint. \\\"Basler Personenschiffahrtsgesellschaft\\\". Yes, that makes sense in the context of Swiss shipping history (Basel has shipping on the Rhine). The receiver is \\\"Basler Rheinschiffahrt A.G.\\\". These are likely related or competing entities.\\n\\nLet's double check the date. \\\"Basel, den 12. M\u00e4rz 1928.\\\" -> 1928-03-12. Correct.\\n\\nLet's check the sender again. The text says \\\"Wir...\\\" (We). The stamp is the signature. The stamp text is faint but readable as \\\"Basler Personenschiffahrtsgesellschaft\\\". Underneath \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". So the sender is the company.\\n\\nLet's check the receiver. \\\"An die Basler Rheinschiffahrt A.G.,\\\".\\n\\nLet's check for a title. There is no subject line. So `letter_title` is null.\\n\\nOne detail: The prompt asks for \\\"sender_persons\\\". Is a company a person? In historical document extraction, usually the entity signing is listed. I will list the company name.\\n\\nOne more check on the prompt: \\\"values corresponding to each key should be stored as a list, even if there is only a single value for a key\\\".\\n\\\"If a piece of information is not included in the letter, set the value for the corresponding key to 'null'\\\".\\nThis implies if the key exists but has no value, use null. If it has a value, use a list.\\n\\nSo:\\n`send_date`: [\\\"1928-03-12\\\"]\\n`letter_title`: null\\n`sender_persons`: [\\\"Basler Personenschiffahrtsgesellschaft\\\"]\\n`receiver_persons`: [\\\"Basler Rheinschiffahrt A.G.\\\"]\\n\\nWait, looking at the stamp again. It looks like \\\"Basler Personenschiffahrtsgesellschaft\\\".\\nActually, looking really closely at crop 4, it says \\\"Basler Personenschiffahrtsgesellschaft\\\".\\nAnd below \\\"Der Gesch\u00e4ftsf\u00fchrer\\\".\\n\\nOkay, I'm confident.\\n\\nOne final check on the schema.\\n`metadata` object containing `send_date`, `letter_title`, `sender_persons`, `receiver_persons`.\\nAll fields are required in the schema?\\n`\\\"required\\\": [\\\"send_date\\\", \\\"letter_title\\\", \\\"sender_persons\\\", \\\"receiver_persons\\\"]`\\nYes.\\nAnd types are `anyOf: [array, null]`.\\nSo `letter_title` can be `null`.\\n\\nLet's assemble.\",\"reasoning_details\":[{\"type\":\"reasoning.text\",\"text\":\"user wants me to extract specific metadata from a historical letter image.\\n\\n**1. Analyze the image:**\\n*   **Header:** \\\"Basel, den 12. M\u00e4rz 1928.\\\" -> This is the date and location.\\n*   **Recipient:** \\\"An die Basler Rheinschiffahrt A.G., Basel.\\\" -> This is the receiver.\\n*   **Body:** \\\"Wir bekennen uns zum Empfang Ihres Schreibens vom 1.ds. und teilen Ihnen h\u00f6flich mit, dass dessen Behandlung auf der Traktandenliste der Vorstands-Sitzung vom 13. M\u00e4rz steht.\\\" (We acknowledge receipt of your letter from the 1st of this month and inform you politely that its treatment is on the agenda of the board meeting on March 13th.)\\n*   **Closing:** \\\"Wir werden nicht verfehlen Ihnen in K\u00fcrze weitere Mitteilungen zukommen zu lassen und zeichnen hochachtungsvoll:\\\" (We will not fail to send you further communications shortly and sign with high respect:)\\n*   **Signature/Stamp:** There is a faint stamp at the bottom right. It reads \\\"Basler Personenschiffahrtsgesellschaft\\\" and below that \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". This indicates the sender is the \\\"Basler Personenschiffahrtsgesellschaft\\\" (Basel Passenger Shipping Company), specifically the Managing Director (\\\"Der Gesch\u00e4ftsf\u00fchrer\\\"). Since no specific person's name is signed, the entity is the sender.\\n\\n**2. Extract the required fields:**\\n\\n*   **letter_title:** There is no explicit title like \\\"Subject: ...\\\". It's a standard business letter acknowledging receipt. I will set this to null as there is no specific title provided in the text.\\n*   **sender_persons:** The stamp says \\\"Basler Personenschiffahrtsgesellschaft\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". So the sender is the company or the managing director of that company. I will use the company name as the sender entity.\\n*   **send_date:** \\\"12. M\u00e4rz 1928\\\". In ISO format: 1928-03-12.\\n*   **receiver_persons:** \\\"An die Basler Rheinschiffahrt A.G.\\\". This is the receiver.\\n\\n**3. Format the output:**\\n*   Construct the JSON according to the schema.\\n*   `letter_title`: null (or empty list if strictly following \\\"list\\\" instruction, but schema says `anyOf: [array, null]`. The prompt says \\\"set the value... to 'null'\\\". Wait, the prompt says \\\"values corresponding to each key should be stored as a list... If a piece of information is not included... set the value... to 'null'\\\". This is slightly contradictory. The example shows lists. The schema allows null. I will follow the schema which allows null for the whole field or a list. Let's look at the example: `\\\"letter_title\\\": [\\\"Petition...\\\"]`. If not found, usually `null` is safer or `[]`. The prompt says \\\"set the value for the corresponding key to 'null'\\\". So I will use `null` for missing info.\\n*   `sender_persons`: [\\\"Basler Personenschiffahrtsgesellschaft\\\"] (or \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" but the company is the entity). Let's include the company name.\\n*   `send_date`: [\\\"1928-03-12\\\"]\\n*   `receiver_persons`: [\\\"Basler Rheinschiffahrt A.G.\\\"]\\n\\n**Refining Sender/Receiver:**\\n*   Sender: The stamp is \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n*   Receiver: \\\"Basler Rheinschiffahrt A.G.\\\"\\n\\n**Final JSON Construction:**\\n```json\\n{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1928-03-12\\\"],\\n    \\\"letter_title\\\": null,\\n    \\\"sender_persons\\\": [\\\"Basler Personenschiffahrtsgesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Basler Rheinschiffahrt A.G.\\\"]\\n  }\\n}\\n```\\nWait, looking closely at the stamp, it's very faint. \\\"Basler Personenschiffahrtsgesellschaft\\\". Yes, that makes sense in the context of Swiss shipping history (Basel has shipping on the Rhine). The receiver is \\\"Basler Rheinschiffahrt A.G.\\\". These are likely related or competing entities.\\n\\nLet's double check the date. \\\"Basel, den 12. M\u00e4rz 1928.\\\" -> 1928-03-12. Correct.\\n\\nLet's check the sender again. The text says \\\"Wir...\\\" (We). The stamp is the signature. The stamp text is faint but readable as \\\"Basler Personenschiffahrtsgesellschaft\\\". Underneath \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". So the sender is the company.\\n\\nLet's check the receiver. \\\"An die Basler Rheinschiffahrt A.G.,\\\".\\n\\nLet's check for a title. There is no subject line. So `letter_title` is null.\\n\\nOne detail: The prompt asks for \\\"sender_persons\\\". Is a company a person? In historical document extraction, usually the entity signing is listed. I will list the company name.\\n\\nOne more check on the prompt: \\\"values corresponding to each key should be stored as a list, even if there is only a single value for a key\\\".\\n\\\"If a piece of information is not included in the letter, set the value for the corresponding key to 'null'\\\".\\nThis implies if the key exists but has no value, use null. If it has a value, use a list.\\n\\nSo:\\n`send_date`: [\\\"1928-03-12\\\"]\\n`letter_title`: null\\n`sender_persons`: [\\\"Basler Personenschiffahrtsgesellschaft\\\"]\\n`receiver_persons`: [\\\"Basler Rheinschiffahrt A.G.\\\"]\\n\\nWait, looking at the stamp again. It looks like \\\"Basler Personenschiffahrtsgesellschaft\\\".\\nActually, looking really closely at crop 4, it says \\\"Basler Personenschiffahrtsgesellschaft\\\".\\nAnd below \\\"Der Gesch\u00e4ftsf\u00fchrer\\\".\\n\\nOkay, I'm confident.\\n\\nOne final check on the schema.\\n`metadata` object containing `send_date`, `letter_title`, `sender_persons`, `receiver_persons`.\\nAll fields are required in the schema?\\n`\\\"required\\\": [\\\"send_date\\\", \\\"letter_title\\\", \\\"sender_persons\\\", \\\"receiver_persons\\\"]`\\nYes.\\nAnd types are `anyOf: [array, null]`.\\nSo `letter_title` can be `null`.\\n\\nLet's assemble.\",\"format\":\"unknown\",\"index\":0}]},\"native_finish_reason\":\"stop\"}],\"created\":1776762440,\"model\":\"qwen/qwen3.6-plus-04-02\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":1526,\"prompt_tokens\":3124,\"total_tokens\":4650,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":0,\"reasoning_tokens\":1428,\"rejected_prediction_tokens\":null,\"image_tokens\":0},\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":0,\"cache_write_tokens\":0,\"video_tokens\":0},\"cost\":0.003991,\"is_byok\":false,\"cost_details\":{\"upstream_inference_cost\":0.003991,\"upstream_inference_prompt_cost\":0.0010153,\"upstream_inference_completions_cost\":0.0029757}},\"provider\":\"Alibaba\"}"}, "scoring": {"f1_macro": 0.53, "f1_micro": 0.62, "cost_summary": {"total_input_tokens": 103320, "total_output_tokens": 83353, "total_tokens": 186673, "input_cost_usd": 0.033579000000000005, "output_cost_usd": 0.16253835, "total_cost_usd": 0.19611735000000002}}, "normalized_score": 53.0}