{"_id": {"$oid": "6a26c67c3572c06bc1455df9"}, "test_id": "T0895", "benchmark": "business_letters", "date": "2026-04-21", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "openrouter", "model": "qwen/qwen3.6-plus", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": false, \"skip_non_signatures\": true}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1932-12-19\"],\"letter_title\":[\"Betr. Personenschiffahrt\"],\"sender_persons\":[\"Basler Rheinschiffahrt-Aktiengesellschaft\"],\"receiver_persons\":[\"Dr. M. Vischer - von Planta\"]}}", "model": "qwen/qwen3.6-plus-04-02", "provider": "openrouter", "finish_reason": "stop", "usage": {"input_tokens": 3124, "output_tokens": 1097, "total_tokens": 4221, "input_cost_usd": 0.0010153, "output_cost_usd": 0.0021391500000000003, "estimated_cost_usd": 0.0031544500000000005}, "duration": 16.801660537719727, "timestamp": "2026-04-21T11:10:39.403501", "parsed": {"metadata": {"send_date": ["1932-12-19"], "letter_title": ["Betr. Personenschiffahrt"], "sender_persons": ["Basler Rheinschiffahrt-Aktiengesellschaft"], "receiver_persons": ["Dr. M. Vischer - von Planta"], "document_number": "letter60"}}, "conversation_id": "8116765c-c20e-47c7-8e7b-9035e96b75d3", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 1, "sender_persons_fn": 2, "receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"gen-1776762622-H1kJjpDZZvC82v0GL8pJ\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n    \\\"metadata\\\": {\\n        \\\"send_date\\\": [\\n            \\\"1932-12-19\\\"\\n        ],\\n        \\\"letter_title\\\": [\\n            \\\"Betr. Personenschiffahrt\\\"\\n        ],\\n        \\\"sender_persons\\\": [\\n            \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"\\n        ],\\n        \\\"receiver_persons\\\": [\\n            \\\"Dr. M. Vischer - von Planta\\\"\\n        ]\\n    }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1932-12-19\"],\"letter_title\":[\"Betr. Personenschiffahrt\"],\"sender_persons\":[\"Basler Rheinschiffahrt-Aktiengesellschaft\"],\"receiver_persons\":[\"Dr. M. Vischer - von Planta\"]}},\"reasoning\":\"user wants me to extract specific metadata from a historical letter image.\\n\\n**1. Analyze the Image:**\\n*   **Header:** \\\"Basler Rheinschiffahrt-Aktiengesellschaft Basel\\\". This is the organization sending the letter.\\n*   **Date:** \\\"Basel, den 19. Dezember 1932\\\". This is the date.\\n*   **Recipient:** \\\"Herr Dr. M. Vischer - von Planta Advokat & Notar Basel\\\". This is the person receiving the letter.\\n*   **Subject Line:** \\\"Betr. Personenschiffahrt\\\". This acts as a title or subject.\\n*   **Body:** Discusses a loan agreement (\\\"Darlehensvertrag\\\") and passenger shipping (\\\"Personenschiffahrt\\\"). Mentions \\\"Verein f\u00fcr die Schiffahrt auf dem Oberrhein\\\".\\n*   **Closing:** \\\"Mit vorz\u00fcglicher Hochachtung : Basler Rheinschiffahrt Aktiengesellschaft\\\".\\n*   **Signature:** There is a signature that looks like \\\"Kunz\\\" or similar, but the sender is primarily the organization \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". The prompt asks for \\\"sender_persons\\\". Since it's a company letter, the company name is the sender. Sometimes specific people sign, but here the signature is illegible/scribbled. The typed name at the bottom is the company. I will use the company name as the sender. Wait, looking closely at the signature, it looks like \\\"Kunz\\\" and maybe another name next to it. However, the prompt asks for \\\"sender_persons\\\". Usually, for business letters, the organization is the sender unless a specific person is named as the author. The letter is signed on behalf of the company. I will list the company as the sender. Or perhaps I should look for a specific person. The signature is hard to read. It looks like \\\"Kunz\\\". Let's stick to the organization name as the primary sender entity, or if I must provide a person, I might have to infer. But the prompt asks for \\\"sender_persons\\\". The letter is from \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". I will treat the organization as the sender. Wait, the schema asks for \\\"sender_persons\\\". If it's a company, I should probably put the company name. Let's look at the example. \\\"Lisa Simpson\\\". That's a person. Here, the sender is a company. I will put the company name.\\n*   **Letter Title:** The subject line \\\"Betr. Personenschiffahrt\\\" serves as the title.\\n\\n**2. Extract Values:**\\n*   **letter_title:** \\\"Betr. Personenschiffahrt\\\" (Subject: Passenger Shipping).\\n*   **send_date:** \\\"19. Dezember 1932\\\" -> ISO format: \\\"1932-12-19\\\".\\n*   **sender_persons:** \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". (The letter is signed by the company).\\n*   **receiver_persons:** \\\"Dr. M. Vischer - von Planta\\\".\\n\\n**3. Format Output:**\\n*   Construct the JSON according to the schema.\\n*   `letter_title`: [\\\"Betr. Personenschiffahrt\\\"]\\n*   `send_date`: [\\\"1932-12-19\\\"]\\n*   `sender_persons`: [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"]\\n*   `receiver_persons`: [\\\"Dr. M. Vischer - von Planta\\\"]\\n\\nLet's refine the sender. The letter is from the \\\"Direktion\\\" (Directorate) of the company. The signature is illegible. I will use the company name.\\nLet's refine the receiver. \\\"Herr Dr. M. Vischer - von Planta\\\". I will include the full name.\\n\\nFinal check of the date: \\\"19. Dezember 1932\\\". Correct.\\nFinal check of the title: \\\"Betr. Personenschiffahrt\\\". Correct.\\n\\nOne detail: The prompt asks for \\\"sender_persons\\\". If it's a company, is it a person? In historical document extraction, usually the entity sending it is listed. I will list the company.\\n\\nLet's look at the signature again. It looks like \\\"Kunz\\\". And maybe \\\"G...\\\". It's too hard to read confidently. I will stick with the company name as the sender, as it is explicitly typed at the bottom.\\n\\nWait, looking at the schema, `sender_persons` expects a list of strings.\\n`receiver_persons` expects a list of strings.\\n\\nLet's assemble the JSON.\",\"reasoning_details\":[{\"type\":\"reasoning.text\",\"text\":\"user wants me to extract specific metadata from a historical letter image.\\n\\n**1. Analyze the Image:**\\n*   **Header:** \\\"Basler Rheinschiffahrt-Aktiengesellschaft Basel\\\". This is the organization sending the letter.\\n*   **Date:** \\\"Basel, den 19. Dezember 1932\\\". This is the date.\\n*   **Recipient:** \\\"Herr Dr. M. Vischer - von Planta Advokat & Notar Basel\\\". This is the person receiving the letter.\\n*   **Subject Line:** \\\"Betr. Personenschiffahrt\\\". This acts as a title or subject.\\n*   **Body:** Discusses a loan agreement (\\\"Darlehensvertrag\\\") and passenger shipping (\\\"Personenschiffahrt\\\"). Mentions \\\"Verein f\u00fcr die Schiffahrt auf dem Oberrhein\\\".\\n*   **Closing:** \\\"Mit vorz\u00fcglicher Hochachtung : Basler Rheinschiffahrt Aktiengesellschaft\\\".\\n*   **Signature:** There is a signature that looks like \\\"Kunz\\\" or similar, but the sender is primarily the organization \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". The prompt asks for \\\"sender_persons\\\". Since it's a company letter, the company name is the sender. Sometimes specific people sign, but here the signature is illegible/scribbled. The typed name at the bottom is the company. I will use the company name as the sender. Wait, looking closely at the signature, it looks like \\\"Kunz\\\" and maybe another name next to it. However, the prompt asks for \\\"sender_persons\\\". Usually, for business letters, the organization is the sender unless a specific person is named as the author. The letter is signed on behalf of the company. I will list the company as the sender. Or perhaps I should look for a specific person. The signature is hard to read. It looks like \\\"Kunz\\\". Let's stick to the organization name as the primary sender entity, or if I must provide a person, I might have to infer. But the prompt asks for \\\"sender_persons\\\". The letter is from \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". I will treat the organization as the sender. Wait, the schema asks for \\\"sender_persons\\\". If it's a company, I should probably put the company name. Let's look at the example. \\\"Lisa Simpson\\\". That's a person. Here, the sender is a company. I will put the company name.\\n*   **Letter Title:** The subject line \\\"Betr. Personenschiffahrt\\\" serves as the title.\\n\\n**2. Extract Values:**\\n*   **letter_title:** \\\"Betr. Personenschiffahrt\\\" (Subject: Passenger Shipping).\\n*   **send_date:** \\\"19. Dezember 1932\\\" -> ISO format: \\\"1932-12-19\\\".\\n*   **sender_persons:** \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". (The letter is signed by the company).\\n*   **receiver_persons:** \\\"Dr. M. Vischer - von Planta\\\".\\n\\n**3. Format Output:**\\n*   Construct the JSON according to the schema.\\n*   `letter_title`: [\\\"Betr. Personenschiffahrt\\\"]\\n*   `send_date`: [\\\"1932-12-19\\\"]\\n*   `sender_persons`: [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"]\\n*   `receiver_persons`: [\\\"Dr. M. Vischer - von Planta\\\"]\\n\\nLet's refine the sender. The letter is from the \\\"Direktion\\\" (Directorate) of the company. The signature is illegible. I will use the company name.\\nLet's refine the receiver. \\\"Herr Dr. M. Vischer - von Planta\\\". I will include the full name.\\n\\nFinal check of the date: \\\"19. Dezember 1932\\\". Correct.\\nFinal check of the title: \\\"Betr. Personenschiffahrt\\\". Correct.\\n\\nOne detail: The prompt asks for \\\"sender_persons\\\". If it's a company, is it a person? In historical document extraction, usually the entity sending it is listed. I will list the company.\\n\\nLet's look at the signature again. It looks like \\\"Kunz\\\". And maybe \\\"G...\\\". It's too hard to read confidently. I will stick with the company name as the sender, as it is explicitly typed at the bottom.\\n\\nWait, looking at the schema, `sender_persons` expects a list of strings.\\n`receiver_persons` expects a list of strings.\\n\\nLet's assemble the JSON.\",\"format\":\"unknown\",\"index\":0}]},\"native_finish_reason\":\"stop\"}],\"created\":1776762622,\"model\":\"qwen/qwen3.6-plus-04-02\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":1097,\"prompt_tokens\":3124,\"total_tokens\":4221,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":0,\"reasoning_tokens\":978,\"rejected_prediction_tokens\":null,\"image_tokens\":0},\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":0,\"cache_write_tokens\":0,\"video_tokens\":0},\"cost\":0.00315445,\"is_byok\":false,\"cost_details\":{\"upstream_inference_cost\":0.00315445,\"upstream_inference_prompt_cost\":0.0010153,\"upstream_inference_completions_cost\":0.00213915}},\"provider\":\"Alibaba\"}"}, "scoring": {"f1_macro": 0.6, "f1_micro": 0.54, "cost_summary": {"total_input_tokens": 178150, "total_output_tokens": 97125, "total_tokens": 275275, "input_cost_usd": 0.05789874999999997, "output_cost_usd": 0.18939375000000003, "total_cost_usd": 0.24729250000000003}}, "normalized_score": 60.0}