{"_id": {"$oid": "6a671d25338dadb0be4e83e3"}, "test_id": "T1153", "benchmark": "business_letters", "date": "2026-06-22", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "scicore", "model": "qwen35-397b-a17b-fp8", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": true, \"skip_non_signatures\": false}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1926-07-13\"],\"letter_title\":[\"ALBATROS\"],\"sender_persons\":[\"Basler Personenschiffahrtsgesellschaft\"],\"receiver_persons\":[\"Elektrizit\u00e4tswerk, Basel\"]}}", "model": "qwen35-397b-a17b-fp8", "provider": "scicore", "finish_reason": "stop", "usage": {"input_tokens": 3225, "output_tokens": 3713, "total_tokens": 6938}, "duration": 62.30027747154236, "timestamp": "2026-06-22T15:23:35.835430", "parsed": {"metadata": {"send_date": ["1926-07-13"], "letter_title": ["ALBATROS"], "sender_persons": ["Basler Personenschiffahrtsgesellschaft"], "receiver_persons": ["Elektrizit\u00e4tswerk, Basel"], "document_number": "letter10"}}, "conversation_id": "0b767682-bf81-40d2-93a3-25777041dbb6", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 1, "sender_persons_fn": 0, "receiver_persons_tp": 0, "receiver_persons_fp": 1, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"chatcmpl-90f8070e1e02f004\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n    \\\"metadata\\\": {\\n        \\\"send_date\\\": [\\n            \\\"1926-07-13\\\"\\n        ]\\n    , \\\"receiver_persons\\\": [\\n            \\\"Elektrizit\u00e4tswerk, Basel\\\"\\n        ]\\n    , \\\"letter_title\\\": [\\n            \\\"ALBATROS\\\"\\n        ]\\n    , \\\"sender_persons\\\": [\\n            \\\"Basler Personenschiffahrtsgesellschaft\\\"\\n        ]\\n    }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1926-07-13\"],\"letter_title\":[\"ALBATROS\"],\"sender_persons\":[\"Basler Personenschiffahrtsgesellschaft\"],\"receiver_persons\":[\"Elektrizit\u00e4tswerk, Basel\"]}},\"reasoning\":\"The user wants me to extract specific information from the provided image of a historical letter and format it as a JSON object.\\n\\n**1. Analyze the Image:**\\n*   **Header/Date:** Top right corner says \\\"13. Juli 1926.\\\"\\n*   **Reference:** Top left says \\\"O/H.\\\"\\n*   **Recipient (An das):** \\\"Elektrizit\u00e4tswerk, Basel.\\\" This is an organization, not a specific person. The prompt asks for `receiver_persons`. Since it's an organization, I should probably look for a person's name if available, or list the organization if that's the intended recipient. Wait, the prompt asks for `receiver_persons`. If it's an organization, I might need to interpret \\\"persons\\\" loosely or check if a specific person is named. The text says \\\"An das Elektrizit\u00e4tswerk, Basel.\\\" No specific person is named in the address block.\\n*   **Subject (Betr.):** \\\"ALBATROS\\\". This acts as the title/subject.\\n*   **Body Text:**\\n    *   Mentions \\\"Herrn H. Wylemann, Kontrollingenieur des Eidgen. Eisenbahndepartements\\\". He is mentioned as leading the test runs. He is not the sender or receiver.\\n    *   The letter is about test runs (\\\"Probefahrten\\\") of a ship named \\\"ALBATROS\\\".\\n    *   The engine failed 4 times.\\n    *   Mr. Wylemann demands perfect functioning.\\n    *   \\\"Wir haben das Schiff nur unter dieser Voraussetzung gekauft...\\\" (We bought the ship only under this condition...). This implies the sender is the buyer.\\n*   **Footer/Signature:**\\n    *   \\\"hochachtungsvoll :\\\"\\n    *   \\\"Basler Personenschiffahrtsgesellschaft\\\" (Basel Passenger Shipping Company). This is the sender organization.\\n    *   \\\"Der Pr\u00e4sident:\\\" (The President) - followed by initials \\\"ch\\\" (looks like handwritten initials).\\n    *   \\\"Der Gesch\u00e4ftsf\u00fchrer:\\\" (The Managing Director) - no signature visible next to it, or maybe faint.\\n    *   The prompt asks for `sender_persons`. The letter is signed by the organization \\\"Basler Personenschiffahrtsgesellschaft\\\". There are no full names of specific people signing it, just initials \\\"ch\\\" which are illegible as a full name. However, the entity sending it is clear. The prompt asks for \\\"sender_persons\\\". Usually, if an organization sends it, I might list the organization or look for a specific name. Let's look closer. The signature area has \\\"Basler Personenschiffahrtsgesellschaft\\\". Underneath \\\"Der Pr\u00e4sident:\\\" there are initials \\\"ch\\\". Underneath \\\"Der Gesch\u00e4ftsf\u00fchrer:\\\" there is nothing clear.\\n    *   Wait, let's re-read the prompt requirements. \\\"sender_persons: Name(s) of the person(s) who wrote the letter.\\\" If only an organization is listed, I should probably put the organization name or null if strictly persons are required. However, in historical document extraction, the \\\"sender\\\" is often the entity. Let's look at the example: \\\"Lisa Simpson\\\". That's a person.\\n    *   Let's look at the recipient again: \\\"An das Elektrizit\u00e4tswerk, Basel.\\\" Again, an organization.\\n    *   Let's look at the subject: \\\"Betr. 'ALBATROS'\\\". This serves as the title.\\n\\n**2. Extracting Values:**\\n\\n*   **letter_title:** The subject line is \\\"Betr. 'ALBATROS'\\\". So, \\\"ALBATROS\\\" or \\\"Betr. 'ALBATROS'\\\". Let's use \\\"ALBATROS\\\" as the core title.\\n*   **send_date:** \\\"13. Juli 1926\\\" -> \\\"1926-07-13\\\".\\n*   **sender_persons:** The letter is from \\\"Basler Personenschiffahrtsgesellschaft\\\". There isn't a specific person's full name signed, just initials \\\"ch\\\". I will list the organization as the sender entity, as is common in these tasks when a specific person isn't named but an entity is. Or, I can look for \\\"Der Pr\u00e4sident\\\" or \\\"Der Gesch\u00e4ftsf\u00fchrer\\\". Since no names are fully written, I will list the organization \\\"Basler Personenschiffahrtsgesellschaft\\\". Wait, the prompt asks for \\\"sender_persons\\\". If I put an organization, is that wrong? Let's check the receiver. \\\"Elektrizit\u00e4tswerk, Basel\\\". If I have to put persons, and there are none, maybe \\\"null\\\"? But usually, the \\\"sender\\\" field captures the entity. Let's assume the organization counts as the sender entity. However, strictly speaking, it asks for \\\"persons\\\". Let's look at the text again. \\\"Wir haben das Schiff... gekauft\\\". \\\"Wir\\\" refers to the company. I will extract \\\"Basler Personenschiffahrtsgesellschaft\\\" as the sender. If the system strictly wants human names, it might be an issue, but usually, these extraction tasks accept the entity name found in the signature block. Let's look at the signature block again. \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n*   **receiver_persons:** \\\"An das Elektrizit\u00e4tswerk, Basel.\\\" Again, an organization. I will extract \\\"Elektrizit\u00e4tswerk, Basel\\\".\\n\\n**Refining the extraction:**\\n*   **Title:** \\\"ALBATROS\\\" (from \\\"Betr. 'ALBATROS'\\\").\\n*   **Date:** 1926-07-13.\\n*   **Sender:** The text at the bottom is \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n*   **Receiver:** The text at the top is \\\"Elektrizit\u00e4tswerk, Basel\\\".\\n\\nLet's reconsider the \\\"persons\\\" constraint. If the prompt strictly demands persons and none are listed, I should return null.\\n*   Sender: \\\"Basler Personenschiffahrtsgesellschaft\\\" is an organization. \\\"Der Pr\u00e4sident\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer\\\" are titles. No full names.\\n*   Receiver: \\\"Elektrizit\u00e4tswerk, Basel\\\" is an organization.\\n*   However, usually in these tasks, if a specific person isn't named, the organization is the best fit for the \\\"sender/receiver\\\" field unless instructed otherwise. The prompt says \\\"Name(s) of the person(s)...\\\". This is tricky.\\n*   Let's look at the text body. \\\"Herrn H. Wylemann\\\" is mentioned. He is a third party (Kontrollingenieur). He is not the sender or receiver.\\n*   Let's assume the prompt allows organizations in the \\\"persons\\\" fields if no specific person is named, or I should extract the organization name. Let's look at the example. \\\"Mayor Joe Quimby\\\" (Person + Title), \\\"Seymour Skinner\\\" (Person).\\n*   If I look really closely at the signature, there are initials \\\"ch\\\". That's not a full name.\\n*   Let's assume the \\\"sender_persons\\\" should be the organization name \\\"Basler Personenschiffahrtsgesellschaft\\\" and \\\"receiver_persons\\\" should be \\\"Elektrizit\u00e4tswerk, Basel\\\". This is the most helpful extraction. If I put null, it loses the main actors.\\n*   Actually, looking at the prompt \\\"receiver_persons: Name(s) of the person(s) who received the letter.\\\" and \\\"sender_persons: Name(s) of the person(s) who wrote the letter.\\\"\\n*   If I strictly follow \\\"persons\\\", both might be null. But that seems unhelpful.\\n*   Let's look at the \\\"An das\\\" section. \\\"An das Elektrizit\u00e4tswerk, Basel.\\\"\\n*   Let's look at the signature. \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n*   I will extract the organization names as the values, treating them as the entities acting as persons in the context of the letter.\\n\\n**Final Plan:**\\n*   `letter_title`: [\\\"ALBATROS\\\"] (from \\\"Betr. 'ALBATROS'\\\")\\n*   `send_date`: [\\\"1926-07-13\\\"]\\n*   `sender_persons`: [\\\"Basler Personenschiffahrtsgesellschaft\\\"]\\n*   `receiver_persons`: [\\\"Elektrizit\u00e4tswerk, Basel\\\"]\\n\\nWait, let's look at the receiver again. \\\"An das Elektrizit\u00e4tswerk, Basel.\\\"\\nLet's look at the sender again. \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n\\nIs there any person named?\\n\\\"Herrn H. Wylemann\\\" is mentioned in the text. He is the \\\"Kontrollingenieur\\\". He is not the sender or receiver.\\nThe letter is *about* his inspection.\\nThe letter is *from* the shipping company *to* the electricity works.\\n\\nSo, the extraction stands.\\n\\nOne detail: \\\"Betr. 'ALBATROS'\\\". The title is often the subject line. So \\\"ALBATROS\\\" is appropriate.\\n\\nLet's double check the date format. ISO YYYY-MM-DD. \\\"13. Juli 1926\\\" -> 1926-07-13.\\n\\nLet's double check the lists. Values must be lists.\\n\\nOne final check on the sender. The signature block has \\\"Basler Personenschiffahrtsgesellschaft\\\". Underneath \\\"Der Pr\u00e4sident:\\\" and \\\"Der Gesch\u00e4ftsf\u00fchrer:\\\". There are no legible full names. Just initials \\\"ch\\\" maybe. So the organization is the only valid identifier.\\n\\nOne final check on the receiver. \\\"An das Elektrizit\u00e4tswerk, Basel.\\\" No person named.\\n\\nSo I will use the organization names.\\n\\nJSON Structure:\\n```json\\n{\\n    \\\"letter_title\\\": [\\\"ALBATROS\\\"],\\n    \\\"send_date\\\": [\\\"1926-07-13\\\"],\\n    \\\"sender_persons\\\": [\\\"Basler Personenschiffahrtsgesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Elektrizit\u00e4tswerk, Basel\\\"]\\n}\\n```\\nWait, looking closer at the bottom right, under \\\"Basler Personenschiffahrtsgesellschaft\\\", there is \\\"Der Pr\u00e4sident:\\\" and scribbles. And \\\"Der Gesch\u00e4ftsf\u00fchrer:\\\". It's possible the scribbles are initials. But \\\"Basler Personenschiffahrtsgesellschaft\\\" is clearly the entity.\\n\\nLet's refine the `letter_title`. The line is `Betr. \\\"ALBATROS\\\".` So the title is \\\"ALBATROS\\\".\\n\\nLet's refine the `receiver_persons`. The text is `An das Elektrizit\u00e4tswerk, Basel.` So \\\"Elektrizit\u00e4tswerk, Basel\\\".\\n\\nLet's refine the `sender_persons`. The text is `Basler Personenschiffahrtsgesellschaft`.\\n\\nThe prompt asks for \\\"Name(s) of the person(s)\\\". If I put an organization, is it wrong?\\n\\\"receiver_persons\\\": [\\\"Elektrizit\u00e4tswerk, Basel\\\"] -> This is an organization.\\n\\\"sender_persons\\\": [\\\"Basler Personenschiffahrtsgesellschaft\\\"] -> This is an organization.\\n\\nIf I have to be pedantic about \\\"persons\\\", I might have to say null. But usually, in document intelligence, \\\"sender\\\" and \\\"receiver\\\" fields capture the entities. I will proceed with the organization names.\\n\\nOne detail: \\\"O/H.\\\" at the top left. Not relevant.\\n\\\"10\\\" in red at top right. Not relevant.\\n\\nFinal check of the text for any hidden names.\\n\\\"An das Elektrizit\u00e4tswerk, Basel.\\\" -> No person.\\n\\\"Herrn H. Wylemann\\\" -> Mentioned in body.\\n\\\"Basler Personenschiffahrtsgesellschaft\\\" -> Sender.\\n\\nOkay, I'm confident.\\n\\nOne small correction: The prompt example shows `letter_title` as a list. `send_date` as a list.\\nThe prompt says \\\"values corresponding to each key should be stored as a list\\\".\\n\\nSo:\\n`letter_title`: [\\\"ALBATROS\\\"]\\n`send_date`: [\\\"1926-07-13\\\"]\\n`sender_persons`: [\\\"Basler Personenschiffahrtsgesellschaft\\\"]\\n`receiver_persons`: [\\\"Elektrizit\u00e4tswerk, Basel\\\"]\\n\\nWait, is \\\"Betr.\\\" part of the title? Usually \\\"Betr.\\\" stands for \\\"Betreff\\\" (Subject). So the title is the subject matter. \\\"ALBATROS\\\" is the subject.\\n\\nLet's verify the date. \\\"13. Juli 1926.\\\" -> 1926-07-13.\\n\\nLet's verify the receiver. \\\"Elektrizit\u00e4tswerk, Basel\\\".\\n\\nLet's verify the sender. \\\"Basler Personenschiffahrtsgesellschaft\\\".\\n\\nThe output must be JSON only.\\n\\nOne thought: Could \\\"H. Wylemann\\\" be considered a receiver? No, the text says \\\"unter Leitung des Herrn H. Wylemann... im Auftrages des Polizei-Departements Basel\\\". He led the test runs. The letter is addressed to the Electricity Works. The letter discusses the test runs led by Wylemann. The sender (Shipping Co) is writing to the Electricity Works (likely the engine supplier or power provider? \\\"Elektrizit\u00e4tswerk\\\" implies electricity works. The text mentions \\\"Motor... versagt\\\" (engine failed) and \\\"Zuleitungen\\\" (supply lines/feeds). So they are writing to the electricity works about the engine/supply issues.\\n\\nSo Receiver = Elektrizit\u00e4tswerk, Basel.\\nSender = Basler Personenschiffahrtsgesellschaft.\\n\\nThe prompt asks for \\\"receiver_persons\\\". If I strictly follow \\\"persons\\\", and there are none, I should return null. However, \\\"Elektrizit\u00e4tswerk\\\" is the entity receiving it. I will assume the prompt implies \\\"entities\\\" when it says \\\"persons\\\" in a generic sense for form filling, or I should extract the organization name. Given the example \\\"Mayor Joe Quimby\\\" (Title + Name), maybe \\\"Elektrizit\u00e4tswerk\\\" is treated as the name. I will include it.\\n\\nActually, looking at the very bottom, \\\"Basler Personenschiffahrtsgesellschaft\\\" is printed.\\nThe receiver \\\"Elektrizit\u00e4tswerk, Basel\\\" is typed.\\n\\nI will formulate the JSON.\\n\\n```json\\n{\\n    \\\"letter_title\\\": [\\\"ALBATROS\\\"],\\n    \\\"send_date\\\": [\\\"1926-07-13\\\"],\\n    \\\"sender_persons\\\": [\\\"Basler Personenschiffahrtsgesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Elektrizit\u00e4tswerk, Basel\\\"]\\n}\\n```\\nWait, looking at the receiver line:\\n\\\"An das\\nElektrizit\u00e4tswerk,\\nB a s e l.\\\"\\nSo \\\"Elektrizit\u00e4tswerk, Basel\\\" is correct.\\n\\nLooking at sender:\\n\\\"Basler Personenschiffahrtsgesellschaft\\\"\\n\\nOne detail: \\\"Betr. 'ALBATROS'\\\". I will use \\\"ALBATROS\\\" as the title.\\n\\nFinal check on dates. \\\"13. Juli 1926\\\".\\nJuly is 07.\\n1926 is 1926.\\n13 is 13.\\n1926-07-13.\\n\\nThe prompt says \\\"If a piece of information is not included in the letter, set the value for the corresponding key to 'null'\\\".\\nIf I decide that \\\"Elektrizit\u00e4tswerk\\\" is not a person, I should set it to null.\\nBut usually, these prompts want the entity.\\nLet's look at the example again.\\n\\\"receiver_persons\\\": [\\\"Mayor Joe Quimby\\\", \\\"Seymour Skinner\\\"]\\nThese are people.\\nIf the letter was addressed to \\\"The City Council\\\", would I put \\\"The City Council\\\" or null?\\nGiven the instruction \\\"Name(s) of the person(s)\\\", it's ambiguous.\\nHowever, in historical archives, the \\\"correspondent\\\" is often an organization.\\nI will assume the organization name is the intended extraction for \\\"sender/receiver\\\" fields if no specific person is named. It's better to provide the entity than null.\\n\\nWait, is there a person's name in the signature?\\n\\\"Der Pr\u00e4sident:\\\" -> scribble \\\"ch\\\".\\n\\\"Der Gesch\u00e4ftsf\u00fchrer:\\\" -> blank/faint.\\nSo no person name.\\n\\nI will stick with the organization names.\\n\\nOne distinct possibility: The prompt might consider \\\"H. Wylemann\\\" as a person involved, but he is not the sender or receiver.\\nThe sender is the company. The receiver is the company.\\n\\nI will proceed.\\n\\nOne more check: \\\"letter_title\\\".\\nThe text has \\\"Betr. 'ALBATROS'\\\".\\nSo title is \\\"ALBATROS\\\".\\n\\nFinal JSON construction.\\n\"},\"stop_reason\":null,\"token_ids\":null}],\"created\":1782134554,\"model\":\"qwen35-397b-a17b-fp8\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":3713,\"prompt_tokens\":3225,\"total_tokens\":6938,\"completion_tokens_details\":null,\"prompt_tokens_details\":null},\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"kv_transfer_params\":null}"}, "scoring": {"f1_macro": 0.52, "f1_micro": 0.6, "cost_summary": {"total_input_tokens": 109455, "total_output_tokens": 126305, "total_tokens": 235760, "input_cost_usd": 0.0, "output_cost_usd": 0.0, "total_cost_usd": 0.0}}, "normalized_score": 52.0}