{"_id": {"$oid": "6acf4720ef3fcc24140a7fdc"}, "test_id": "T1393", "benchmark": "business_letters", "date": "2026-07-25", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "openrouter", "model": "moonshotai/kimi-k3", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": false, \"skip_non_signatures\": true}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1926-02-16\"],\"letter_title\":[\"null\"],\"sender_persons\":[\"Basler Rheinschiffahrt-Aktiengesellschaft\"],\"receiver_persons\":[\"Christ\"]}}", "model": "moonshotai/kimi-k3", "provider": "openrouter", "finish_reason": "stop", "usage": {"input_tokens": 12218, "output_tokens": 1355, "total_tokens": 13573, "cached_tokens": 12032, "input_cost_usd": 0.036654, "output_cost_usd": 0.020325000000000003, "estimated_cost_usd": 0.056979}, "duration": 54.69717860221863, "timestamp": "2026-07-25T19:54:52.680288", "parsed": {"metadata": {"send_date": ["1926-02-16"], "letter_title": ["null"], "sender_persons": ["Basler Rheinschiffahrt-Aktiengesellschaft"], "receiver_persons": ["Christ"], "document_number": "letter01"}}, "conversation_id": "df867916-8911-4f38-bca7-d069abe7b251", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 1, "sender_persons_fn": 2, "receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"gen-1785002047-rqYZ7I2uaU6MClwnAuMI\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n    \\\"metadata\\\": {\\n        \\\"letter_title\\\": [\\\"null\\\"],\\n        \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n        \\\"sender_persons\\\": [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"],\\n        \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n    }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1926-02-16\"],\"letter_title\":[\"null\"],\"sender_persons\":[\"Basler Rheinschiffahrt-Aktiengesellschaft\"],\"receiver_persons\":[\"Christ\"]}},\"reasoning\":\"Let me analyze this historical letter carefully.\\n\\nThe letter is from \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" (Basler Rhine Shipping Company), located at Marktgasse 5 III, Basel.\\n\\nDate: \\\"Basel, den 16. Februar 1926\\\" - so 1926-02-16.\\n\\nRecipient: \\\"Herrn Christ, i/Fa. Paravicini, Christ & Co., Basel, Kaufhausgasse 7\\\" - addressed to Herr Christ.\\n\\nSender: The letter is signed by \\\"BASLER RHEINSCHIFFAHRT-AKTIENGESELLSCHAFT\\\" with two handwritten signatures. The signatures are hard to read but appear to be something like \\\"Gm\u00fcnder\\\" or similar and \\\"D. Riller\\\" or \\\"A. Riller\\\" or similar. Let me think about this.\\n\\nLooking at the signatures: The first one looks like it could be \\\"Gminder\\\" or \\\"Gmunder\\\" - hard to tell. The second looks like \\\"Dr. Riller\\\" or \\\"A. Riller\\\" or \\\"D. Miller\\\"...\\n\\nActually, for the sender_persons, since the letter is written on behalf of the company \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" and signed by the company name, the sender would be the company itself, or possibly the signatories if legible.\\n\\nThe signatures are difficult to decipher. The first could be \\\"Gmeinder\\\" or similar, the second \\\"D. Ritter\\\" or \\\"A. Riller\\\"... \\n\\nHistorically, the Basler Rheinschiffahrt-Aktiengesellschaft was founded in 1925/1926. The directors might have been known figures. Without being able to clearly read the signatures, I should probably list the company as the sender.\\n\\nThe letter title: There's no explicit title. It's a business letter. The letterhead says \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". There's no Betreff (subject line). So letter_title would be null.\\n\\nReceiver: Herr Christ (of Firma Paravicini, Christ & Co., Basel). The salutation confirms \\\"Sehr geehrter Herr Christ\\\".\\n\\nSender: Basler Rheinschiffahrt-Aktiengesellschaft - this is a company, not a person. The instructions ask for \\\"sender_persons: Name(s) of the person(s) who wrote the letter.\\\" Since no individual persons are clearly identified (signatures are illegible), I could either put the company name or null. \\n\\nLooking at the signatures more carefully: First signature appears to start with \\\"G\\\" - maybe \\\"Gundlach\\\"? \\\"Gm\u00fcr\\\"? The second starts with what looks like \\\"D\\\" or \\\"A\\\" and then \\\"Riller\\\" or \\\"Ritter\\\" or \\\"M\u00fcller\\\"...\\n\\nActually the second signature looks like \\\"Dr. Riller\\\" or \\\"A. Rillen\\\"... Hmm. It could be \\\"D. Miller\\\" too.\\n\\nGiven uncertainty, I think the safest is to put the company name as sender, since that's who \\\"wrote\\\" the letter (the letter is signed with the company stamp/name). But the key asks for persons. If signatures are illegible, perhaps null is appropriate, but the company name is clearly the sender entity.\\n\\nThe example shows persons' names. But for a corporate letter, the sender is often the company. I'll include \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" as the sender since it's the entity that wrote the letter.\\n\\nDate: 1926-02-16 (also repeated on pages 2 and 3 as 16.2.26).\\n\\nTitle: null (no title/subject line present).\\n\\nReceiver: Herr Christ - perhaps \\\"Christ\\\" or \\\"Herr Christ\\\". The full name might be just \\\"Christ\\\" as surname. The company is Paravicini, Christ & Co. I'll put \\\"Christ\\\" or \\\"Herr Christ\\\". Probably best: \\\"Christ\\\".\\n\\nLet me format the JSON:\\n\\n{\\n    \\\"letter_title\\\": [\\\"null\\\"],\\n    \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n    \\\"sender_persons\\\": [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\\n\\nWait, instructions say: \\\"If a piece of information is not included in the letter, set the value for the corresponding key to 'null'.\\\" The example shows lists. So for missing info, the value should be \\\"null\\\" - probably as a string in the list, or null itself. The example doesn't show null case. I'll use [\\\"null\\\"] to keep list format, or maybe null. Given \\\"the values corresponding to each key should be stored as a list, even if there is only a single value for a key\\\", I'll use [\\\"null\\\"].\\n\\nActually, \\\"set the value for the corresponding key to 'null'\\\" - this suggests the value is \\\"null\\\" (string). But also values should be lists. I'll go with [\\\"null\\\"] for consistency.\\n\\nFor the sender, should I try to decipher the signatures? The signatures appear to be two: one possibly \\\"Gm\u00fcnder\\\"/\\\"Gmeinder\\\"/\\\"Gunder...\\\" and the other \\\"D. Riller\\\"/\\\"A. Riller\\\"/\\\"Dr. Miller\\\". \\n\\nHistorically, Basler Rheinschiffahrt AG (BRAG) - directors around 1926... I'm not certain of specific names. The signatures are too unclear to confidently transcribe. The typed sender is the company. I'll list the company.\\n\\nActually, let me reconsider: maybe the first signature is \\\"Gundert\\\" or \\\"Gundel\\\"... The second \\\"Dr. Ritter\\\"? Given ambiguity, company name is safest.\\n\\nThe receiver's full name: just \\\"Christ\\\" is given, no first name. So \\\"Christ\\\" or \\\"Herr Christ\\\". I'll use \\\"Christ\\\".\\n\\nFinal answer:\\n{\\n    \\\"letter_title\\\": [\\\"null\\\"],\\n    \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n    \\\"sender_persons\\\": [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\",\"reasoning_details\":[{\"type\":\"reasoning.text\",\"text\":\"Let me analyze this historical letter carefully.\\n\\nThe letter is from \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" (Basler Rhine Shipping Company), located at Marktgasse 5 III, Basel.\\n\\nDate: \\\"Basel, den 16. Februar 1926\\\" - so 1926-02-16.\\n\\nRecipient: \\\"Herrn Christ, i/Fa. Paravicini, Christ & Co., Basel, Kaufhausgasse 7\\\" - addressed to Herr Christ.\\n\\nSender: The letter is signed by \\\"BASLER RHEINSCHIFFAHRT-AKTIENGESELLSCHAFT\\\" with two handwritten signatures. The signatures are hard to read but appear to be something like \\\"Gm\u00fcnder\\\" or similar and \\\"D. Riller\\\" or \\\"A. Riller\\\" or similar. Let me think about this.\\n\\nLooking at the signatures: The first one looks like it could be \\\"Gminder\\\" or \\\"Gmunder\\\" - hard to tell. The second looks like \\\"Dr. Riller\\\" or \\\"A. Riller\\\" or \\\"D. Miller\\\"...\\n\\nActually, for the sender_persons, since the letter is written on behalf of the company \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" and signed by the company name, the sender would be the company itself, or possibly the signatories if legible.\\n\\nThe signatures are difficult to decipher. The first could be \\\"Gmeinder\\\" or similar, the second \\\"D. Ritter\\\" or \\\"A. Riller\\\"... \\n\\nHistorically, the Basler Rheinschiffahrt-Aktiengesellschaft was founded in 1925/1926. The directors might have been known figures. Without being able to clearly read the signatures, I should probably list the company as the sender.\\n\\nThe letter title: There's no explicit title. It's a business letter. The letterhead says \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". There's no Betreff (subject line). So letter_title would be null.\\n\\nReceiver: Herr Christ (of Firma Paravicini, Christ & Co., Basel). The salutation confirms \\\"Sehr geehrter Herr Christ\\\".\\n\\nSender: Basler Rheinschiffahrt-Aktiengesellschaft - this is a company, not a person. The instructions ask for \\\"sender_persons: Name(s) of the person(s) who wrote the letter.\\\" Since no individual persons are clearly identified (signatures are illegible), I could either put the company name or null. \\n\\nLooking at the signatures more carefully: First signature appears to start with \\\"G\\\" - maybe \\\"Gundlach\\\"? \\\"Gm\u00fcr\\\"? The second starts with what looks like \\\"D\\\" or \\\"A\\\" and then \\\"Riller\\\" or \\\"Ritter\\\" or \\\"M\u00fcller\\\"...\\n\\nActually the second signature looks like \\\"Dr. Riller\\\" or \\\"A. Rillen\\\"... Hmm. It could be \\\"D. Miller\\\" too.\\n\\nGiven uncertainty, I think the safest is to put the company name as sender, since that's who \\\"wrote\\\" the letter (the letter is signed with the company stamp/name). But the key asks for persons. If signatures are illegible, perhaps null is appropriate, but the company name is clearly the sender entity.\\n\\nThe example shows persons' names. But for a corporate letter, the sender is often the company. I'll include \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" as the sender since it's the entity that wrote the letter.\\n\\nDate: 1926-02-16 (also repeated on pages 2 and 3 as 16.2.26).\\n\\nTitle: null (no title/subject line present).\\n\\nReceiver: Herr Christ - perhaps \\\"Christ\\\" or \\\"Herr Christ\\\". The full name might be just \\\"Christ\\\" as surname. The company is Paravicini, Christ & Co. I'll put \\\"Christ\\\" or \\\"Herr Christ\\\". Probably best: \\\"Christ\\\".\\n\\nLet me format the JSON:\\n\\n{\\n    \\\"letter_title\\\": [\\\"null\\\"],\\n    \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n    \\\"sender_persons\\\": [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\\n\\nWait, instructions say: \\\"If a piece of information is not included in the letter, set the value for the corresponding key to 'null'.\\\" The example shows lists. So for missing info, the value should be \\\"null\\\" - probably as a string in the list, or null itself. The example doesn't show null case. I'll use [\\\"null\\\"] to keep list format, or maybe null. Given \\\"the values corresponding to each key should be stored as a list, even if there is only a single value for a key\\\", I'll use [\\\"null\\\"].\\n\\nActually, \\\"set the value for the corresponding key to 'null'\\\" - this suggests the value is \\\"null\\\" (string). But also values should be lists. I'll go with [\\\"null\\\"] for consistency.\\n\\nFor the sender, should I try to decipher the signatures? The signatures appear to be two: one possibly \\\"Gm\u00fcnder\\\"/\\\"Gmeinder\\\"/\\\"Gunder...\\\" and the other \\\"D. Riller\\\"/\\\"A. Riller\\\"/\\\"Dr. Miller\\\". \\n\\nHistorically, Basler Rheinschiffahrt AG (BRAG) - directors around 1926... I'm not certain of specific names. The signatures are too unclear to confidently transcribe. The typed sender is the company. I'll list the company.\\n\\nActually, let me reconsider: maybe the first signature is \\\"Gundert\\\" or \\\"Gundel\\\"... The second \\\"Dr. Ritter\\\"? Given ambiguity, company name is safest.\\n\\nThe receiver's full name: just \\\"Christ\\\" is given, no first name. So \\\"Christ\\\" or \\\"Herr Christ\\\". I'll use \\\"Christ\\\".\\n\\nFinal answer:\\n{\\n    \\\"letter_title\\\": [\\\"null\\\"],\\n    \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n    \\\"sender_persons\\\": [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"],\\n    \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\",\"format\":\"unknown\",\"index\":0}]},\"native_finish_reason\":\"stop\"}],\"created\":1785002047,\"model\":\"moonshotai/kimi-k3\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"fpv0_0d4e787e\",\"usage\":{\"completion_tokens\":1355,\"prompt_tokens\":12218,\"total_tokens\":13573,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":0,\"reasoning_tokens\":1280,\"rejected_prediction_tokens\":null,\"image_tokens\":0},\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":12032,\"cache_write_tokens\":0,\"video_tokens\":0},\"cost\":0.0244926,\"is_byok\":false,\"cost_details\":{\"upstream_inference_cost\":0.0244926,\"upstream_inference_prompt_cost\":0.0041676,\"upstream_inference_completions_cost\":0.020325}},\"provider\":\"Moonshot AI\"}"}, "scoring": {"f1_macro": 0.62, "f1_micro": 0.57, "cost_summary": {"total_input_tokens": 263709, "total_output_tokens": 78273, "total_tokens": 341982, "input_cost_usd": 0.7904358, "output_cost_usd": 1.174095, "total_cost_usd": 1.9645308}}, "pricing": {"bucket_date": "2026-07-25", "age_days": 0, "input_price_per_million": 3, "output_price_per_million": 15}, "timing": {"total_response_s": 3935.69, "mean_response_s": 122.99, "slowest_response_s": 1244.65, "inputs_timed": 32, "inputs_stored": 32, "span_s": 1215.27}, "normalized_score": 62.0}