{"_id": {"$oid": "6ac4c7b5164e0f17c0b234cc"}, "test_id": "T1573", "benchmark": "business_letters", "date": "2026-08-15", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "huggingface", "model": "meta-models/Muse-Glimmer-30B:together", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": false, \"skip_non_signatures\": true}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1926-02-16\"],\"letter_title\":null,\"sender_persons\":null,\"receiver_persons\":[\"Christ\"]}}", "model": "meta-models/Muse-Glimmer-30B", "provider": "huggingface", "finish_reason": "stop", "usage": {"input_tokens": 11886, "output_tokens": 1563, "total_tokens": 13449, "cached_tokens": 11872, "input_cost_usd": 0.0041601, "output_cost_usd": 0.0023445, "estimated_cost_usd": 0.006504599999999999}, "duration": 17.250179529190063, "timestamp": "2026-08-15T16:59:01.279735", "parsed": {"metadata": {"send_date": ["1926-02-16"], "letter_title": null, "sender_persons": null, "receiver_persons": ["Christ"], "document_number": "letter01"}}, "conversation_id": "1536c35f-bd2a-413c-ae30-3557635d3294", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 0, "sender_persons_fn": 2, "receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"owJvRH3-fdNmn-a2b90f2a1e80f6f9\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n    \\\"letter_title\\\": null,\\n    \\\"sender_persons\\\": null,\\n    \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1926-02-16\"],\"letter_title\":null,\"sender_persons\":null,\"receiver_persons\":[\"Christ\"]}},\"reasoning\":\"IDENTITY and PURPOSE:\\n\\nWe need extract values of keys from letter and return JSON where values corresponding to each key should be stored as a list, even if single value.\\n\\nKeys:\\n- letter_title: Title of the letter.\\n- sender_persons: Name(s) of the person(s) who wrote the letter.\\n- send_date: The exact or approximate date the letter was written.\\n- receiver_persons: Name(s) of the person(s) who received the letter.\\n\\nTake deep breath... Do not use OCR. Use ISO format YYYY-MM-DD for dates. If info not included, set value to \\\"null\\\". Do not return anything except JSON.\\n\\nWe have three images. First page: Basler Rheinschiffahrt-Aktiengesellschaft, Basel, den 16. Februar 1926. Address to Herrn Christ, i/Fa. Paravicini, Christ & Co., Basel. Kaufhausgasse 7.\\n\\nLetter title? Maybe no explicit title. Could be null? Or maybe the company name is title? The example letter_title is \\\"Petition for Environmental Protection\\\". In this letter, maybe no title. Could be null. Or maybe the letter is just a business letter, no title. Might be null.\\n\\nSender persons: The letter is from Basler Rheinschiffahrt-Aktiengesellschaft. Signatures at bottom page 3: two signatures, maybe names? Signatures are illegible. Could be \\\"Gys...?\\\" and \\\"A. Miller\\\"? The signature looks like \\\"A. M\u00fcller\\\"? Hard to read. Might be not extractable. The sender is the company. Sender persons: maybe the signatories. But names not legible. Could be null? The letter is from the company, but persons who wrote? Might be the company as sender. The key is sender_persons: Name(s) of the person(s) who wrote the letter. Signatures are present but not readable. Could be null.\\n\\nAlternatively, maybe the sender is Basler Rheinschiffahrt-Aktiengesellschaft. But that's an organization, not a person. Might be null.\\n\\nReceiver persons: Herrn Christ, i/Fa. Paravicini, Christ & Co. The address is Herrn Christ. So receiver person is Christ. Possibly \\\"Christ\\\". Might be \\\"Herrn Christ\\\". Could be \\\"Christ\\\". The letter is addressed to Herrn Christ, i/Fa. Paravicini, Christ & Co., Basel. Kaufhausgasse 7. So receiver persons: [\\\"Christ\\\"]? Or \\\"Herrn Christ\\\". Might include full name? No first name given. So just Christ.\\n\\nSend date: Basel, den 16. Februar 1926. Also page 2 and 3 show 16.2.26. So date is 1926-02-16.\\n\\nLetter title: maybe null. Could be the company letterhead? Might be no title. Could be empty list? The instruction says if info not included set value to \\\"null\\\". So letter_title: null? But value should be stored as a list, even if single value. Hmm conflict: \\\"return them in a JSON file where the values corresponding to each key should be stored as a list, even if there is only a single value for a key\\\". And \\\"If a piece of information is not included in the letter, set the value for the corresponding key to 'null'.\\\" So if null, is it a list containing null? Or value is null. Example shows list. Probably if null, the value is null not a list? Might be [\\\"null\\\"]? Hmm.\\n\\nInterpretation: The value for each key should be a list. If info not included, set the value to \\\"null\\\". Could mean the list contains null? Or the value is null. Might be better to set to null (not a list). But instruction says values should be stored as a list. Might be contradictory. Could do \\\"letter_title\\\": null. But that is not a list. Hmm.\\n\\nMaybe letter_title is the company name? Or the subject? The letter has no explicit title. Could be null.\\n\\nSender persons: The signatures are illegible, but maybe the letter is signed by the company, not persons. Could be null.\\n\\nReceiver persons: \\\"Christ\\\". Could be \\\"Herrn Christ\\\". Might be \\\"Christ\\\". Let's output [\\\"Christ\\\"].\\n\\nSend date: [\\\"1926-02-16\\\"].\\n\\nLetter title: maybe null. Could be [\\\"null\\\"]? Might be better to output null.\\n\\nLet's see typical extraction tasks. Usually they'd want letter_title as null if no title. Might output \\\"letter_title\\\": null. But they said values should be stored as a list. Hmm.\\n\\nMaybe the title is \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"? That's the letterhead. Could be considered title? Might be the sender.\\n\\nAlternative interpretation: letter_title could be the subject line? No subject.\\n\\nMaybe the letter title is not present, so null.\\n\\nSender persons: The letter is signed by two persons, but names not legible. Could be null.\\n\\nReceiver persons: The letter is addressed to Herrn Christ. So receiver persons is Christ. Could also include Paravicini, Christ & Co. But that's a company. The person is Christ.\\n\\nMaybe also copy to Herrn Ryniker, Schiffbau-Ingenieur Basel, Herrn Dr. Krasting, Basel. But that's copy, not receiver.\\n\\nSo output JSON:\\n\\n{\\n  \\\"letter_title\\\": null,\\n  \\\"sender_persons\\\": null,\\n  \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n  \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\\n\\nBut sender_persons as list? Could be []? Hmm.\\n\\nMaybe sender persons is the company? Could be \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\". But that's not a person.\\n\\nCould be the signatures are \\\"Gys...?\\\" and \\\"A. M\u00fcller\\\". The second signature looks like \\\"A. M\u00fcller\\\". Could be \\\"A. M\u00fcller\\\". First signature maybe \\\"Gys...\\\". Hard to read.\\n\\nGiven instruction \\\"Do not use OCR.\\\" So we cannot read signatures. So sender persons unknown. So null.\\n\\nLetter title unknown. Null.\\n\\nReceiver persons: \\\"Christ\\\". Could be \\\"Herrn Christ\\\". Let's use \\\"Christ\\\".\\n\\nMaybe also include \\\"Paravicini, Christ & Co.\\\"? That's a firm.\\n\\nI'll output with lists for all, and null as null not list? Might be safer to output list with one element null? The example shows list of strings. If null, maybe set to null.\\n\\nI'll assume they want value as list, but if null, the value is null (not a list). Might be acceptable.\\n\\nAlternative: set letter_title: []? No.\\n\\nI'll output:\\n\\n{\\n  \\\"letter_title\\\": null,\\n  \\\"sender_persons\\\": null,\\n  \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n  \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\\n\\nIf they strictly want list, could be:\\n\\n\\\"letter_title\\\": [],\\n\\\"sender_persons\\\": [],\\n...\\n\\nBut instruction says set value to \\\"null\\\". I'll go with null.\\n\\nMaybe sender_persons could be [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"]? But that's not a person.\\n\\nHmm.\\n\\nCould be the sender person is the company representative, but name not legible, so null.\\n\\nOk final.\\n\\nLet's output JSON.\"},\"token_ids\":null,\"routed_experts\":null,\"seed\":null}],\"created\":1786805925,\"model\":\"meta-models/Muse-Glimmer-30B\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"vllm-0.1.dev19059+g66826a22d-tp2-46311a71\",\"usage\":{\"completion_tokens\":1563,\"prompt_tokens\":11886,\"total_tokens\":13449,\"completion_tokens_details\":null,\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":11872,\"created_cache_tokens\":0,\"multimodal_tokens\":{\"image\":11550}}},\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null,\"prompt\":[]}"}, "scoring": {"f1_macro": 0.68, "f1_micro": 0.65, "cost_summary": {"total_input_tokens": 253540, "total_output_tokens": 44932, "total_tokens": 298472, "input_cost_usd": 0.08873899999999996, "output_cost_usd": 0.06739800000000001, "total_cost_usd": 0.15613699999999997}}, "pricing": {"bucket_date": "2026-08-15", "age_days": 0, "input_price_per_million": 0.35, "output_price_per_million": 1.5}, "timing": {"total_response_s": 1106.39, "mean_response_s": 34.57, "slowest_response_s": 623.41, "inputs_timed": 32, "inputs_stored": 32, "span_s": 702.48}, "normalized_score": 68.0}