{"_id": {"$oid": "6ac4c7b5164e0f17c0b234e7"}, "test_id": "T1571", "benchmark": "business_letters", "date": "2026-08-15", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "huggingface", "model": "meta-models/Muse-Glimmer-30B:together", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1926-02-16\"],\"letter_title\":[\"null\"],\"sender_persons\":[\"null\"],\"receiver_persons\":[\"Christ\"]}}", "model": "meta-models/Muse-Glimmer-30B", "provider": "huggingface", "finish_reason": "stop", "usage": {"input_tokens": 11886, "output_tokens": 913, "total_tokens": 12799, "cached_tokens": 320, "input_cost_usd": 0.0041601, "output_cost_usd": 0.0013695, "estimated_cost_usd": 0.0055296}, "duration": 14.080541133880615, "timestamp": "2026-08-15T16:29:26.001820", "parsed": {"metadata": {"send_date": ["1926-02-16"], "letter_title": ["null"], "sender_persons": ["null"], "receiver_persons": ["Christ"], "document_number": "letter01"}}, "conversation_id": "b1d46605-8c4f-4315-8065-dbabd86bf187", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 0, "sender_persons_fn": 2, "receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"owJmLUQ-fdNmn-a2b8e3e61ed2d7aa\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n    \\\"letter_title\\\": [ \\\"null\\\" ],\\n    \\\"sender_persons\\\": [ \\\"null\\\" ],\\n    \\\"receiver_persons\\\": [ \\\"Christ\\\" ]\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1926-02-16\"],\"letter_title\":[\"null\"],\"sender_persons\":[\"null\"],\"receiver_persons\":[\"Christ\"]}},\"reasoning\":\"IDENTITY and PURPOSE: extract values of keys from letter and return JSON.\\n\\nKeys:\\n- letter_title: Title of the letter.\\n- sender_persons: Name(s) of person(s) who wrote the letter.\\n- send_date: exact or approximate date letter was written.\\n- receiver_persons: Name(s) of person(s) who received the letter.\\n\\nValues as list even if single.\\n\\nFrom images: first page header \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\" Markt... Basel, den 16. Februar 1926.\\n\\nReceiver: Herrn Christ, i/Fa. Paravicini, Christ & Co., Basel. Kaufhausgasse 7.\\n\\nSender? The letter is from Basler Rheinschiffahrt-Aktiengesellschaft. Signatures at bottom page 3: \\\"hochachtend BASLER RHEINSCHIFFAHRT-AKTIENGESELLSCHAFT\\\" with two signatures. Names not legible? Might be? The signatures look like \\\"G... \\\" and \\\"A. M\u00fcller\\\"? Could be A. M\u00fcller. Might be not clear. Could be sender persons is the company? The key says Name(s) of the person(s) who wrote the letter. Might be the company as sender? Or persons signing. Might be null if not identifiable. The signatures are illegible. Could be \\\"G... \\\" and \\\"A. M\u00fcller\\\". Might be better to set sender_persons to null? Or the company name? The example uses persons. The letter is from Basler Rheinschiffahrt-Aktiengesellschaft. Might be sender is the company, but persons? Could be no person name given. The letter is signed by two persons but names not readable. Might be null.\\n\\nLetter title? Maybe no title. Could be null. Or the company name is header. Might be no title. Could be null.\\n\\nSend date: Basel, den 16. Februar 1926. Also page 2 and 3 show 16.2.26. So ISO 1926-02-16.\\n\\nReceiver persons: Herrn Christ. Possibly \\\"Christ\\\" is the person. The address is i/Fa. Paravicini, Christ & Co. So receiver is Herrn Christ. Might be just Christ. Could be \\\"Christ\\\". Might include full? The letter is addressed to Herrn Christ, i/Fa. Paravicini, Christ & Co., Basel. Kaufhausgasse 7. So receiver person is Christ. Possibly also the company? But key is persons. So [\\\"Christ\\\"]? Might be \\\"Herrn Christ\\\". Could be \\\"Christ\\\". I'll use \\\"Christ\\\". Or \\\"Herr Christ\\\". Might be better to keep as \\\"Christ\\\". Could also include \\\"Paravicini, Christ & Co.\\\" but that's a company.\\n\\nLetter title: maybe no title. Could be null.\\n\\nSender persons: maybe null because signatures illegible. Could be the company name as sender? The instruction says Name(s) of the person(s) who wrote the letter. The letter is from Basler Rheinschiffahrt-Aktiengesellschaft. No person name clearly printed. Signatures are illegible. So set null.\\n\\nAlternatively, could infer sender persons as the company? Might be considered an entity not a person. I'd set null.\\n\\nReturn JSON with keys in order? Not required.\\n\\nOutput JSON:\\n\\n{\\n  \\\"letter_title\\\": [null],\\n  \\\"sender_persons\\\": [null],\\n  \\\"send_date\\\": [\\\"1926-02-16\\\"],\\n  \\\"receiver_persons\\\": [\\\"Christ\\\"]\\n}\\n\\nMaybe receiver persons list includes \\\"Herrn Christ\\\". Could be \\\"Christ\\\". Might also include \\\"Paravicini, Christ & Co.\\\" but that's firm.\\n\\nMaybe letter_title could be \\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"? That's the letterhead. Might be considered title? Hmm.\\n\\nThe example letter_title is \\\"Petition for Environmental Protection\\\". Here no explicit title. Could be null.\\n\\nI'll output as above.\\n\\nIf want sender persons as company? Could be [\\\"Basler Rheinschiffahrt-Aktiengesellschaft\\\"]. But it's not a person.\\n\\nI'll assume null.\\n\\nReturn JSON only.\\n\\n\"},\"token_ids\":null,\"routed_experts\":null,\"seed\":null}],\"created\":1786804153,\"model\":\"meta-models/Muse-Glimmer-30B\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"vllm-0.1.dev19059+g66826a22d-tp2-46311a71\",\"usage\":{\"completion_tokens\":913,\"prompt_tokens\":11886,\"total_tokens\":12799,\"completion_tokens_details\":null,\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":320,\"created_cache_tokens\":11552,\"multimodal_tokens\":{\"image\":11550}}},\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null,\"prompt\":[]}"}, "scoring": {"f1_macro": 0.69, "f1_micro": 0.73, "cost_summary": {"total_input_tokens": 396690, "total_output_tokens": 76278, "total_tokens": 472968, "input_cost_usd": 0.13884149999999992, "output_cost_usd": 0.11441700000000003, "total_cost_usd": 0.2532585}}, "pricing": {"bucket_date": "2026-08-15", "age_days": 0, "input_price_per_million": 0.35, "output_price_per_million": 1.5}, "timing": {"total_response_s": 3406.24, "mean_response_s": 59.76, "slowest_response_s": 1474.62, "inputs_timed": 57, "inputs_stored": 57, "span_s": 1642.37}, "normalized_score": 69.0}