{"_id": {"$oid": "6ac4c7b5164e0f17c0b234b5"}, "test_id": "T1572", "benchmark": "business_letters", "date": "2026-08-15", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "huggingface", "model": "meta-models/Muse-Glimmer-30B:together", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": true, \"skip_non_signatures\": false}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1926-03-29\"],\"letter_title\":null,\"sender_persons\":null,\"receiver_persons\":[\"Dr. W. Krasting\"]}}", "model": "meta-models/Muse-Glimmer-30B", "provider": "huggingface", "finish_reason": "stop", "usage": {"input_tokens": 4186, "output_tokens": 987, "total_tokens": 5173, "cached_tokens": 320, "input_cost_usd": 0.0014650999999999998, "output_cost_usd": 0.0014805, "estimated_cost_usd": 0.0029455999999999996}, "duration": 11.957304954528809, "timestamp": "2026-08-15T16:57:00.574613", "parsed": {"metadata": {"send_date": ["1926-03-29"], "letter_title": null, "sender_persons": null, "receiver_persons": ["Dr. W. Krasting"], "document_number": "letter05"}}, "conversation_id": "abc69b9e-6878-4e04-bcc9-18954b253454", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 0, "sender_persons_fn": 0, "receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0}, "raw_response": "{\"id\":\"owJupuB-fdNmn-a2b90c574808d90d\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1926-03-29\\\"],\\n    \\\"letter_title\\\": null,\\n    \\\"sender_persons\\\": null,\\n    \\\"receiver_persons\\\": [\\\"Dr. W. Krasting\\\"]\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1926-03-29\"],\"letter_title\":null,\"sender_persons\":null,\"receiver_persons\":[\"Dr. W. Krasting\"]}},\"reasoning\":\"IDENTITY and PURPOSE:\\n\\nYou are presented with a series of images constituting a historical letter. Your task is to extract the values of the following keys from the letter and return them in a JSON file where the values corresponding to each key should be stored as a list, even if there is only a single value for a key:\\n\\n- letter_title: Title of the letter.\\n- sender_persons: Name(s) of the person(s) who wrote the letter.\\n- send_date: The exact or approximate date the letter was written.\\n- receiver_persons: Name(s) of the person(s) who received the letter.\\n\\nTake a deep breath and think step by step about how to best accomplish this goal. Map out all the claims and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for dates. If a piece of information is not included in the letter, set the value for the corresponding key to \\\"null\\\". Do not return anything except the JSON file.\\n\\nWe need extract. Letter shows:\\n\\nBasel, den 29. M\u00e4rz 1926.\\n\\nHerrn Dr. W. Krasting,\\nInternationale Ausstellung f\u00fcr Binnenschiffahrt und Wasserkraftnutzung Basel 1926\\nBasel.\\n\\nIch best\u00e4tige Ihnen dankend den Erhalt Ihres Schreibens vom 26. dies und teile Ihnen mit, dass die Besprechungen mit verschiedenen unserer Vorstandsmitglieder wegen einer Betriebsgemeinschaft weiter gef\u00fchrt werden. Ich glaube, dass wir n\u00e4chstens mit einem Vorschlage werden an Sie herantreten k\u00f6nnen.\\n\\nHochachtungsvoll\\n\\nNo signature visible? Maybe sender not signed? The letter ends with Hochachtungsvoll but no name. Might be missing. Could be sender is unknown. So sender_persons null? Or maybe the letter is from someone but signature not in image. The image shows no signature. So sender_persons is null.\\n\\nReceiver persons: Herrn Dr. W. Krasting. So receiver_persons: [\\\"Dr. W. Krasting\\\"] or \\\"Herrn Dr. W. Krasting\\\". Probably name is Dr. W. Krasting.\\n\\nLetter title? No title. Maybe null.\\n\\nSend date: Basel, den 29. M\u00e4rz 1926. So 1926-03-29.\\n\\nLetter title maybe null.\\n\\nSender persons: null (no signature). Could be maybe the sender is the Internationale Ausstellung? No.\\n\\nThe letter is addressed to Dr. W. Krasting. The sender is unknown. Could be the letter is from the organization? No name.\\n\\nThus output JSON with lists.\\n\\nletter_title: null? The instruction says set value to \\\"null\\\". But values should be stored as a list, even if single value. If not included, set value to \\\"null\\\". Does that mean the list contains null? Or the value is null not a list? Example shows list. Hmm. It says return them in a JSON file where the values corresponding to each key should be stored as a list, even if there is only a single value for a key. If a piece of information is not included in the letter, set the value for the corresponding key to \\\"null\\\". Conflict: list vs null. Probably set the value to null, not a list containing null. Might be they want \\\"letter_title\\\": null. But they said values should be stored as a list. Might be they want [\\\"null\\\"]? Hmm.\\n\\nInterpretation: If not included, set the value for the corresponding key to \\\"null\\\". So the value is null, not a list. Might be exception. I'll follow instruction: set to null. But also values should be list. Could do \\\"letter_title\\\": null. Might be acceptable.\\n\\nBetter to make list with null? Could be [\\\"null\\\"]? No.\\n\\nI'll assume if missing, set to null (not list). Might be okay.\\n\\nSender_persons: null.\\n\\nReceiver_persons: [\\\"Dr. W. Krasting\\\"] or [\\\"Herrn Dr. W. Krasting\\\"].\\n\\nProbably name is Dr. W. Krasting.\\n\\nLetter_title: null.\\n\\nSend_date: [\\\"1926-03-29\\\"].\\n\\nReturn JSON.\\n\\nLet's output:\\n\\n{\\n  \\\"letter_title\\\": null,\\n  \\\"sender_persons\\\": null,\\n  \\\"send_date\\\": [\\\"1926-03-29\\\"],\\n  \\\"receiver_persons\\\": [\\\"Dr. W. Krasting\\\"]\\n}\\n\\nIf they want list for null, maybe [\\\"null\\\"]? Hmm.\\n\\nAlternative: letter_title: []? No.\\n\\nI'll go with null.\\n\\nEnsure JSON only.\\n\\n\"},\"token_ids\":null,\"routed_experts\":null,\"seed\":null}],\"created\":1786805810,\"model\":\"meta-models/Muse-Glimmer-30B\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"vllm-0.1.dev19059+g66826a22d-tp2-46311a71\",\"usage\":{\"completion_tokens\":987,\"prompt_tokens\":4186,\"total_tokens\":5173,\"completion_tokens_details\":null,\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":320,\"created_cache_tokens\":3856,\"multimodal_tokens\":{\"image\":3850}}},\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null,\"prompt\":[]}"}, "scoring": {"f1_macro": 0.78, "f1_micro": 0.88, "cost_summary": {"total_input_tokens": 143150, "total_output_tokens": 30608, "total_tokens": 173758, "input_cost_usd": 0.050102499999999967, "output_cost_usd": 0.045911999999999994, "total_cost_usd": 0.09601449999999997}}, "pricing": {"bucket_date": "2026-08-15", "age_days": 0, "input_price_per_million": 0.35, "output_price_per_million": 1.5}, "timing": {"total_response_s": 341.46, "mean_response_s": 13.66, "slowest_response_s": 19.72, "inputs_timed": 25, "inputs_stored": 25, "span_s": 103.18}, "normalized_score": 78.0}