{"_id": {"$oid": "6ac4c7b6164e0f17c0b2393c"}, "test_id": "T1782", "benchmark": "business_letters", "date": "2026-09-18", "tags": {"document-type": ["letter"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "layout": ["prose"], "entry-type": ["person", "location"], "task": ["information-extraction"]}, "contributors": ["eric_decker", "maximilian_hindermann", "lea_kasper", "anthea_alberto", "pema_frick", "jose_luis_losada_palenzuela", "sorin_marti", "elena_spadini"], "hidden": false, "config": {"provider": "openrouter", "model": "qwen/qwen3.8-max-0902", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge and an expert in the field of 20th century Swiss history", "prompt_file": "prompt.txt", "rules": "{\"skip_signatures\": true, \"skip_non_signatures\": false}", "legacy_test": false}, "prompt": "            IDENTITY and PURPOSE:\n\n            You are presented with a series of images constituting a historical letter. Your task is to extract the\n            values of the following keys from the letter and return them in a JSON file where the values\n            corresponding to each key should be stored as a list, even if there is only a single value for a\n            key:\n\n            - letter_title: Title of the letter.\n            - sender_persons: Name(s) of the person(s) who wrote the letter.\n            - send_date: The exact or approximate date the letter was written.\n            - receiver_persons: Name(s) of the person(s) who received the letter.\n\n            Take a deep breath and think step by step about how to best accomplish this goal. Map out all the claims\n            and implications on a virtual whiteboard in your mind. Do not use OCR. Use the ISO format YYYY-MM-DD for\n            dates. If a piece of information is not included in the letter, set the value for the corresponding key\n            to \"null\". Do not return anything except the JSON file.\n\n            EXAMPLE:\n\n            {\n                \"letter_title\": [\"Petition for Environmental Protection\"],\n                \"send_date\": [\"1993-03-12\"],\n                \"sender_persons\": [\"Lisa Simpson\"],\n                \"receiver_persons\": [\"Mayor Joe Quimby\", \"Seymour Skinner\"],\n            }\n\n            OUTPUT:", "results": {"text": "{\"metadata\":{\"send_date\":[\"1926-03-29\"],\"letter_title\":[\"Best\u00e4tigung des Schreibenhalts und Weiterf\u00fchrung der Besprechungen wegen einer Betriebsgemeinschaft\"],\"sender_persons\":null,\"receiver_persons\":[\"Dr. W. Krasting\"]}}", "model": "qwen/qwen3.8-max-0902", "provider": "openrouter", "finish_reason": "stop", "usage": {"input_tokens": 3127, "output_tokens": 513, "total_tokens": 3640, "cached_tokens": 3072, "input_cost_usd": 0.006254, "output_cost_usd": 0.003078, "estimated_cost_usd": 0.009332}, "duration": 11.605019330978394, "timestamp": "2026-09-18T09:54:04.372046", "parsed": {"metadata": {"send_date": ["1926-03-29"], "letter_title": ["Best\u00e4tigung des Schreibenhalts und Weiterf\u00fchrung der Besprechungen wegen einer Betriebsgemeinschaft"], "sender_persons": null, "receiver_persons": ["Dr. W. Krasting"]}}, "conversation_id": "3ddb5e88-643e-4ad5-bc72-8d99f07d1525", "score": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0, "sender_persons_tp": 0, "sender_persons_fp": 0, "sender_persons_fn": 0, "receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0, "field_scores": {"send_date": {"response": ["1926-03-29"], "ground_truth": ["1926-03-29"], "score": null, "counts": {"send_date_tp": 1, "send_date_fp": 0, "send_date_fn": 0}}, "sender_persons": {"response": [], "ground_truth": [], "score": null, "counts": {"sender_persons_tp": 0, "sender_persons_fp": 0, "sender_persons_fn": 0}}, "receiver_persons": {"response": ["Krasting, Wilhelm"], "ground_truth": ["Krasting, Wilhelm"], "score": null, "counts": {"receiver_persons_tp": 1, "receiver_persons_fp": 0, "receiver_persons_fn": 0}}}}, "raw_response": "{\"id\":\"gen-1789718033-ebDXYpsMraj3vLYSfsfo\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\n      \\\"1926-03-29\\\"\\n    ],\\n    \\\"letter_title\\\": [\\n      \\\"Best\u00e4tigung des Schreibenhalts und Weiterf\u00fchrung der Besprechungen wegen einer Betriebsgemeinschaft\\\"\\n    ],\\n    \\\"sender_persons\\\": null,\\n    \\\"receiver_persons\\\": [\\n      \\\"Dr. W. Krasting\\\"\\n    ]\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"metadata\":{\"send_date\":[\"1926-03-29\"],\"letter_title\":[\"Best\u00e4tigung des Schreibenhalts und Weiterf\u00fchrung der Besprechungen wegen einer Betriebsgemeinschaft\"],\"sender_persons\":null,\"receiver_persons\":[\"Dr. W. Krasting\"]}},\"reasoning\":\"The letter is dated Basel, den 29. M\u00e4rz 1926 (Basel, March 29, 1926). The addressee is Herrn Dr. W. Krasting, Internationale Ausstellung f\u00fcr Binnenschiffahrt und Wasserkraftnutzung Basel 1926, Basel. The sender is not explicitly named (no signature visible; only \\\"Hochachtungsvoll\\\" with no name). Therefore sender_persons should be null. Letter title: none explicitly stated. Perhaps no title. The content is a confirmation of receipt of the letter dated the 26th and a notice that discussions regarding a Betriebsgemeinschaft (operating consortium) will continue. Title: null? The instructions say letter_title: title of the letter. If not present, null.\\n\\nSend date: 1926-03-29.\\n\\nReceiver: Dr. W. Krasting.\\n\\nSender: null (no signature name). Possibly inferred from context? The letter mentions \\\"unserer Vorstandsmitglieder\\\" (our board members), but no name is given. Therefore null.\\n\\nLetter title: null.\\n\\nThus JSON:\\n\\n{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1926-03-29\\\"],\\n    \\\"letter_title\\\": null,\\n    \\\"sender_persons\\\": null,\\n    \\\"receiver_persons\\\": [\\\"Dr. W. Krasting\\\"]\\n  }\\n}\\n\\nShould the receiver include the organization? The schema asks for person names. \\\"Herrn Dr. W. Krasting\\\" is the person. Possibly include as \\\"Dr. W. Krasting\\\".\\n\\nLetter title: perhaps could be given as something like \\\"Best\u00e4tigung des Erhalt eines Schreibens\\\" but not explicitly stated. Usually in datasets like this, the title is derived from the subject line; here there is no subject line. Therefore null.\\n\\nOutput only JSON conforming to the schema: an object with \\\"metadata\\\" containing the four keys.\\n\",\"reasoning_details\":[{\"type\":\"reasoning.text\",\"text\":\"The letter is dated Basel, den 29. M\u00e4rz 1926 (Basel, March 29, 1926). The addressee is Herrn Dr. W. Krasting, Internationale Ausstellung f\u00fcr Binnenschiffahrt und Wasserkraftnutzung Basel 1926, Basel. The sender is not explicitly named (no signature visible; only \\\"Hochachtungsvoll\\\" with no name). Therefore sender_persons should be null. Letter title: none explicitly stated. Perhaps no title. The content is a confirmation of receipt of the letter dated the 26th and a notice that discussions regarding a Betriebsgemeinschaft (operating consortium) will continue. Title: null? The instructions say letter_title: title of the letter. If not present, null.\\n\\nSend date: 1926-03-29.\\n\\nReceiver: Dr. W. Krasting.\\n\\nSender: null (no signature name). Possibly inferred from context? The letter mentions \\\"unserer Vorstandsmitglieder\\\" (our board members), but no name is given. Therefore null.\\n\\nLetter title: null.\\n\\nThus JSON:\\n\\n{\\n  \\\"metadata\\\": {\\n    \\\"send_date\\\": [\\\"1926-03-29\\\"],\\n    \\\"letter_title\\\": null,\\n    \\\"sender_persons\\\": null,\\n    \\\"receiver_persons\\\": [\\\"Dr. W. Krasting\\\"]\\n  }\\n}\\n\\nShould the receiver include the organization? The schema asks for person names. \\\"Herrn Dr. W. Krasting\\\" is the person. Possibly include as \\\"Dr. W. Krasting\\\".\\n\\nLetter title: perhaps could be given as something like \\\"Best\u00e4tigung des Erhalt eines Schreibens\\\" but not explicitly stated. Usually in datasets like this, the title is derived from the subject line; here there is no subject line. Therefore null.\\n\\nOutput only JSON conforming to the schema: an object with \\\"metadata\\\" containing the four keys.\\n\",\"format\":\"unknown\",\"index\":0}]},\"native_finish_reason\":\"stop\"}],\"created\":1789718033,\"model\":\"qwen/qwen3.8-max-0902\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":513,\"prompt_tokens\":3127,\"total_tokens\":3640,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":0,\"reasoning_tokens\":414,\"rejected_prediction_tokens\":null,\"image_tokens\":0},\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":3072,\"cache_write_tokens\":0,\"video_tokens\":0},\"cost\":0.003956,\"is_byok\":false,\"cost_details\":{\"upstream_inference_cost\":0.003956,\"upstream_inference_prompt_cost\":0.000878,\"upstream_inference_completions_cost\":0.003078}},\"provider\":\"Alibaba\"}"}, "scoring": {"f1_macro": 0.53, "f1_micro": 0.63, "cost_summary": {"total_input_tokens": 97746, "total_output_tokens": 74915, "total_tokens": 172661, "input_cost_usd": 0.19549200000000005, "output_cost_usd": 0.44948999999999995, "total_cost_usd": 0.6449819999999999}}, "pricing": {"bucket_date": "2026-09-16", "age_days": 2, "input_price_per_million": 2.0, "output_price_per_million": 6.0}, "timing": {"total_response_s": 1516.48, "mean_response_s": 63.19, "slowest_response_s": 176.58, "inputs_timed": 24, "inputs_stored": 24, "span_s": 217.88}, "normalized_score": 53.0}