{"_id": {"$oid": "6a1010f996d87b97b14cf8f5"}, "test_id": "T1061", "benchmark": "personnel_cards", "date": "2026-05-22", "tags": {"document-type": ["index-card"], "writing": ["handwritten", "typed", "printed"], "century": [20], "layout": ["table", "form"], "task": ["transcription", "document-understanding", "data-correction"], "language": ["de", "fr"]}, "contributors": ["tabea_wullschleger", "maximilian_hindermann"], "hidden": false, "config": {"provider": "genai", "model": "gemini-3.5-flash", "dataclass": "Table", "temperature": 0.0, "role_description": "You are a historian specializing in 20th century personnel records and administrative documents", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract all information from this personnel card table and return it as a structured JSON object. The card contains rows documenting employment history with positions, locations, salary information, dates, and remarks.\n\nREQUIRED JSON STRUCTURE:\n```json\n{\n  \"rows\": [\n    {\n      \"row_number\": 1,\n      \"dienstliche_stellung\": {\n        \"diplomatic_transcript\": \"exact text from card\",\n        \"interpretation\": \"expanded/standardized form or null\",\n        \"is_crossed_out\": false\n      },\n      \"dienstort\": {\n        \"diplomatic_transcript\": \"exact text from card\",\n        \"interpretation\": \"expanded/standardized form or null\",\n        \"is_crossed_out\": false\n      },\n      \"gehaltsklasse\": {\n        \"diplomatic_transcript\": \"exact text from card\",\n        \"interpretation\": \"expanded/standardized form or null\",\n        \"is_crossed_out\": false\n      },\n      \"jahresgehalt_monatsgehalt_taglohn\": {\n        \"diplomatic_transcript\": \"exact text from card\",\n        \"interpretation\": \"standardized numeric form or null\",\n        \"is_crossed_out\": false\n      },\n      \"datum_gehalts\u00e4nderung\": {\n        \"diplomatic_transcript\": \"exact text from card\",\n        \"interpretation\": \"YYYY-MM-DD format or null\",\n        \"is_crossed_out\": false\n      },\n      \"bemerkungen\": {\n        \"diplomatic_transcript\": \"exact text from card\",\n        \"interpretation\": \"expanded/standardized form or null\",\n        \"is_crossed_out\": false\n      }\n    }\n  ]\n}\n```\n\nCOLUMN DEFINITIONS:\n1. **dienstliche_stellung**: Official position or job title (e.g., \"Assistent\", \"Professor\", \"Sekret\u00e4r\")\n2. **dienstort**: Place of service or work location (e.g., \"Basel\", \"Z\u00fcrich\")\n3. **gehaltsklasse**: Salary class or grade (e.g., \"III\", \"IV\", roman numerals or numbers)\n4. **jahresgehalt_monatsgehalt_taglohn**: Salary amount (annual/monthly/daily wage)\n5. **datum_gehalts\u00e4nderung**: Date of salary change or effective date\n6. **bemerkungen**: Remarks, notes, or additional comments\n\nFIELD EXTRACTION RULES:\n\n**diplomatic_transcript** (REQUIRED for all fields):\n- Transcribe EXACTLY as written on the card\n- Include all abbreviations, punctuation, and formatting as they appear\n- Preserve original capitalization and spacing\n- Include currency symbols and separators (e.g., \"Fr. 2'400.-\", \"3.700.-\")\n- For dates, copy the exact format (e.g., \"1. Jan. 1946\", \"1.April 1945\")\n- Use empty string \"\" for empty cells\n- Be sure to escape ditto marks (repetition marks such as `\"` indicating \"same as above\")\n- DO NOT expand abbreviations or standardize formats\n- Do not transcribe checkmarks\n\n**interpretation** (OPTIONAL - use null if not applicable):\n- For ditto marks (repetition marks such as `\"`): Replace with the actual repeated value from the previous row in the same column\n  - For example, if previous row's dienstliche_stellung is \"Hilfsleiterin\" and current row has `\"`, interpret as \"Hilfsleiterin\"\n  - Do NOT add explanatory text like \"wie oben\" or similar - just provide the repeated value\n- Never expand abbreviations\n- For dates: Convert to ISO format YYYY-MM-DD\n  - \"1. Jan. 1946\" \u2192 \"1946-01-01\"\n  - \"1.April 1945\" \u2192 \"1945-04-01\"\n- For salary amounts: Extract numeric value only (remove currency symbols, separators)\n  - \"Fr. 2'400.-\" \u2192 \"2400\"\n  - \"3.700.-\" \u2192 \"3700\"\n  - \"5.094.-\" \u2192 \"5094\"\n  \"6,30.-\" \u2192 \"6.3\"\n- For salary class: Convert roman numerals to arabic if clear\n  - \"III\" \u2192 \"3\"\n  - \"IV\" \u2192 \"4\"\n- Do not convert roman numerals to arabic if not salary, date, or salary class\n- Use null if no interpretation/expansion is needed or if the field is empty\n- \"+ {word}\" where {word} is a word does not need interpretation\n\n**is_crossed_out** (REQUIRED for all fields):\n- Set to true if text in this field is crossed out, struck through, or deleted\n- Set to false if text is normal (not crossed out)\n- Empty cells should have is_crossed_out: false\n\nROW HANDLING:\n- Number rows sequentially starting from 1 (top to bottom)\n- Include ALL rows that have ANY content in ANY column\n- Empty rows (all cells empty) should be omitted\n- A row with only one filled cell should still be included\n\nIMPORTANT NOTES:\n- Return ONLY the JSON object, no additional text or explanation\n- Every field must have all three sub-fields: diplomatic_transcript, interpretation, is_crossed_out\n- diplomatic_transcript must never be null (use empty string \"\" for empty cells)\n- interpretation can be null when no expansion/standardization applies\n- Process the entire table from top to bottom\n- Maintain consistent row numbering throughout\n", "results": {"text": "{\"rows\":[{\"row_number\":1,\"dienstliche_stellung\":{\"diplomatic_transcript\":\"Koch\",\"interpretation\":null,\"is_crossed_out\":false},\"dienstort\":{\"diplomatic_transcript\":\"\",\"interpretation\":null,\"is_crossed_out\":false},\"gehaltsklasse\":{\"diplomatic_transcript\":\"\",\"interpretation\":null,\"is_crossed_out\":false},\"jahresgehalt_monatsgehalt_taglohn\":{\"diplomatic_transcript\":\"4600.-\",\"interpretation\":\"4600\",\"is_crossed_out\":false},\"datum_gehalts\u00e4nderung\":{\"diplomatic_transcript\":\"3. Dez. 45\",\"interpretation\":\"1945-12-03\",\"is_crossed_out\":false},\"bemerkungen\":{\"diplomatic_transcript\":\"+ Zulagen\",\"interpretation\":null,\"is_crossed_out\":false}}]}", "model": "gemini-3.5-flash", "provider": "genai", "finish_reason": "STOP", "usage": {"input_tokens": 2347, "output_tokens": 296, "total_tokens": 3306, "input_cost_usd": 0.0035205, "output_cost_usd": 0.0026639999999999997, "estimated_cost_usd": 0.0061845}, "duration": 5.790379524230957, "timestamp": "2026-05-22T09:33:37.273996", "parsed": {"rows": [{"row_number": 1, "dienstliche_stellung": {"diplomatic_transcript": "Koch", "interpretation": null, "is_crossed_out": false}, "dienstort": {"diplomatic_transcript": "", "interpretation": null, "is_crossed_out": false}, "gehaltsklasse": {"diplomatic_transcript": "", "interpretation": null, "is_crossed_out": false}, "jahresgehalt_monatsgehalt_taglohn": {"diplomatic_transcript": "4600.-", "interpretation": "4600", "is_crossed_out": false}, "datum_gehalts\u00e4nderung": {"diplomatic_transcript": "3. Dez. 45", "interpretation": "1945-12-03", "is_crossed_out": false}, "bemerkungen": {"diplomatic_transcript": "+ Zulagen", "interpretation": null, "is_crossed_out": false}}]}, "conversation_id": "03cd755d-25c2-41f5-965f-ef54ec367620", "score": {"f1_score": 1.0, "precision": 1.0, "recall": 1.0, "true_positives": 12, "false_positives": 0, "false_negatives": 0, "field_scores": {"rows[0].gehaltsklasse.is_crossed_out": {"response": false, "ground_truth": false, "score": 1.0}, "rows[0].bemerkungen.diplomatic_transcript": {"response": "+ Zulagen", "ground_truth": "+ Zulagen", "score": 1.0}, "rows[0].dienstort.diplomatic_transcript": {"response": null, "ground_truth": null, "score": 1.0}, "rows[0].dienstliche_stellung.interpretation": {"response": null, "ground_truth": null, "score": 1.0}, "rows[0].datum_gehalts\u00e4nderung.is_crossed_out": {"response": false, "ground_truth": false, "score": 1.0}, "rows[0].datum_gehalts\u00e4nderung.interpretation": {"response": "1945-12-03", "ground_truth": "1945-12-03", "score": 1.0}, "rows[0].dienstliche_stellung.diplomatic_transcript": {"response": "Koch", "ground_truth": "Koch", "score": 1.0}, "rows[0].gehaltsklasse.interpretation": {"response": null, "ground_truth": null, "score": 1.0}, "rows[0].jahresgehalt_monatsgehalt_taglohn.is_crossed_out": {"response": false, "ground_truth": false, "score": 1.0}, "rows[0].jahresgehalt_monatsgehalt_taglohn.interpretation": {"response": "4600", "ground_truth": "4600", "score": 1.0}, "rows[0].bemerkungen.interpretation": {"response": null, "ground_truth": null, "score": 1.0}, "rows[0].dienstort.interpretation": {"response": null, "ground_truth": null, "score": 1.0}, "rows[0].gehaltsklasse.diplomatic_transcript": {"response": null, "ground_truth": null, "score": 1.0}, "rows[0].dienstort.is_crossed_out": {"response": false, "ground_truth": false, "score": 1.0}, "rows[0].datum_gehalts\u00e4nderung.diplomatic_transcript": {"response": "3. Dez. 45", "ground_truth": "3. Dez. 45", "score": 1.0}, "rows[0].jahresgehalt_monatsgehalt_taglohn.diplomatic_transcript": {"response": "4600.-", "ground_truth": "4600.-", "score": 1.0}, "rows[0].bemerkungen.is_crossed_out": {"response": false, "ground_truth": false, "score": 1.0}, "rows[0].dienstliche_stellung.is_crossed_out": {"response": false, "ground_truth": false, "score": 1.0}}, "total_fields": 19}, "raw_response": "{\"sdk_http_response\":{\"headers\":{\"x-gemini-service-tier\":\"standard\",\"content-type\":\"application/json; charset=UTF-8\",\"vary\":\"Origin, X-Origin, Referer\",\"content-encoding\":\"gzip\",\"date\":\"Fri, 22 May 2026 07:33:37 GMT\",\"server\":\"scaffolding on HTTPServer2\",\"x-xss-protection\":\"0\",\"x-frame-options\":\"SAMEORIGIN\",\"x-content-type-options\":\"nosniff\",\"server-timing\":\"gfet4t7; dur=5409\",\"alt-svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\",\"transfer-encoding\":\"chunked\"},\"body\":null},\"candidates\":[{\"content\":{\"parts\":[{\"media_resolution\":null,\"code_execution_result\":null,\"executable_code\":null,\"file_data\":null,\"function_call\":null,\"function_response\":null,\"inline_data\":null,\"text\":\"{\\n  \\\"rows\\\": [\\n    {\\n      \\\"row_number\\\": 1,\\n      \\\"dienstliche_stellung\\\": {\\n        \\\"diplomatic_transcript\\\": \\\"Koch\\\",\\n        \\\"interpretation\\\": null,\\n        \\\"is_crossed_out\\\": false\\n      },\\n      \\\"dienstort\\\": {\\n        \\\"diplomatic_transcript\\\": \\\"\\\",\\n        \\\"interpretation\\\": null,\\n        \\\"is_crossed_out\\\": false\\n      },\\n      \\\"gehaltsklasse\\\": {\\n        \\\"diplomatic_transcript\\\": \\\"\\\",\\n        \\\"interpretation\\\": null,\\n        \\\"is_crossed_out\\\": false\\n      },\\n      \\\"jahresgehalt_monatsgehalt_taglohn\\\": {\\n        \\\"diplomatic_transcript\\\": \\\"4600.-\\\",\\n        \\\"interpretation\\\": \\\"4600\\\",\\n        \\\"is_crossed_out\\\": false\\n      },\\n      \\\"datum_gehalts\u00e4nderung\\\": {\\n        \\\"diplomatic_transcript\\\": \\\"3. Dez. 45\\\",\\n        \\\"interpretation\\\": \\\"1945-12-03\\\",\\n        \\\"is_crossed_out\\\": false\\n      },\\n      \\\"bemerkungen\\\": {\\n        \\\"diplomatic_transcript\\\": \\\"+ Zulagen\\\",\\n        \\\"interpretation\\\": null,\\n        \\\"is_crossed_out\\\": false\\n      }\\n    }\\n  ]\\n}\",\"thought\":null,\"thought_signature\":\"ErARCq0RAQw51scGTsRRzHxL560VIV-AKfaPR22NH9uuPCAKOm5qmbUGJz36qu-g1peNWlBuLHYFh1tlQ_FBqGnHnIvsPQN8-5jsn92ywVOHx_YW6t8w9CM8h398zowIYhJmsSMuHe7MfIv16t9YCrHvLMdIZygxabFacHPt-aSeq0n7b5sqf8TyO2WZIxsregeJLUgVa1Ie9GGVFZQYHdqYYHOt29eeuf0aEkiPQCABTT3lRZY7ML4XxMLsW3jbCGdT8Tu0iI_f9Yw4jfo6sWOi087pCXK-OXAFtFhJdBxv6oEyqeuFt2n5Hhv80H09kTiQNK4LkRQOGUm_M4SPgccOC1MriIr_ScZiZccVo1_MO9XjNmBwG-Cm5ISOlMHVcujt6j-6cGvfIYv1t_eq8oTXxjcdHsO4XNs62pXrYbSwO2hNyVxW2roPAotE1FAJEAvV1mT_d_f-r8ROriSTW6MlDtLWMvqBDMrYrcy80x8C5OzJOenx6xIxRizAp52vdnUd4MHUaF04GolkdAGjORH3XLa2FcoeN0Bzmpu0a40K9iaicyWVUM__O5H9BUd_Vpr-TcYdY0xs9EOFOWheZH9dX-PCZyixR6YKJT5lJA0CB5NuO005v8_Z2-T9AAvi3CXCaem5IfWJ1XaSzm9uat3rCTpwLvXw-mT9YYasHzmaF5ZQsucJj1qBewDCrE3U1hrb0UrcWIINMdq_m7iBinROp84_BvFYPbYH2eUNgNgiUdtas5Kq8mXHLI3FktK3HAnZtrgmRGN0vtbxmGCqXxzS5uy1TcIMtUzizkZ0h9YbbAIr5B9JR7-Iny13GhPe3dxi9HBnTelim6IGoM-ihyN85GQh1hNaPEpfVSoSNJwgZdNxfNmuXJM5tHrw2jkU7HF8PwyOf1DkztVtbzLcl1PibUcTG2NZLbAkxMXOMSK3l01PyWxWxrYe1UhVqwhIEFjvpHthLMbvHY4qtSOb4kaEntsKOGE6Ej5WXFPzUtHj2L8WylpM22zuR8MHPwzUGAZUvIIv_RqvTnbEcDiTBiMHsfpjbuTUog1gtonjfK2rqP0IKkZrtBy6SPCnCsmrEKrzIxBu_2QH6Kmke8bgO9r33p3F7-EbbOgPVAUdBqWmikPbh7Y_Sn1UqseFJT5T9D3Od4nut0ArAaneI8yeXc4YJ4Si5mWnA8NkOsHOYySArn_oo5ammpMlQueHzO512hnLb5DPxA_C91jhIUNUtdUZTapCkhhSHhfGCQeMFBcPpPLG8jOz_og4_J-AiVCTxuaPm-80sUVziulLUGk-zYCxq8XAIvO1BPhM0HnEOHwiuy65FquCYk6HAmxNRae8LV_G-5FL5KrLKW079N0sMNhB3YXa385j5-p4IOqSyKCgEtJguZnlFA1edI3jPLUr4imphcjnK3b0867gWz_jLk4YPfU9WJMfAx9MdPX5vJ9xOnz0MRKBOh-U5sqk926J6fKAW1gDUeoA7cMB-1QG-UdB728cETWhN8_cywfRv27XPrLoiDg6Ya5Z0tkX-McQp0TjW7NLoCl_smZI25x3HTeIF_KSHOemIOf4QIsh2aEOFgrmNnx8dPjJCMStP3_Fm4pZxO5SYnYtSJOyoF6hj9Uxu0ZuAZC6kcz-jnIV0qHoz-a2dim9rYxh18IUIKLKL9Tunc2WofAY6ScOtpXLGxpgpCVVJxrLgPrwQwzen8AVfHy0HMU7qSX9mpY5ClpfN7bSRk6Efhz4i3AGJcTSYEeAtjV_Ku6h83hbMPzPyiBUUSVmiJGMbyI9oN0YOD_ctuu3J0GpUrBMHqGVwilF2VMAajJ9I3O84T2dD2pVzGPEhTvlEyG6pSn120EQMz_WUfHmPdN089l0HgxO0qQFKUpQNSx_2u6qJFY3T7BOFV7eQYRh-_w26pBJZMEDBpYA2iwEJ9SFklV91UnvtRaGGrPWuVtPS0Z0T2eeVCs130eDU0QGyGys4Xc5A95eu2xpg3H51V6ha8y9w8nrWM-txg5UJYBnY3-EFzmjW_4bQWcndXvVOO-J9FM-l95Bue7hu3o0EsKW1NxN63DC4TBOPFAhu8h_6GF9i4-4swctwG_EmUfAEMKzgn8dIsbloHqELbs7krRAbtzQEHJUhGJeSGam1R2u-U77uRwvROF_L8xFBe1qkc1SJX1tKUqcoPBHqnOAG7FGO4KxcWz6w4X7l5d155_ZEeAHxEiueDvsW1aPY3BJkojF1lBMfuijAlO7hQzNwYVQGY5CahfjrWRxc-IGgMZ5hyK6LbZE_RTKhcvrHK_-tWeAjGs46ATo7PTwlWKLnmpnCpPrUNWdCPzWSf5YdSYLi8MUJ7hvT3H6RqUNx35p32YZh2ykLaWHfehwMRQLMDoOF0UTv2z_vsF3DpTQE7IGJ1cNDCQHX8ujG9KTbcy8hJJ-4PUXuqNurTPhvG3PjBbhIQSV4RUPxRUL5Pw0g9k_WbkAGngd8ACyBIY4h697qgboCCmKP0DxMS6ncbfVNxe37HgrFmPsv1tPIKmDH1TndB1xeCeODK4y_MklAk53LliQg2LTh-cc7pyVqCkkqCejyRvCzKXPKMmbmUqwECMag32Pc4-Squ93e0x-mA-jcrUqpYqV3nZ9fshtab16jLYornElmCGCgqekoniYyqmzES77EUwIpGjR18lcaEzStX7gz4fhfttom-fZjjR6qQ9Q32kdMv-wi69AIp1p3x5XWIqJgWUVRhWqvHrx3Xo9SlroDF5N0NVQThakXiaJmrE35Xd1kqpUqe-cbKZXVP2l6s25r7AUTqlkKbHTOHw-Rek-_bDlSeBfGuXaWg_Ww1zyaWrsQwd3MyqE3AEDtNFIvyUULFLDcslBQfi2DP-6vALoPRdkUyVpjj_eGldwmcl5-qA2OPVc1DGfziraWCJWsvBtHTxPX55-Qn9F4O3Mm1BpfTUKy5RZmPGfakgwmhk-ewgIDKqKQU96mHuc9g==\",\"video_metadata\":null}],\"role\":\"model\"},\"citation_metadata\":null,\"finish_message\":null,\"token_count\":null,\"finish_reason\":\"STOP\",\"grounding_metadata\":null,\"avg_logprobs\":null,\"index\":0,\"logprobs_result\":null,\"safety_ratings\":null,\"url_context_metadata\":null}],\"create_time\":null,\"model_version\":\"gemini-3.5-flash\",\"prompt_feedback\":null,\"response_id\":\"zAYQapCyCOqkkdUP867KwQk\",\"usage_metadata\":{\"cache_tokens_details\":null,\"cached_content_token_count\":null,\"candidates_token_count\":296,\"candidates_tokens_details\":null,\"prompt_token_count\":2347,\"prompt_tokens_details\":[{\"modality\":\"TEXT\",\"token_count\":1267},{\"modality\":\"IMAGE\",\"token_count\":1080}],\"thoughts_token_count\":663,\"tool_use_prompt_token_count\":null,\"tool_use_prompt_tokens_details\":null,\"total_token_count\":3306,\"traffic_type\":null},\"automatic_function_calling_history\":[],\"parsed\":{\"rows\":[{\"row_number\":1,\"dienstliche_stellung\":{\"diplomatic_transcript\":\"Koch\",\"interpretation\":null,\"is_crossed_out\":false},\"dienstort\":{\"diplomatic_transcript\":\"\",\"interpretation\":null,\"is_crossed_out\":false},\"gehaltsklasse\":{\"diplomatic_transcript\":\"\",\"interpretation\":null,\"is_crossed_out\":false},\"jahresgehalt_monatsgehalt_taglohn\":{\"diplomatic_transcript\":\"4600.-\",\"interpretation\":\"4600\",\"is_crossed_out\":false},\"datum_gehalts\u00e4nderung\":{\"diplomatic_transcript\":\"3. Dez. 45\",\"interpretation\":\"1945-12-03\",\"is_crossed_out\":false},\"bemerkungen\":{\"diplomatic_transcript\":\"+ Zulagen\",\"interpretation\":null,\"is_crossed_out\":false}}]}}"}, "scoring": {"f1_micro": 0.9713721618953602, "f1_macro": 0.9083606557377049, "micro_precision": 0.9658421672555948, "micro_recall": 0.9769658459094519, "total_instances": 61, "total_tp": 2460, "total_fp": 87, "total_fn": 58, "cost_summary": {"total_input_tokens": 143605, "total_output_tokens": 61166, "total_tokens": 204771, "input_cost_usd": 0.21540750000000009, "output_cost_usd": 0.550494, "total_cost_usd": 0.7659015000000001}}, "normalized_score": 97.13721618953602}