{"_id": {"$oid": "69d4eb12b877594bc5718cac"}, "test_id": "T0686", "benchmark": "library_cards", "date": "2026-03-16", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "genai", "model": "gemini-3.1-pro-preview", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"M\u00fchlhaus\",\"first_name\":\"Richard\"},\"publication\":{\"title\":\"Untersuchungen des Blutbildes bei akuten u. chronischen Osteomyelitiden.\",\"year\":\"1927\",\"place\":\"M\u00fcnchen\",\"pages\":\"17\",\"publisher\":\"Wolf\",\"format\":\"8\u00b0\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"U 28. 5030\",\"subjects\":null}}", "model": "gemini-3.1-pro-preview", "provider": "genai", "finish_reason": "STOP", "usage": {"input_tokens": 1529, "output_tokens": 177, "total_tokens": 3011, "input_cost_usd": 0.003058, "output_cost_usd": 0.002124, "estimated_cost_usd": 0.005182}, "duration": 14.158718347549438, "timestamp": "2026-03-16T19:26:35.736558", "parsed": {"type": {"type": "Dissertation or thesis"}, "author": {"last_name": "M\u00fchlhaus", "first_name": "Richard"}, "publication": {"title": "Untersuchungen des Blutbildes bei akuten u. chronischen Osteomyelitiden.", "year": "1927", "place": "M\u00fcnchen", "pages": "17", "publisher": "Wolf", "format": "8\u00b0", "editor": null}, "library_reference": {"shelfmark": "U 28. 5030", "subjects": null}}, "conversation_id": "50df3c4e-16a0-4fa7-a56c-8d6efbb7581a", "score": {"f1_score": 1.0, "precision": 1.0, "recall": 1.0, "true_positives": 10, "false_positives": 0, "false_negatives": 0, "field_scores": {"author.last_name": {"response": "M\u00fchlhaus", "ground_truth": "M\u00fchlhaus", "score": 1.0}, "library_reference.shelfmark": {"response": "U 28. 5030", "ground_truth": "U 28. 5030", "score": 1.0}, "publication.pages": {"response": "17", "ground_truth": "17", "score": 1.0}, "publication.format": {"response": "8\u00b0", "ground_truth": "8\u00b0", "score": 1.0}, "publication.year": {"response": "1927", "ground_truth": 1927, "score": 1.0}, "publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.title": {"response": "Untersuchungen des Blutbildes bei akuten u. chronischen Osteomyelitiden.", "ground_truth": "Untersuchungen des Blutbildes bei akuten u. chronischen Osteomyelitiden", "score": 0.993006993006993}, "publication.publisher": {"response": "Wolf", "ground_truth": "Wolf", "score": 1.0}, "author.first_name": {"response": "Richard", "ground_truth": "Richard", "score": 1.0}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "type.type": {"response": "Dissertation or thesis", "ground_truth": "Dissertation or thesis", "score": 1.0}, "publication.place": {"response": "M\u00fcnchen", "ground_truth": "M\u00fcnchen", "score": 1.0}}, "total_fields": 12}, "raw_response": "{\"sdk_http_response\":{\"headers\":{\"content-type\":\"application/json; charset=UTF-8\",\"vary\":\"Origin, X-Origin, Referer\",\"content-encoding\":\"gzip\",\"date\":\"Mon, 16 Mar 2026 18:26:42 GMT\",\"server\":\"scaffolding on HTTPServer2\",\"x-xss-protection\":\"0\",\"x-frame-options\":\"SAMEORIGIN\",\"x-content-type-options\":\"nosniff\",\"server-timing\":\"gfet4t7; dur=14071\",\"alt-svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\",\"transfer-encoding\":\"chunked\"},\"body\":null},\"candidates\":[{\"content\":{\"parts\":[{\"media_resolution\":null,\"code_execution_result\":null,\"executable_code\":null,\"file_data\":null,\"function_call\":null,\"function_response\":null,\"inline_data\":null,\"text\":\"{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"M\u00fchlhaus\\\",\\n    \\\"first_name\\\": \\\"Richard\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Untersuchungen des Blutbildes bei akuten u. chronischen Osteomyelitiden.\\\",\\n    \\\"year\\\": \\\"1927\\\",\\n    \\\"place\\\": \\\"M\u00fcnchen\\\",\\n    \\\"pages\\\": \\\"17\\\",\\n    \\\"publisher\\\": \\\"Wolf\\\",\\n    \\\"format\\\": \\\"8\u00b0\\\",\\n    \\\"editor\\\": null\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"U 28. 5030\\\",\\n    \\\"subjects\\\": null\\n  }\\n}\",\"thought\":null,\"thought_signature\":\"EpQgCpEgAb4-9vsiFy_dEW0tGfWZB04v8aywRXzVnBjpVmHy5YlxRDUQOXaJwNWBbY6QL5Yp7Jm4ugpmQh_8WRMAOFGdlPdlM2pS-X006zhxKatcr4prCIQRGtIUoZrcevXEnpxE5CUw16YmfZ2fcoFrYDyjZBg0Rfz5mlfASXshXmeTVoSOJmQLbmFGkvFgGPutPTq9YNyuqnzptzloJRMtH9YBpnOo3fZwKcOKJ9qJdvxMzvZP67xW_WzvUW9ghjPmA1GZGzTNZOuIrPLgDxmuAgbyfqDaVecXbKRo4DsYpfJ4HFZLcDaskiJsBAKHxhYNnL1RJ5Ta9hBWdkQ4TQkJ4a3bDFSgQeX0yyWGBAGrVzKUuJDyQQehiG2d83p9nFT-9kgxqLLgONo21v3qU67QDkHD3c0bIBUYPofYv3UHKtD3w0XNiivpZuSP_VRxXrlZXpNjtcyV7jq1iAqZWsVXpxSJV-GWiYTIX37lrYGnSaasDblfttEL5YxORcx-VdwrhCjDN1MyyYrNARBS96uFALKBkG3E6eX9r4mmCt_YdjlkkXXHNAMQdqLGW6n_gGcMoijMqMRuDn8Vrirrxf0KmGAG4ueVdbAPAFYXBImkyxi4snMbhd3orC4SUYfRP2GCSw2_3kSWzu05TxsvX_grp3B31XjMXHmT44uQ9AUKyqaq-cu6zNexCV05QE0Le3Q_E5CUsIF-3yLIQGDtvU2fxImCNIqJcPezdZ45mRXgdGxY75AHziOeBLektakqygLScL_1pORfvaKxTZKPG8a6H476KJNQLtaZlFfAQAkDLa3oP_32DUPrvozBOoKwEHo1cvi1Z863w6CHpNssHu2xmvnTwimh_XgAbrCYZXVOXJwq7JxbMyNErQ4-UYAxbeze0iEhdOqaD68-kCpfHXA4BjiO1dhdrvifMV4GOTUiOwpeCuuECQY3HExp_J1Adf99cDcQsWgaxThxHVt8EBjUVvp4tpAjBthHDs6RveXKyF2V8lhma0ApxY8Tz12E9H5-gNHZcOovQ3xDhxa9_D9rey0kVDMIap-MzOc8KRTJQmi2bxIb02n8v2RDdx92J2alr0LeLjKsKZ0KFr9YiWEsSFehIBDOVzV54Luv7QL4a7DVksTmqTodcxgirJeilP9yaf7fJNDW4kpM4hJG_iFsW7QgQjhWXG_1cdUZuM9JZI0ns8nU33gbL47XoF8Y183HUSwByG6j4YORII8SqOGdfHo-9usN73kHZ_kQVzDURR1w2Blh51ovsEZgG9nJ3Vkxx9oYQaKCrn9Pteh5AXefsloHahJsDTKrjX0wsIAwtdpPMBGyELTDFEeJKMTo3eUPkeZ8aMeOUvTRY75fwdFWw3NopKJv3rxQ5QfyyeBskflA861QzBvVSaodZiwseSnZzK1Ryiy0qH11gcvDgNU11zCmvYOI6iwqOQ_lwps-AwWwbeXJIBvu5kV3Li75P5LkSQMOip2u9YwINIE1XxpoNMZfKI6yt4bDHAO1HWyax16wNQeN3prMyQKAj00guFnwM-QCPzLgqszh46v-vgWSJmq2JzYcx2WuxRGvOiJAa7mqQAKZk-Rq5YLngo4qMdL_x9WZIX_fpEdsJUngHbTIjZhkulKIZ32bxUBGUtHt9CNx9uV2Absqk0fIESX59-pI7N_fbGmnsJdly6j_PyDRNEzl9UGomx3RV3eY64r4g0hYKF6dg6dD0E9RZKzccynVXDX4yqbjoJ-5_dv-d7sOVT8JAhd-SqeixmEHrnoBA8qiQIOR_oM819DZPz0OCnbGukmiVWNug4VMPLWnXssf6jNeysGRxd5P1Wr33hUs1LsuG83aOae8EbVVwC5DQfoo6o6iACMiQAoyBE6ZkAcqpq5a71hANip8OCNBRjbNFYvNYLEz3JLLg4Vum3CzZ93vbRpG8FOrrPwvJWy6MyOKZNu-gtc0DSCPANq1FQCBEuhfseqXIhkuMKXrMAXSAFtG7lKexd3dj1zVuISB802lvHCcr2HuEwUN_lnz5hwVF4VUbN2vlse69At27QGLwFdkIB-1i7wvPzAczMIQ5KhkTJGGnX7iRCJewzfKSwZ9tMeCX7Liz2VQoSQNd5A1enIUECTH8clKq5LJMYXYrmdxtrP86u87ZV3KCQJs_H9tp7HMgzhQviG344MfEVjja7rfGFlDsKeAa1VbUaqRDfgnEno5NHc1XGM2410eY7d-1rWyfvEneV9S5YM4UVlqhj8Jd1o0ggaYDlGnk2e2oZYlcFDB2kO9_gkhQr0orq8kqpdpv7TZgDZRpmOSkdLKssfquEVUv5gzyS6RE7HJ-L_gvfGN5-bSdrU0TUMMbapd-L1PGNYRdV3r09m97t8tiY9JbB7Jngu3auFhGn7Y9jXekUZtw-JiUZRR9iEIOf6tjZlEj8BRx3BFdIZ9gjvqqnCyS1CNx-2NEupHTCEe2q7alR22N1wXUenTIbL8ErmzSjLVAh_5bhvi9iIlo99v0F__tW2KX35za4FvNNuPAEui7Wuj7FKm-SeGqWzwShZxw--sqS5JB2nFg_cYhaQu3QngpJ9OuJcQMJDlp38GA2BErD5k5FFobagaIC3BW_3tLwfQXZkMpU-t89JgV_T3psjsoak7H0XyLGBqxDbtssE_v6jKuIu2rM8FYtyuK3Qdjob6mNYqtVf0qwDs93Z3PKCkSsasFL40ca8Cs8dLXEspXy9XBBHSBrrDPpL__cxueRjx59LnD5HQ3HnMGFxf8ktgulCtsiRlHC5fFVC-dXMJgPQoEcQm_GKpkKupMPL_8b-5FrcZ-RphaVRj7lspKrYNQmrO68jOqFGhh0gCiPq_ap5LnGxh5-jKbjbPa-RkOWqyWjHrZ26ARMfV1WHie2IpZDU4o32uBVHBB84dWrk1yeCyvIsA-pTu_fa6sTS6eJac8iJ8_edSLj_u_ldUFZOqlgHfRg3n7NbjQzxQOmxf7ZObF9KrF7HCAwHyme7fX3R-TEnT0SfEZZbi1doHNAMuqd3ta20QOCuCkihpGLLHZ74o90T5aCY16AC_stwKgpRTDMYAEdZPE99NzW_lsC_ZAdTARlR6Nx32nCDiPiIurXOyCTACMG_D2B2SGQKCRaWmq8niKX9_yy9FZjPPpMcF7CYTRkMB_SHaSotz3GDTHx7_u48V71x4j1UtIhBiAgPhmQLtHBcgqiS7qcoX4sN5BgBjtZ4gdrhO-DBg_cGSG_3eqNgepLzCM8278myykHghOFRP-yQGxM0O2L5EqzQ9H6nsNLyVKyqvk6c4WcmtIkFluFMz3BHTInIowYVVdUh74GdqvOKAmCfcLZxRV2Z6uFLLH8MaAYIzpHiNTe0FNgSoUC-7C9aMl_Gd4IvUlWWzwC6iaERVB8Qn0aKZPfkMoOz_Vt_8Q7mxhFuRkxUKVqbMtUqGQi403b4YEaaEZM7AAVHYpmFYrtCHR9KloNQOP0uvxEVNreQ2nrdoyWzf3EdCYmp9p_4UpmDIv4eLOtntsjwD2SHKHvsj2lhuzpK-wPe8LYRVROC5QvC7QqNCRo5D7y5UMYyqNPe4fKldUqmoBkMcSMTB9PZcOsqqpa8p4QKvkSRW2yR9mDvZKeqJnt_-50Eia2EuFxF7nz-14Uq7aYFVMTiL_OW-Wl4rp26leIRN2bpO88R-aOJMssN5cSdgxOwTdLbG77hNYkFZBPGzJYMJ0d8W_c2zK47jWPwyrj4T5HMSBMrUUe5VURa3pkDLv3mRtuBLLi_js8yxVGpvf4rJ8ZZBUpXWujXxX_Mj4r1IVSvT-0xfYDSW5WNdaj1TH1XXyMF1TDghoLyw0RyNMHRB4JCuf_6dg17o4moLupeP05GmHulqYE1ZtXZOQSi8RzaAfWmkN3YGtkR3IlSGvyG_qfRUFrwhObjGvlPExFcufqtcMbEzpz04Ilp7SAd4-zWT_0K6GUepY8FasdmXlM558-DcO-x4WaHaU3Wx6K-HKQXR9j2eTMyrpCOdGGwcrxdU5vleYvZ0IjJNEguBWLsvmaPW9YNzCx2SzLP0RJwgFzebBb2xJJwrHETas9jQnd0not9fuN0Z3wKDIQ-diamaJLljcSKKGX1WuDFo_Ea3VBSRY-mLo0SR3di1HXakD1GjnsbauLXodYEZGpNvrghL7CSm6Qoqn6AzAEL0VX0RFh_WHfMlg6Imc9JYRlAIZgs7qD2q_AGH50mxJzN4kvzht6KdFkfAsNpBFRk-QFIfWKYgz3iY18UqWV2OXK_bVXJBEI4TRmaz3pfCP4XvhXnSYTYNClzctftm7BSxt5clp5kh6dzDxtXkOsuUoPbFI6zWoROOQ2Ud56APZ-ioMNKM7kBegd8cMXLUy623amYypzwXF5ORjhgL8E0o0j_8nik8V74OwroEqv9G3w9F7KSWP7D2-KulvFLNX5vXz8dW4zqzwrrM6AjaesG2S-HtZ0XGkgGalsrZvbrjKipnuo6gkqiZ0zZVgmkdONrdmxPOYXIQbko1trzXF1KcjvYaB2QUgFUvF5WdIgwvuTOWgHB4jHjMCb5dqYYibJ2h0P_ZQnX_pf1uEysWTDNx3MN7C4E4aK5_hvddpTeP1x2vcKAx--pOnK6rnpWN9XtSWZBAh5Qto2kTI22OKSaMf9MRhwnzH9GU8c6uPKa-59gfra7c4Qb68wEHiRo3fAbI1jyV7OU25prf0AHsKQAsPgQUDpsXcmMFWT49jVC0vydMg_2pceKqLW-1yPIlrhnMC-YawOlnsiCbHUHQkNq4LLL5gwXdi_7Pvqft5cR23oWJpfx7UOO0rAelLNNA9UdcY2p5TCalq2-Zb34_xLWbHexEMgZHt0ArZucLT1G0Z8d1GYhyQFRjmaPYwuMqSJ8D4aRI07MxzGVQ3LEV-kmBWAlG_5u60iFWazo-G96bDG7zuFwfPAF3HdgdMM0Ne-N1UrSZos4Qb2zjdROocB4M1Dy5JeIgjIGlQmGETXb46a5Ry_FgJa5y1R7X5RG0SEoJvSjiCY56sw89xJY24MrS0S1Wsx_LIUQ5EYjhq6BhcLfeZxZ7cyVnvUI0zIHrqXvOo_vqm6kZ_Oyaea6aC3jtEDJhksCnv_SRJKYrPiuH5c1cx3RHjFhuED65e1VKydSm2pjybCQ-Yd_Oyq_desiHwactSi6BqPc4qaTOdvdYQr_BnBLwK_ZpPenVC0y29vnMUWoUbE-QHigyBwN-bpSP2h2YPArHh7AvRb_kEKX9pcRkN2DvCVKXM6sGHvi16lhdwAUo8AGvd7q0NP0zzh4_Mmu1rYkJn2zY7d8p9aiP7phKT70XICMXOEsdpcH11GdQvSUTbIpTJN4OWBSgOPDy1k6H5RHEeV3NZi4Gk6jf6JBxiLFGfx1z9NZJrc_F08N9Gu3uqLWIjN9Q7pKXqMS9VtW7T9BePjgB2UbQ7jOu5wbNCXNE9qhwEpicjncxCOw_bAono-Ul-37QV9NclYCANsREv0r4oDYrO-OhTgG-QVbx2Wq6\",\"video_metadata\":null}],\"role\":\"model\"},\"citation_metadata\":null,\"finish_message\":null,\"token_count\":null,\"finish_reason\":\"STOP\",\"avg_logprobs\":null,\"grounding_metadata\":null,\"index\":0,\"logprobs_result\":null,\"safety_ratings\":null,\"url_context_metadata\":null}],\"create_time\":null,\"model_version\":\"gemini-3.1-pro-preview\",\"prompt_feedback\":null,\"response_id\":\"Yku4abfOIPPMnsEP4Lif4AI\",\"usage_metadata\":{\"cache_tokens_details\":null,\"cached_content_token_count\":null,\"candidates_token_count\":177,\"candidates_tokens_details\":null,\"prompt_token_count\":1529,\"prompt_tokens_details\":[{\"modality\":\"IMAGE\",\"token_count\":1075},{\"modality\":\"TEXT\",\"token_count\":454}],\"thoughts_token_count\":1305,\"tool_use_prompt_token_count\":null,\"tool_use_prompt_tokens_details\":null,\"total_token_count\":3011,\"traffic_type\":null},\"automatic_function_calling_history\":[],\"parsed\":{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"M\u00fchlhaus\",\"first_name\":\"Richard\"},\"publication\":{\"title\":\"Untersuchungen des Blutbildes bei akuten u. chronischen Osteomyelitiden.\",\"year\":\"1927\",\"place\":\"M\u00fcnchen\",\"pages\":\"17\",\"publisher\":\"Wolf\",\"format\":\"8\u00b0\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"U 28. 5030\",\"subjects\":null}}}"}, "scoring": {"f1_micro": 0.89019926799512, "f1_macro": 0.8801140684410647, "micro_precision": 0.8745505393527767, "micro_recall": 0.9064182194616978, "total_instances": 263, "total_tp": 2189, "total_fp": 314, "total_fn": 226, "cost_summary": {"total_input_tokens": 406240, "total_output_tokens": 48633, "total_tokens": 454873, "input_cost_usd": 0.812479999999998, "output_cost_usd": 0.5835960000000002, "total_cost_usd": 1.3960759999999999}}, "normalized_score": 88.01140684410646}