{"_id": {"$oid": "6ab1687f1481417a2aae4dca"}, "test_id": "T1425", "benchmark": "library_cards", "date": "2026-07-25", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "genai", "model": "gemini-3.6-flash", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Reference\"},\"author\":{\"last_name\":\"Montaghemi\",\"first_name\":\"Nourhomadeh\"},\"publication\":{\"title\":\"Sonnenbichler - Montaghemi, Nourhomadeh\",\"year\":\"\",\"place\":\"\",\"pages\":\"\",\"publisher\":\"\",\"format\":\"\",\"editor\":\"\"},\"library_reference\":{\"shelfmark\":\"\",\"subjects\":\"\"}}", "model": "gemini-3.6-flash", "provider": "genai", "finish_reason": "STOP", "usage": {"input_tokens": 1529, "output_tokens": 70, "total_tokens": 2941, "input_cost_usd": 0.0022935, "output_cost_usd": 0.000525, "estimated_cost_usd": 0.012883499999999999, "reasoning_tokens": 1342, "reasoning_cost_usd": 0.010065}, "duration": 7.554077625274658, "timestamp": "2026-07-25T19:45:37.967428", "parsed": {"type": {"type": "Reference"}, "author": {"last_name": "Montaghemi", "first_name": "Nourhomadeh"}, "publication": {"title": "Sonnenbichler - Montaghemi, Nourhomadeh", "year": "", "place": "", "pages": "", "publisher": "", "format": "", "editor": ""}, "library_reference": {"shelfmark": "", "subjects": ""}}, "conversation_id": "792ff144-ad9a-4c5d-8d9d-c9eecb45c0ae", "score": {"f1_score": 0.29, "precision": 0.25, "recall": 0.3333333333333333, "true_positives": 1, "false_positives": 3, "false_negatives": 2, "field_scores": {"publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.pages": {"response": null, "ground_truth": null, "score": 1.0}, "publication.year": {"response": null, "ground_truth": null, "score": 1.0}, "publication.place": {"response": null, "ground_truth": null, "score": 1.0}, "author.first_name": {"response": "Nourhomadeh", "ground_truth": null, "score": 0.0}, "publication.format": {"response": null, "ground_truth": null, "score": 1.0}, "publication.publisher": {"response": null, "ground_truth": null, "score": 1.0}, "library_reference.shelfmark": {"response": null, "ground_truth": null, "score": 1.0}, "publication.title": {"response": "Sonnenbichler - Montaghemi, Nourhomadeh", "ground_truth": "Montaghemi, Nourhomadeh", "score": 0.7419354838709676}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "type.type": {"response": "Reference", "ground_truth": "Reference", "score": 1.0}, "author.last_name": {"response": "Montaghemi", "ground_truth": "Sonnenbichler", "score": 0.34782608695652173}}, "total_fields": 12}, "raw_response": "{\"sdk_http_response\":{\"headers\":{\"x-gemini-service-tier\":\"standard\",\"content-type\":\"application/json; charset=UTF-8\",\"vary\":\"Origin, X-Origin, Referer\",\"content-encoding\":\"gzip\",\"date\":\"Sat, 25 Jul 2026 17:45:38 GMT\",\"server\":\"scaffolding on HTTPServer2\",\"x-xss-protection\":\"0\",\"x-frame-options\":\"SAMEORIGIN\",\"x-content-type-options\":\"nosniff\",\"server-timing\":\"gfet4t7; dur=7471\",\"alt-svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\",\"transfer-encoding\":\"chunked\"},\"body\":null},\"candidates\":[{\"content\":{\"parts\":[{\"media_resolution\":null,\"code_execution_result\":null,\"executable_code\":null,\"file_data\":null,\"function_call\":null,\"function_response\":null,\"inline_data\":null,\"text\":\"{\\\"type\\\":{\\\"type\\\":\\\"Reference\\\"},\\\"author\\\":{\\\"last_name\\\":\\\"Montaghemi\\\",\\\"first_name\\\":\\\"Nourhomadeh\\\"},\\\"publication\\\":{\\\"title\\\":\\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\",\\\"year\\\":\\\"\\\",\\\"place\\\":\\\"\\\",\\\"pages\\\":\\\"\\\",\\\"publisher\\\":\\\"\\\",\\\"format\\\":\\\"\\\",\\\"editor\\\":\\\"\\\"},\\\"library_reference\\\":{\\\"shelfmark\\\":\\\"\\\",\\\"subjects\\\":\\\"\\\"}}\",\"thought\":null,\"thought_signature\":\"EqAlCp0lARFNMg_8ppXx6s74t2S7ZvK5WCmmwYBNHV9wwj04QaSYooKm0nq1IB9-ipwoTrpmTak1DAIHqV-vj9xx98AkJBEzfhsXHjG1Q7HAtUFmvH91bAA_JBlmDOd3sB3onMXbDXwYAaaMA0K-K0FfQ2i_MDpL1AfL5V-wO9ZGkqqyHmIw0Oqg33sGfCaXfY3ogeKprV4AmgGnWQU_T3lEXPi3aI1iLCYrb_Lm5i1qHkOfbI5mxprU_esBKSGDqtZrBgzt_NJf0c8iW-X75q7RqQXDh5sp-tkbb2m-oBOwx44LTb0G_SUa_WXxlklg2u4fv0DrRojBkjj3HJ0NzIv5xubXdSTHa2DWeaLYK0qvYsWGtF8G5eQQ1IYDBtS41-22nuKJLWFj-0QmoP_wEqt_gIJxIqc80DKhfC-UPrrS7kZabI_gpMloZgnJdlRP-IuTwrg0b_qpBCa9QK3FYHv86Tc9juBISQxpx1YqEqZRpUNp6O_2EIHj7ii9qbHpyL3riB45F8x2YHT1-ubFXDxmYOlXxzSsQaVHzPCkZSvPxdZyElwjaEn14Di_Si2sdQiENhGzo5L8W0C1-eOqH3qdpF85DDe8wIw0Swihplyld_99prVmi-0VKEoc5FzilLMcQGlJa5csDfh3jfn2YmhKPLO3s3KQyz3za9IDF3DtyiAPT8UDWD6ZLIttr1ph_ouSKK9UxLEeBDko2MhSwy8xp3cH0LLM9aNZUEE4aiird6snPDOD1tpTdrrIyzKG9mwThQnQP4--Wj2-9iyXPDTBhjLeiVjHDHGRXyoneyaMIdNHdipmTTIQmu783slAlalbwRJj1jVEt8u2WPqFkLQgrepfs62e9ujbX39geWWpJ1ddofDpXtiflLMqvNyxmhtOiKKUVsvFV_IXkAShTLOoCic4exwoh_qAIUHZ2RWWmUHyVmBc19TPkLByZO3eDo3cbsL-WPQtP_bYNH0SG1QbmySAXip-NJos696IX_1I5hHMEjaUq_yWufZVfyvJivIdaIu73Kd8AYjDtcXXp6T3AM20XPwPtcUHhkd8mzsV6kO4US1tmNB8mHxH6130qnu8mlfCmvyKUdAb7bOCzI4-WqPzdz9fj8Bs1ImBwJi2ce9oZNqEIGYp6tv-uhNQ1dS_xk36LQGW68JqMDBk_ZDwI58gPJVmtMdhdQzPQHF6JZrkfkD6NyWTdOlZh9D5W9pn7nitaW2VBNflHuWuBRhEZiPzlIZ6v8O-1lSsk-2xZrDsFwazz4NXqZk-8UXoDVDBGbK7Z50_H0VyepvQ2xIiaECYOa5-yr-NYEVuhaq2Toxf5aRCwtmApS-OBIOpGWaPAPqKiLaG8vFbth1-6S5SQC--_VLFh2nCfFvIKq4ntUq4_79w7Nrf7tUsDgijtWxwyhjSJdinwNyL4Yp8-DgNA_uMfYq8LdDAZbVtQDNfBD17SPvKJM8fFvbQxh0cpO1aqvgPRW3c66HUIH3WqgSVFIPJh8qeeXtV0LAizjamtdP0d7OPgu9scr587FSPd8g-cO_cVj7sDwyBSfddEluPqKlubp1-UaULz6tVWNcSo4lVl_NjTiF1AdXGhIX-lmcfyDAjScpdtAp2tqJG0scXS0VNdxtwDMjinmr1OUaHsSUYn_QFhaG5eOzuK6j4-3YNQ6uN4N1_6nbeHbro_6BPpmPv2zmAsZxRqE6Wz9Q2WZXq3-FKLGFGRBz15e4Eou40lSoBTEDli-ewedR0d7gLZH11C2NcTkW3CyDRGoMLBv5XI9nDq3IV9uZ8bbeyQI5SzCsLRI_Y7WQm0xgorzcguKRh5ZH6QCNYRrj_M7fj7VwEqX_6MvJvZRldiV6LP3MNEMgsGAmkPDAstr5Ik4V00759ohC4pz-fLeVJAi7Gv9ybzd4RFK91HhGxqDGiX72Wetld5Z_Hgs3tcsK_isvc_1ZrQiygah9Qej2zK-uvnDgCYSiMQTcYlAbH6F2_x_iuNrwMDOknsBhv6O1NjmpfTcDuBYLeJ1lMxaWFplxefKJS5TjgBMHuoKGpXtQLqdv7v2C0JNzGK9qUqXs1GuZVT8X0ar-kK9U8pOxmsAdU8MleaAbCCoK4ZFlLVx10h6y1z4gKniFqSLfDB1BmnkXz7tc3CqsMv2Zi6pnUpYiQPyW52lJfGnwjmmdcQ42b-xcq1zPfx78D7wMY5p1YjAHkM0R0k12G5A55CX-ZXOSPoJsnZX6tmwF4JSicFgsqaeEVYV7bExHv3VrxX4U6ZZgFl77OcOZ1zFuPIVsPWDnasGqFnLgfuXZ4rs6_zCdbjhmVtTq-1XYh8q6syC1v3WnT2k0iTdIYvfE4CyhVco9DgcrcOdDEIZRed7poOdcK1YPe_AA1aPq-XYZoFsJLXy18ysONQ8kSeZGi8rcCLcREADQGJAHTmuiCqd8zUYXTt94lKOtaXJjI1d9VkAoOFE0EJeyEtR0VCaGhkAqoAwdERPIi6lzwei9XKNuYKnMJpSv7cDr96QmkTT6usbCWWOSno6gVRMMazCdnechBiQ3Jy3m2-my1lmEjQ67owFv3A4CT7r3yA9qBW6Mvavzp7wuqPA-pUhqhc9qnmGSOOeBgS0FzZaDWtJYyZzzJ6US_i4JmPCu-v0XOvovIShqMuIbB3JADxgWpBTSnJnWEgEAn-6sxjcY_84hcJ48TRjPZXVKkmvPEjOYlrpPVqZMZpjxnsBq8iQi9pauV7SuyNEbo5pgmayVMLeytJ8xfD-zD41htoBp-E6TGQMJRkqTrvWR4B8ZpklegVY1hU0RDTD4-QB71Bqdm3x6Lz5EGpS54z-Kau1D2rZMpzyh6bJ3tdSAvv1Ccn7v3x9JWfeABcWPD0Zt_0w4yHS3nyOod8JeMJ4YA5lTZRvG3BWQaBbt-zeBzzbJFAfMkgcSNa7RjIpyT3q04o_eth7R2Z4lHvcuk0vA2PTFi5-mGcCElHVeyN9XpmdqNxXOnsS2zVXBVGEmRHpKv_5xHDyNUXqbEdAirUw2v_tUnZm7FDzjiFzyZQtSclZJEcr_FgYmcQb2VlWSufx0Z90ynljv4IuFfOEnW0cD-JvA9uo0GlsB1AuaG3t-KHAqy6jylSPVPBU-XYcbRMefkZyHiPKCd6IJQvkhP1Lev_D8NH3TS1W_Cpx-Zsn_PdGHdZWCC83Uu-wuYg9zfVSRmmU2ZJxf65l_09SnvLmy-NZUU_cePuPDLbDDYrXFtOLeJ2j8BSaPVwq7KUe7CgdMax8JxA21I1f7LjDCH0BDd6ctGYucFiASc20ucvANGiboU32D5hUCXd75rXhC0kN_i3yty_4q3x9sL3lK0Y_trD7zzEOQStMJGTyGWX6-Bo5Wk8BeJAA_T1S8kGQatsYbP0l1oGjBACepnDWOO7xafkClXp-htnJ37qeH11Qat327cBi9NpYnOHpAja7KZ1TBJgjeoLrKycYxyBSJUY_6pshE3u54CWq6iPvZWizPlvA7AUCGwyd-uT92-4T_Bl9gOrhpo2GS4c-JwUlNdI9XMSgEHolS1ewXwLpde0Xx47fAfowaOpxtCpJC-xFaKKwZPXTIekfJwwh7Pxq0HtVvsvC8X5WKhMYfDSlf-SnE2ROh4QzIsT4gexpQpDqpQrRAkYR0aJWru8O-YqEmvRmyO7XVVsbrE2bq0ENlTeZJ_5y7PKEgf-Q-sFv9EyyLEQxwyF6w_i3d8vl1q76clz7buZPZRqPRokec7eXA07HFAiP2RYJr0xnPAC1g1tAM7dPzEFPjxhs9tvMGba4cqNSpuOpa-HU5wGxQc4BMYwh9Zy4LtkzLXPK8NEKKxiRB8FEmMjInIZobLKIGqZLbo37qlt-CFL3zm93X5a16vJ0YUwJ8GOYQ319GrWOB7Q9MU7rMeYrTJ9-9f6cL7TxjrhKQUvo0nTbJCrGeQ1izn_1mKdckypNPO2PWTZgnK3bTzXAP2W0p75Yu8dbbjdMcYl-WP71hczOViua4_i0Lka-QRqM4ZYMHFAdtaQg6MGkCnMfAB-hJ6qmwl15dws1nAuUbjyoO8UyvT9kA9WP4nCDuEkqLKSFKo-3zIhB8FR_TYUeNp_n5htk8vE-1wN675dDKldSjDEnkJJ8LTLp4a5i1pCE9Zfpg3Z2yNYaVqLmVN0HJ67RErblNaPsdSeDbZ1Iip6DIUXla9H_317g0_bTyfNR5JHvyH-2t66CJdYiITzM4D-tiCAA9nmyr4rIfLLZXYfvJv1YcFQ_NZKc3D3OnA3P_T4tMmALYxKYBFxs5VrLjLmw_4zLTjyDL_XEo8DDrP3iTzNQVHCtdiOxao9k6_gfJbIgU_-LynTOLZ2FGOltMDiac_Z0aUENuFsV0do_3NjPZUP_Syl2SWRcEY30ozXO2khF8aRLuJa-SkY-1_KGtaREjHLTjKxtaQdsCFj9l5FnyuBZQoLIQyOTpsH3weGHZV6B6TuPTfSEV6F-xmGfjlV8ZAT_41R6ybXT3RrxCNuChYgF9l-s34dnMI6WmLcdaPubkJ7cMNP6xzwGdynDZrXTtWMDc7PMPzruvx9HCh-0QNCNh4KsmUViPCZ8ROeusArvoWJbegmAQzpdVeMJAMNxZxL1TiszsIt_Xqhx6Hg0keo5TsaknOOm49HTK4ETe2FgaI9imdD1FPOwACVdDBV4Z0PMZrwYznK-XrTejIPi_JyPh6hVpqqyIBIHEquetMWn5Coq1MRIma_SMywH6SPlIThod9dpwTd1fS3gBoQzfa0iVIkE-ZtYBTj73IbMX9Ah6CeYsog7PNIKmGfGq20pXHfxnA0_-D-2FxJF5IPFJ_Y918yyTSQ-HFCDkt8TuahoAx3_3N8rJOcfdgnHhDannZzI8uA02D5_n1mPJZG7YIv62Zc2LVj6kiZOebI-n2agSFRc0kNB3IJOP-F5T1hHgWIRPU-g8uvVjCTfwajJ6IZBP7HcuRXRLM4Xh9tH5RzOQHdtbZWAkFHXGl8Nhtiyg2GcKksi1Jwjb2hM4B38k9qe1zE2sCrgN-j9yOfRfRnc-CR2GEzZ_FTtalKdu-fnwdC9Cr7nyJQTXqfmXEUCZ0DLc_4j4DJFwrGbyE7m4jsXCCTOD9IF53d-Tju-4jH39hBXoyYVUcyAQUs_qAB30DCYOx7e-OdByADSNUsJzRFkYylDhXFQZJUL6oJmhlV4-yTxeNsxVQJM8AzRFi72n8rav1eW5Lix2O9seYA59tylbaqivQVY6-AJyRPS6s3P-Itq-eFpAfUxC9zzAuRr0WbWWv6nihnM_xXttjCnvI_HSlePyY_u2yN5UmRy2nsJgZIehIVts7fcS5gfOKE6hBg7oyPHcgObrqAUaFkY1DtigVoBM7AF56klHAIemUd4k8SKFJHQ7s4nPBVMtlebfY2X8P832tfO7UUIfi1Tr-FfL3Gz-JgEG0jOURn5J77ObkjmNZqIZIQLVv7NwIoX_OdLkizez1timfp5TK-KhkP6sOtslMdEj_8t1CB4GmhBXE6hPN-tHGfMgWjFLHl26R83rOclo2XRMtiE3w88rElkgW0sTd5k9rVmMF-ucaBiSDmVid5pdaad-NGT0sccyqX44ixFRWWw7ykbfosIzeXNk068beOePpFnCPJ62qoh3aNANtBzhrZjL37hbkD6O72AnBbLlZv1-Mgvobdv_7xMhT4DjRwHPFScMK_gRaGhEcpd-sNE4BX4rDGPti8vRotmh0xRgX_QHYRTr_7LGyzxFBwjrvpM87_j_bSG3L8klfgQ-Enb3QqCGHDVfzJuZE2GX02OeKnQrAoK-Uub76n-OrLB29Zg-iUZ_ndNEmXLOKknKvDyE___KS6kGhW27bbJ8mYeDOJp6Qks-O-fOcN8L3NuveOu18pfLzUFg9TNG3waGlZU16vRtOCF4pvRMbs_VeWAQQeb7O_zXX0FYDSgP3JMvDmwrMNNAxws-F9Qq7lWgzlXUrxwkbmS_RbRsWxdgTJ71EuBxerv-k-p6hcWUyg-Ol1USqlsfjwxleA6D-nspqypFDwoY2WVAzyZSol-qU_knLN4jlTVxLKclvisiLV5t70SIUP20FlQxkjr_W9BObho4yJ15U0i_QgRwLY71gYvlWj0jmrPqpDZUZfrHGNvh2xQCHrpNCIawXkhBBE0LIKodxV2j_72q8A44fjzdONuDh0MYcZzSzCEy5NiJyWPUnZIOxJEqKUVihTeH3eiEtkgxm5r8QdeFdnyh-eWYz2sRzKgWlFLC-26c8MFZPVKQTCB1Og33E6HVjXPbQjTMWI-Splpo9HVQ8Ea9D5zGmhRLnGY7e2jGudhujLEjUT2--0swPCCL76BwFPWfBCDlLPQUwnKQXa1UBriUJx3NOCv8ErCY8mhtxcA==\",\"video_metadata\":null}],\"role\":\"model\"},\"citation_metadata\":null,\"finish_message\":null,\"token_count\":null,\"finish_reason\":\"STOP\",\"grounding_metadata\":null,\"avg_logprobs\":null,\"index\":0,\"logprobs_result\":null,\"safety_ratings\":null,\"url_context_metadata\":null}],\"create_time\":null,\"model_version\":\"gemini-3.6-flash\",\"prompt_feedback\":null,\"response_id\":\"OvZkatugN8m0kdUPk42_iAg\",\"usage_metadata\":{\"cache_tokens_details\":null,\"cached_content_token_count\":null,\"candidates_token_count\":70,\"candidates_tokens_details\":null,\"prompt_token_count\":1529,\"prompt_tokens_details\":[{\"modality\":\"TEXT\",\"token_count\":454},{\"modality\":\"IMAGE\",\"token_count\":1075}],\"thoughts_token_count\":1342,\"tool_use_prompt_token_count\":null,\"tool_use_prompt_tokens_details\":null,\"total_token_count\":2941,\"traffic_type\":null},\"automatic_function_calling_history\":[],\"parsed\":{\"type\":{\"type\":\"Reference\"},\"author\":{\"last_name\":\"Montaghemi\",\"first_name\":\"Nourhomadeh\"},\"publication\":{\"title\":\"Sonnenbichler - Montaghemi, Nourhomadeh\",\"year\":\"\",\"place\":\"\",\"pages\":\"\",\"publisher\":\"\",\"format\":\"\",\"editor\":\"\"},\"library_reference\":{\"shelfmark\":\"\",\"subjects\":\"\"}}}"}, "scoring": {"f1_micro": 0.8847402597402597, "f1_macro": 0.8736882129277567, "micro_precision": 0.8674890569040987, "micro_recall": 0.9026915113871635, "total_instances": 263, "total_tp": 2180, "total_fp": 333, "total_fn": 235, "cost_summary": {"total_input_tokens": 406240, "total_output_tokens": 39802, "total_tokens": 446042, "input_cost_usd": 0.6093599999999991, "output_cost_usd": 0.298515, "total_cost_usd": 3.433995, "total_reasoning_tokens": 336816, "reasoning_cost_usd": 2.52612}}, "pricing": {"bucket_date": "2026-07-25", "age_days": 0, "input_price_per_million": 1.5, "output_price_per_million": 7.5}, "timing": {"total_response_s": 3031.31, "mean_response_s": 11.53, "slowest_response_s": 39.24, "inputs_timed": 263, "inputs_stored": 263, "span_s": 158.76}, "normalized_score": 87.36882129277566}