{"_id": {"$oid": "6a671d25338dadb0be4e84b4"}, "test_id": "T1425", "benchmark": "library_cards", "date": "2026-07-25", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "genai", "model": "gemini-3.6-flash", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"VAYSSI\u00c8RE\",\"first_name\":\"Emile\"},\"publication\":{\"title\":\"D\u00e9viation du compl\u00e9ment et grossesse\",\"year\":\"1912\",\"place\":\"Montpellier\",\"pages\":\"78 p.\",\"publisher\":null,\"format\":\"in-8\u00ba\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"Montpellier. Th. m\u00e9d. 1911-1912. N\u00ba 30.\",\"subjects\":null}}", "model": "gemini-3.6-flash", "provider": "genai", "finish_reason": "STOP", "usage": {"input_tokens": 1546, "output_tokens": 110, "total_tokens": 2914, "input_cost_usd": 0.002319, "output_cost_usd": 0.000825, "estimated_cost_usd": 0.003144}, "duration": 7.515953063964844, "timestamp": "2026-07-25T19:47:03.404248", "parsed": {"type": {"type": "Dissertation or thesis"}, "author": {"last_name": "VAYSSI\u00c8RE", "first_name": "Emile"}, "publication": {"title": "D\u00e9viation du compl\u00e9ment et grossesse", "year": "1912", "place": "Montpellier", "pages": "78 p.", "publisher": null, "format": "in-8\u00ba", "editor": null}, "library_reference": {"shelfmark": "Montpellier. Th. m\u00e9d. 1911-1912. N\u00ba 30.", "subjects": null}}, "conversation_id": "15716bfd-ba53-4797-bef8-a63ab10ec5be", "score": {"f1_score": 0.59, "precision": 0.5555555555555556, "recall": 0.625, "true_positives": 5, "false_positives": 4, "false_negatives": 3, "field_scores": {"publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.pages": {"response": "78 p.", "ground_truth": "78", "score": 0.5714285714285714}, "publication.year": {"response": "1912", "ground_truth": 1912, "score": 1.0}, "publication.place": {"response": "Montpellier", "ground_truth": "Montpellier", "score": 1.0}, "author.first_name": {"response": "Emile", "ground_truth": "Emile", "score": 1.0}, "publication.format": {"response": "in-8\u00ba", "ground_truth": "8\u00b0", "score": 0.2857142857142857}, "publication.publisher": {"response": null, "ground_truth": null, "score": 1.0}, "library_reference.shelfmark": {"response": "Montpellier. Th. m\u00e9d. 1911-1912. N\u00ba 30.", "ground_truth": null, "score": 0.0}, "publication.title": {"response": "D\u00e9viation du compl\u00e9ment et grossesse", "ground_truth": "D\u00e9viation du compl\u00e9ment et grossesse", "score": 1.0}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "type.type": {"response": "Dissertation or thesis", "ground_truth": "Dissertation or thesis", "score": 1.0}, "author.last_name": {"response": "VAYSSI\u00c8RE", "ground_truth": "Vayssi\u00e8re", "score": 0.11111111111111116}}, "total_fields": 12}, "raw_response": "{\"sdk_http_response\":{\"headers\":{\"x-gemini-service-tier\":\"standard\",\"content-type\":\"application/json; charset=UTF-8\",\"vary\":\"Origin, X-Origin, Referer\",\"content-encoding\":\"gzip\",\"date\":\"Sat, 25 Jul 2026 17:47:03 GMT\",\"server\":\"scaffolding on HTTPServer2\",\"x-xss-protection\":\"0\",\"x-frame-options\":\"SAMEORIGIN\",\"x-content-type-options\":\"nosniff\",\"server-timing\":\"gfet4t7; dur=7490\",\"alt-svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\",\"transfer-encoding\":\"chunked\"},\"body\":null},\"candidates\":[{\"content\":{\"parts\":[{\"media_resolution\":null,\"code_execution_result\":null,\"executable_code\":null,\"file_data\":null,\"function_call\":null,\"function_response\":null,\"inline_data\":null,\"text\":\"{\\\"type\\\":{\\\"type\\\":\\\"Dissertation or thesis\\\"},\\\"author\\\":{\\\"last_name\\\":\\\"VAYSSI\u00c8RE\\\",\\\"first_name\\\":\\\"Emile\\\"},\\\"publication\\\":{\\\"title\\\":\\\"D\u00e9viation du compl\u00e9ment et grossesse\\\",\\\"year\\\":\\\"1912\\\",\\\"place\\\":\\\"Montpellier\\\",\\\"pages\\\":\\\"78 p.\\\",\\\"publisher\\\":null,\\\"format\\\":\\\"in-8\u00ba\\\",\\\"editor\\\":null},\\\"library_reference\\\":{\\\"shelfmark\\\":\\\"Montpellier. Th. m\u00e9d. 1911-1912. N\u00ba 30.\\\",\\\"subjects\\\":null}}\",\"thought\":null,\"thought_signature\":\"EpIiCo8iARFNMg-twyO7DowiVSyhPIlaCj-UVOZy5lVrB0eV4j_hZ7qDJ2TL6Iz_QwlCb1aYeq32mY0-kXYbCyxHc1wb7mUkjor1Zehv68KE3FNArtYLxlYtpR4QBsk_5pxFVH3YLCR0VcWTOiDXA4QyMvEs-cnFcivmjjIuk4D8xKQmu9jSGsQwxINDs4G-p25TfNRVgg-V2EyHPk8tPOlNLp117WLAhmze0FX-uNhzyX8nUvnBV69hp2jXNrqMogkP2KB9pOjgo1oUrf6cauyowoCZtfBL7kZFu04EdRnFHnYcBV2s81Y-lRmC5aIAe1MNoGgtjO8RWqw_sGxsqkEy_G26bPibKfMMi1h0mRw1ScoDGtKrTEao1djfMEKhEoSX7TeyKqlIGTPPgUPBhW3OtM2PWVKLe7Vdqh33MZ9562bbyTy6k4ESa2VUZSEDyyiJtw_VevCLCakr5dEX3sbCGc8TERkeR28Eivc-RpehTl4J1MtOZdAyWf-2DezEy6Y7Km_2C2_gcVgehW93bgTPrI3KLGr-QeL64zQq33xv7RbzLJYGA5I3DcdNdJnMzOTs6mmGkwOb8uZ0lF7d5Fdd24cTlkSYb1zr1bHEQY0lGaAFfJ9tlLy0RYocQkiRW8H96oVc0dNCtDar8v522mrtK1VxIWWpaw3I5IZOQ61tCSwrRN_TWhNYlKAti0WpxkNAsRS43Zd1NaS9DY1V14Pm8w99MTCdZiPGJguVVgs4rIONIs5fEdvWorssbQX0-RG0N4li-ma5tTM8KuwpwOYdfupJXkoiDLbxlytaIS4o9AdG8A_6pNqHaYws9Qm8XvvPjg61zzLrzJ9QrfOZT_bTwdAdYPzWwrIpG6uiQky3cNMO4YNa2H0_adP4_rmrUeeNRJoetCsnOA3zzHr5TxL_46_SrEikD2WHLzBi0FxII3jnKzl69PZzX7AH1YeOCwXWevSgNN7xxi9NgtmWrofC2eAitG2ScVzJe1BzPky9FYqJD_W0E_hoQslV6b4N2hMV2DloQHJfCFkfpu-_KyiMZ6Q9AZIKsMeEJvnWthoDi6D3ISiiA0Lwukrl7JutUCAakdoYbsNdEbkyuVTNVAZcg0AOsO6Q19pAhchigkkw62WawF9g0jfcIAqWTw5d1VaTmLxVOg1qsRKrN4ju3R5YQo9XUHMlKECs7ziN5SHpExKtar4dO_TkTuSG398mhJYzZ-S9sS9oZUAUnwIAedsdhQ-IeGqy5UPB3gVDChiM0CZPnri6QMSogPpZ4bg_s3brX0eKUMkve3vk2Tk5Mxq88ISIbZLjgIH96cPWSksmam_faDkJwVBXHp10gphlTHnGp5ZaVvuV3gTCHoGp8ru-avTs7rL7HoN6e5q2OzpWs-8a2JwcCcRDWjjgIHT2thXino8QwbsDZZAIbjCE8fjaDGfGOaTD5PKp7t5ARoRtii-1Bs36S0Pz0vc8GaZEMNhFg8ESy5ABNV8Eja3TM3oNcIJ6dPImGl3iNtZRJgf8UxEzPkW4pHvlHZhGTVXi3n67I_75ZAWWk4OybKzi72Pd6MLgtYO0QmOI00Ao2-iFwzsmAXWxe3nhDNxHEPnN0eYwBAMcv237eUlfeGixCRIzp9Zyv3UBz9gMmAtP2boPOFKMkU7smzoUtT5jwAX3-VeE9iyd_lfETnU2fBi2VIq3-pHwIERbVkf_GANYrf6rimP8-WXSlpqXFMoYpFyPGnvQSrqzYQJDtSeNtuIS5dsURi3_e9CtZKhXqA7SMxo8WDTYhAnR2o9tjI0JoP-Lwa-ww6zDKv78WFrCEP1KywAGui3RH3HapwkV_BwZbzBYHhi3q-rAuklUuCHD7T5wEpvdB-kWjZglYKAK3vOnzMSbWim-87JFXuAJPp4AtYmV6ObwBhnGkpojfFEfA2A31j3RylycDr6Q_o23C7A7bXeFlD3ztfs4rKf0SbAP8OhyBD8fwdJiNv4Uk5xweoTt9yPt-BVjPLzd96Wh4Kzpb3QBh4_0gKW7P1aV5HmKyM_wjZddCpp5f-tkdsYia6FEO2Vt6Mo9bMHF_AbWKIlx4bh6c2V2szFHwbP2QYGBWZbUHe4_IVY871heXd5ZsMVvnZeUFq7dQoAxlgpveAXSUi7Wpu_ljWRjEuPmOoZQGxdqI-KGbgZepekYBEI-4yLjyu1f_ghFoxZ_etneUjHF-GG5a57ZAeUyt2C0wiHbtA-YOJUgvfhFFiJsrxgPL0ymP3qnMnHS6RDoqFnsaKw4HFUpKWax9_RMDD56l3JLzJf6l8MTW_BHqK8XfUjJaAx-RhJ0Ft3ryF0-BAxmTYkVXyYzHvJFH-YzSU_73eYyaLiaULsnS2XKghlqu96eDY2xXi78CSb-5Bww25G0wi4zoHTkrKs7b65I6G40lGKa8GCLsyV1gjMi5Cutromasl1yg_1Hg2VdRT6ePdnWnlE9p1P_xgnDLouUIpjvAyK62b4M9VecAuaUOM_KLxfuU_Ds2RFYIUbhrRPly-xbWFjlGvVvP_ghkPvyCPzUPsJyPcAcAtUMHnZlqvh--POdOXw4SvJp1mXqYU65cbpOH3wzPuk6px5VlnT7MK6qt8b3yyQ8Pct2TJd1s-yiR2ujFk907d2iDQWHsZNnos-XsjaC0RKteCWK-lrFQ_7VZf3YHaR8_NLlj5CCY4qBe9FDhaglBZk_-WMGc_UpRUX4yb_daQi8_P2HqBeTTa7p9B2SZKDT_ZlH_Xfo6WIqLdwEbWOl27MxpN3NrjEF2-hHKYkrkbEA5zy8-DDrr47V4J37Un7_xxHQ-wYF1ABS3yyBN2sU14I12BfODjezLBALJHga1ThdKJUfE1bM75RfAN1wYZEqU-ma9AwySIRGTFRpGclzDHEGCVd6o-aLdnTnl1-SH5nXSIvWJ6DBMQwf_QKZCwWyzlDKanIyk65EdxyXIk6-6cvQw7aSC9Gil_7RnB6TTiv8tJd-lklitEVs-ntymylQkLuAV31-3_kfkKhSV53o6SqPsfrkBYuaxwhbthVgHQ5TqheuJLKXyTOD2GcDEzu1yo7KHQbsqhRmwupc72VKMQc4rigIpkAlZbcFMJBZPwsKGnwqyL9zwrKSw0xugIr8ptfwEZMkGyB_KTM5-mU8v60Es8NlTxJ0irE5RwSEiiDoXJ8xmo_Zy2ewid-JB3Lw4s2rH5Z4jvE-TRcWWlC6gXzj_Dhn0RRxQzdO0jqzLkanvnTbton4bqkOlbP9dgp4x5uAzgWWKpxh2OBsQWR557AIV5s_whJUBXurHwfjEz_NmbRp7i-FMEC2UQO6R6jQhc1nMoWtq-zRExHT18_zwTcWogwh6lET5Q2PZdWVrNeI4zGDqeHLoYrgj4WAXzSyfIvO4CP20wQp7fxtA5hTSO4vo9pfgQoLEH3CEHK-DDhMWwoMCk2OXDGGLR3UzQQz9045jaHUZVeugME39woldCqLX72hA6YejOFX-UK--p2ug5f39AB5zNNTg7b8An-3yyWXF84cCcl4sPD-qYKzUk7s-Cj6AbTg_Qpe6VBW8twl3Iov31FNXI-fs8RMa-QQX6G5IMVlniI32fsqOTUaRo79dp1pYX1gzg-Fiu65bAtFbDo25uLf2S2LqNx3TnhfCM9gVa216ZL0MHtx1N2tfZ-jRZT_exbAv-tVZtNxL42xq28IxWWVVIWSaxZmOW9aJStV3pZvA9uEqz1P0r3WZPBiX6w9g07uK9yu3_RyunMv4I6GQH_0ZNXVkcGpKBXsGaW6gwo51PMswgss7td3CyTaH2gSXpNxhmmocb0gMno-2tbJQn3sZ200uoyV0Bz_HcVyhG5KD8efDsNht5o2aqK4yNPjPlmbos5gMg5Gbbr57-9g10B0X1KbBuinjMSqpRCTy6jLoNeijEfP02XXZUuvwc1GerAFVgqnsNktPpgc0slDS4XrakisxqXek-tVgNWHbmIYQK49FaUyXo89dSnqwNiIxlNFvupO0_-r-gwYxoIl3kpY91wAcX_7nxTcr8LEAJe_NqaatiwJXK9bxBR93JVxBI4W3tFHdAmHH61Mv2zgzOZhXLQ0NVBaUTL9poVR2Dc374UedVfBsG4FA8kFn0w0q5JVQj_EOBjdxYhm9SSFSvuoWb6kTefScij3NKieATMJMwoSAEISiFxgTJ9eQ2wDCbDAHlBBqMfoY6tHy3XQmgHUf7F70vLS98muPfE1QmxnQfoFR5eL9mQpsxa95qh4AExyx8IL23nsP2MUt850pZMFDXLZvDn9vctGFFzme4IJ5NxN3Q2NRLml137WhXeFpvnCKOtrq-DiBbPIrWBuldDgnndyYtDP11e67N5LSYPt68dNGu8E_WEHEtq9DOHhu3E36Nwsphy33t4uokwm0hCYfkUKObd1q-Us6pjKHGakwoBGY5QmJImkyNRK3Ns3xrfFDZGNclzI7XMbbQrgzL1OlRVA0f-nCjAMzC7CXhxqN49vDKmW-WHObaNLSQod8jY2mhDUt6HcVxTwX04Hx-Df8You4jRTGnSIC--uUacaU2-QCTYlk8g0KJgCYV5g1h8lMk1AcZu29BrUXGos1hwAClkRBTPpiy6BVzePb4XIi50SvCHShFF1OpErmPTbs57LeCqc2rc16aXMe2_rn8N8Jc2U4fgyrEYzAXlPZ8V98vXs4vIGC85OQFih-hrcjkLAYFBK_oQnMipvrow3KORyP1Q2ZIc4EMjX-3SXpQxQToRnunsYVTURd4Bv4Y2SL7iPYDPcosxlvWIsiBZv3YuyNzh9Jxyv-fuOo6TYC0cx3b3kssn9xJNplVpgYVPaBbkYacpWxOAFmIhOo2Ee4fnkIPJ8dymgN0ElQfBQIAGnLQtP_vkyBCkggkGLaBfm9ltKVpnLINHP6vU9ojoDtYNXVTwLDqOm8GKySjJvlk6d6w6-f9NSW7yEE-O5y1kz_5zWNLuBkQLYT959aQoHUCE5_QjyOm5_nBvSDZWPqxdbEck15FVHmtwkbjq2kphqZJYKxCva5unCfucdNkRdV7AhwbJogo2mRhQ6wN-fTgUkXLa9gZE1xQaeI_9dw2DsCSIkdeoc8nuuMVdbBZ6324rAqUdDX-Qxww1alnqd2IWu40cXJkG8sNumka2BS3CfR8dgtabYkFiEIL7gT8MMRbFJfLO6FNc5tmqSHji-0Yee3lXSwmGzTNpxcsCIfvqN_BRMemRoLZAXbhRYSD2XPWyjO8pfmBzwMwzvQnp9XZze8uPcMVha9ugTgu3d3B6L6idMLiNoZAzM2D89TpRw0jFTuX7BhU0xVTNINE_TAF0Qd0r3O7b8_jWOVulejo8WEB6a-w7swVZTPKcMMY0lf9Y6FoVPxgIkNEFJxbLRRXtzBd2Z4YKg9Y5YfErJJmPrjo5_s6r1hEH4Q0HL7JoTA8RxlmCfQRC42tygHIb4SHi6YrqkTyNVI-sLCQMHUiGfCgFXL7rnseWum7Xz7d47RxLXCBQGgDbMtCCWEn4L3fdGKX9TlJTq6BAYBJ7ju28FC7CDHK0bFSoXlxTA_R4ShXlDVvm94hMtKtVwx8Vr33NJVL3a_J9Dj1FbV_huYCKY00oEt8aG84yY9Xzmiegf3SBgnjzUZnGMEA8W7819b7_HvkW0BKo0A7ECu_9Zx3imceM2mUMkOS0UPRCk-8C_MfLeRkTritxnOhxeHlKdaOkHYqPjvIKIm4JKLaDm_yLBOoUqM_ZA2jvV_1D2NuF90dq4nfF1BhDfqJqGC3avxf6KkNezX_Z6QfIYFckjkvpAu-C2FejGEJ8suP-Ceear-nVZjoONyQ756mOOTU1p_85f_3sUpVOX8xqTSAGvF8Y=\",\"video_metadata\":null}],\"role\":\"model\"},\"citation_metadata\":null,\"finish_message\":null,\"token_count\":null,\"finish_reason\":\"STOP\",\"grounding_metadata\":null,\"avg_logprobs\":null,\"index\":0,\"logprobs_result\":null,\"safety_ratings\":null,\"url_context_metadata\":null}],\"create_time\":null,\"model_version\":\"gemini-3.6-flash\",\"prompt_feedback\":null,\"response_id\":\"kfZkatzOBMHznsEP0ozZ2Qs\",\"usage_metadata\":{\"cache_tokens_details\":null,\"cached_content_token_count\":null,\"candidates_token_count\":110,\"candidates_tokens_details\":null,\"prompt_token_count\":1546,\"prompt_tokens_details\":[{\"modality\":\"TEXT\",\"token_count\":454},{\"modality\":\"IMAGE\",\"token_count\":1092}],\"thoughts_token_count\":1258,\"tool_use_prompt_token_count\":null,\"tool_use_prompt_tokens_details\":null,\"total_token_count\":2914,\"traffic_type\":null},\"automatic_function_calling_history\":[],\"parsed\":{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"VAYSSI\u00c8RE\",\"first_name\":\"Emile\"},\"publication\":{\"title\":\"D\u00e9viation du compl\u00e9ment et grossesse\",\"year\":\"1912\",\"place\":\"Montpellier\",\"pages\":\"78 p.\",\"publisher\":null,\"format\":\"in-8\u00ba\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"Montpellier. Th. m\u00e9d. 1911-1912. N\u00ba 30.\",\"subjects\":null}}}"}, "scoring": {"f1_micro": 0.8847402597402597, "f1_macro": 0.8736882129277567, "micro_precision": 0.8674890569040987, "micro_recall": 0.9026915113871635, "total_instances": 263, "total_tp": 2180, "total_fp": 333, "total_fn": 235, "cost_summary": {"total_input_tokens": 406240, "total_output_tokens": 39802, "total_tokens": 446042, "input_cost_usd": 0.6093599999999991, "output_cost_usd": 0.298515, "total_cost_usd": 0.9078749999999994}}, "normalized_score": 87.36882129277566}