{"_id": {"$oid": "69d4eb12b877594bc5718bbc"}, "test_id": "T0504", "benchmark": "library_cards", "date": "2026-01-25", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "genai", "model": "gemini-3-flash-preview", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"Simpson\",\"first_name\":\"Marcus\"},\"publication\":{\"title\":\"Eine Vergleichung der Wieland'schen Shakespeare-\u00dcbersetzung mit dem Originale.\",\"year\":\"1898\",\"place\":\"M\u00fcnchen\",\"pages\":\"133\",\"publisher\":\"Kastner & Lossen\",\"format\":\"8\u00b0\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"M\u00fcnchen, Phil. Diss. v. 1898\",\"subjects\":\"\"}}", "model": "gemini-3-flash-preview", "provider": "genai", "finish_reason": "STOP", "usage": {"input_tokens": 1520, "output_tokens": 184, "total_tokens": 3103, "input_cost_usd": 0.00076, "output_cost_usd": 0.000552, "estimated_cost_usd": 0.001312}, "duration": 12.943895816802979, "timestamp": "2026-01-25T03:22:56.185114", "parsed": {"type": {"type": "Dissertation or thesis"}, "author": {"last_name": "Simpson", "first_name": "Marcus"}, "publication": {"title": "Eine Vergleichung der Wieland'schen Shakespeare-\u00dcbersetzung mit dem Originale.", "year": "1898", "place": "M\u00fcnchen", "pages": "133", "publisher": "Kastner & Lossen", "format": "8\u00b0", "editor": null}, "library_reference": {"shelfmark": "M\u00fcnchen, Phil. Diss. v. 1898", "subjects": ""}}, "conversation_id": "f89b419b-877a-486b-8dbc-ebdfe103b735", "score": {"f1_score": 0.95, "precision": 0.9, "recall": 1.0, "true_positives": 9, "false_positives": 1, "false_negatives": 0, "field_scores": {"library_reference.shelfmark": {"response": "M\u00fcnchen, Phil. Diss. v. 1898", "ground_truth": null, "score": 0.0}, "publication.year": {"response": "1898", "ground_truth": 1898, "score": 1.0}, "publication.title": {"response": "Eine Vergleichung der Wieland'schen Shakespeare-\u00dcbersetzung mit dem Originale.", "ground_truth": "Eine Vergleichung der Wieland'schen Shakespeare-\u00dcbersetzung mit dem Originale", "score": 0.9935483870967743}, "publication.format": {"response": "8\u00b0", "ground_truth": "8\u00b0", "score": 1.0}, "author.last_name": {"response": "Simpson", "ground_truth": "Simpson", "score": 1.0}, "type.type": {"response": "Dissertation or thesis", "ground_truth": "Dissertation or thesis", "score": 1.0}, "publication.publisher": {"response": "Kastner & Lossen", "ground_truth": "Kastner & Lossen", "score": 1.0}, "author.first_name": {"response": "Marcus", "ground_truth": "Marcus", "score": 1.0}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "publication.pages": {"response": "133", "ground_truth": "133", "score": 1.0}, "publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.place": {"response": "M\u00fcnchen", "ground_truth": "M\u00fcnchen", "score": 1.0}}, "total_fields": 12}, "raw_response": "{\"sdk_http_response\":{\"headers\":{\"content-type\":\"application/json; charset=UTF-8\",\"vary\":\"Origin, X-Origin, Referer\",\"content-encoding\":\"gzip\",\"date\":\"Sun, 25 Jan 2026 02:22:56 GMT\",\"server\":\"scaffolding on HTTPServer2\",\"x-xss-protection\":\"0\",\"x-frame-options\":\"SAMEORIGIN\",\"x-content-type-options\":\"nosniff\",\"server-timing\":\"gfet4t7; dur=12930\",\"alt-svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\",\"transfer-encoding\":\"chunked\"},\"body\":null},\"candidates\":[{\"content\":{\"parts\":[{\"video_metadata\":null,\"thought\":null,\"inline_data\":null,\"file_data\":null,\"thought_signature\":\"Ep0kCpokAXLI2nzrXr-TExPqSGFNrYiTNj0-zSQuoRKALVrfYDBxKQQW3wVEHD_27cbijiBzu8xBfK76qexZnapbe_hk5uutQ-T092q9kDZwsKD8ZjhoBLH10KIuqJK91vaUj6cRROJYnwl3aoVC4elJKoPzte25aJD3uZBFWuBLdrneW8L5yIuvOi9ekYWZccmvH-yKr-3G5YZDv4UkN-DPhEFNnB3YJQMKKj7tOYjqpUe9na3JRJiA_SCONwt6L1suHpyOzUwTHo-6-bB9IQRTeiWI3akxQHu0akvNyWJ0vhXsSfy_1JQcOS_2IONomh3TfuTk21xtFIH1J2T9QZexWJJIYHanE051e5HQ5JYnjrBLwruP-_kT_uugVu6WlWM94ZLIcHv61Y-EvEX3B3ZnLYPMqb-mUz6ly1Gkg8Zvs33tviYuH134_O76DKZ-5kcQ8dAyNebk3CI1g8H-IOXmm_jadhrakSEdVt7JDipaKlV3mz2DbUb5xyf4Z6hZjeYpr1z835s8baO9wKrWdIIWPvDuImkRNKxAR4n26GIqAdkwUPaRsbMlC1R_KF3ok0PMPQfjS1BttiXqEhMkqJHcAr1qUu9pV1iNLi3AWCgE06ILVp4X9SMcaRSyW2j4KjIFfgfUOQmp0di_H0x_2YZdXRzryV6Io4WS5lPbnbc4EVzFlH3zrcj36GyJG3MfX2XLFMNkHeyF8gQvQLi55kL4174aswPkPiaO6g8HtHhRdvof-nc7uKYNS5gkSdiFJdg7NJdcFKyrd2i6FPlkUR-4dQ7zswyhO0_U0g0scXP9IJiBt_S0EpmzjMldTbyOGOJHtwAzL0LtdHZHn_rTfjoyDjtOiPb2EQT3Zu7GuptkJqlsFIy2EHtb_Qp_NY2DyYBHMIUYsDNVpCWPoZCnV8ktQhSc-iZsTjA7JrV6TeHFfBqLMwK8zu7ObCubw7-igYqtascuUsP927t7mNjvRyBuolcpUVFmwibnLDW_HQD5Svqjetnz3XbcSVX1nQIAlTGp096MrrjgD6OwMdt79ne3NPsSglVXpS3hg9Y00Nhh39EIjcO6FC4Mc5591nysGhvlHyiwKforFArAqOLrJq0oD9RZnuzWdVaUr6l6akIObDjfsd51f_v56Nz763FKLqG2TeU_IOcxcx3zwH7rpir555zWAtzOiTTcagsHuulUS4j6C7qByOCTMGqXx7JdzwWvH7YhB04XjcnLVJWw5fvtzKECu6MbEKqyRY65wvbDpMxRD0XUEbqGSYFK9nA1dfPwl2t17CFyzXVQ74RLYtIKieFv0SckctsVFGtdhCsrZ3Uw91cwVX7iLlAyso3Fa1oQrvX0TcFxCjT1oWGI5FxRrvQHHQuDqFvKGmMzylReenRLrJPlFTTcKwuqQmW2er-hRlSMC3FhFV4ZLY888nXuCRJFjPq_4hyuqRfdqFA-60cO7NaBdI1gtqO1KuhQ933QEko0FHWMDmXLY7I8-NZQ1jOTqO4PYrX_U33yNM-_FAzmZ6ZAixAviENvXXZZH0_9HPCTEcMx5IzoJNPMz1pi9Zr38QG_pKtXCui7OmMgXrHJ0VkJeveWGgapiklkQWnXpnbyk-e_nXuD25c-4zlQtBAPjfR10v9FEBAHVh0XGrvZMD6w6eoVtn9FzY4b_QUikfJqZW6F5qu14hP8gxt9TZVvR0u3bCSQMWF0UNfRvTU1asXuciuL-e3oMdI5V_z8gXJ2Vqheq9hGd6Dq9-y_caiZG6sTZfWcSTwwTfvLRLQwIh9CECmui6HbxyCrN3RFaXwBnZfT6sOg9GXkwC01YvtbeiPEsNIA8ayV643PVxF0igvQFbljU1m9RaPdCvEBcaR4Rknz9f3V10lhHl7vDYoXDmfJ-KnO-42x5tCpEKioRRFP_QIeOAa8Th9GCuQrpwl4yn12LspTbMj2FdfOt5UoredZzQxlK6BO4p12zQP788NzV8D4ZHV51v2X4Le4O-za9gDNRcm12oYJuq5Oqb_7Y59gtjKubFxWEOLRF-phyT4T4CAxCdiv_34Q2pZf_6D80gU8vu240dF9aJu7QTdj_EqCquRXEceZSZO6XpMX8Ck5htXy2GOU0AQLxBTp_wYPKYo_0KVtfl8ic0wHVrQjbrlBPXKJY8jo5ahTj67YoBWX8WB0y29yfuzqD8aorw6vv6Orkgabk6PNA2kSA55nEr_Wkw70rfNAo7BrDQK8wY8Szuvm96oTO0Pd8UOsTVUzJjrM2i8rUqehIvEwIsbuBnhfyD7rCfJJ_wtL70fzHrUYGxi5RYjYxFzmBHkicbGgof7N1NFK9joqGCfcYJGKQwEgyWAfJK7TgjyLQGUV3fgiwilj0nOglDabvGm_Y93oXsbqoahCP5f-qUFFi4mceAA4orH09mO_dr6WlVhQRCfe_VWcHBGOXp9L9y-vvJCtTv4ShqM9VDT_gv-eO3UbtfC168iri4fKUOYz2ZHIYmyENAtBbRPNYrLUHoZ5nsF8Teb1GJ_AOWLiNWTInxGClrQTyWAA0HlEv6TVlLZ3RILnhMIwKQMTbacHTjsWnNLWX0RNlIdV4gN3DJBzBse1EuNYQ4uxWQsmdZ3EuZ3tD7dAQ32QvNYw0g5rrgvPT5wgI9cP8ztCBGksFxdDmbisOmJQysKm9st1uPMHgDZf9Zuw5gRrjtM1JXNkflfeXA8UhWzIFtKhhsWp7BwCeqcz75jAR-p_2CJMmBwS8htBIl1BHeZU_GZTT17XUFQYC3XqxXG83feWo5V2Mxaqprtp7H_LHQQr6yJU_kzpKG5gdL-8PMUYlBir0ghL39RE5tteSCjhjvLDfzNCUjH3osT64alFitMP0eMPJ6BbjQEvAdHAhfo3xdZls1oUT7HY2PGBuyx95uljza3eQd9eTdU5MwtGv596MObIkjCoFlW1roB_jlXDuGXgMlJ1EQGsF3q534ResTrLcS8gu4zlI-vz6Qm_WVA0B-O5fmBMU-cksO0wrNEe-QpvhbrY4px48L4-YUlKmBukpeldNlPnEPAum6gTNGu2IYCmB7bUyxzbtD4NpIQ1mEs_VQxM4flA-gytjCG6t64MvLaCoM7kR7idzTUYbikxHq3Jg5oEXbLTmr8ZNctr-_6nU9MlbUCEcF2hvj1N_9iwkn8nzBL-_IMZRfhZyzuGtvXPHRJOssskJ5lK1IPnLOhD-9k4rk8PJVrXpB-tJAd_vH0BOp8-Xft74MPF_-lDsqVnaZfwVUaCUDsn8p6nyccr5gM43LTpbGOPtF1c6v76G3YDacsHetkQW0eRAMrIu5Sz0RIzkqF9CYylc0BikaI3cZdrU7RxgRD9ANrnXUjvhJBIWgpp80sL_NtdyvWUr7URTmaujOFgNZSpjbQrPhGdc94BfEEdyWv8A4bWeDr4bGaNXcft_U3NM618D0xThq8rmJG_32ZSXc9iJs13B6FY4vnOPsS7iYLxdAh0zq_vlwLne2NNRPAjwJPQyTFrVUSOdR-lrUHcd_OgOb8wqkcqrcBmZ_i9i_twzvIFuRAQdNNaSvbjiZ4s3XBpFbDCIdKsJIKedyFeN9Fp5xzkZJBIXSqyXj7HquKSHW5wO1_s9MSZscEiiFeYLXDjLRSJ1BZ_DXjHnJaWjf4sjDlsNRxVRxqQ_t7ZVuOcL7LhQhsYjwXgvX4lu46K-3jaob-1ztantA8hRwTAAXjMn42yd7TT8h8-o2__KQfnATJjUp9GvmSZqbLPevSFP-_8u2Tnn3ubhTbU89OmhmJllHpz1grUXPbO48oJEXrzdWKRBXc40SwHbEaNrSww0uwbBScLLP5W0noi4F97y6owuPPgJ0z3zxzuXwQTL9PU4RuMQTufIGvxZrJAtXyx916PjylhwfMgg5OP-oo5kZ0-rXVmHoygr10U-6pt2x9MFH4lkjDyQ2j3k_5K4F9oYHBK1GjaQMR8dPTNJBcTxw4KsoFlVRd8zOHHzMaba832y6q4kcRXyDduEKG5OQUEGJ0BNKwE4KAj-z_xmsADXiResC0PK1DLcXZoFKi1UWvamj3UzS8ADKc8mtpasrqjneI8cw4A9PScuOZzgmy5yiao5AcOwXgfbnF9dw8qLSGIUWU4qGvHTSX79f_9uTayDzsQznthegKdJFcxrrGSE8MSFejkSYsla3K1NJUpFF4K51RJAkHojiDGoxGc25orzjNLxxj5dTnnPrJjP_Rtddp2JV16U5eTxXp89tW89Fp087Ce9RZLjgmhGCm21ba0jdDZ3ceQbzfVD4Bq-7iWx3BPTqyFLZ8sf_7mdA9cr7YqvWievD9UYNnIG7MlZv7JvFmmRzDFHJg8dEQ0dPBhfCXGFmGuVd57esCG2LPPpwXxp3JqcaZHCPwhees6gnnvgy3W6i5ZkT99rQbyg-IP3UCrN04xE2ZeUDGWLVkq6hfMX-Md5AJPGyQs6mrUmM9gHH9qI1etW3QDCM1mlVfUyeAaWd4CtAzVDSvMLaAhx-LFlMprTbTWztcsowPShasGUZCMR99OvADS7JPBxzBTa4IpHZESqx-9cSoSgXHdvRV3N7VJ0SlJH2Lxhu9nTo8VFWa1_PR3XTw48OANmDkeAOk3s5WfRqzAvsi0_7NtrNDU9mC8F4mOVvR7B6oqyotzM3EB7mz0iwSy_9k2k7kDTfhHCOrXFFLHaD05f_8xY6ggtHugidKhEPomV8c1hIhhXetY8kFx3O9sLE91hDR2LyLDDWL3g5g-ut3tcl1QsDwWV9HryTav10CNE-Qb1-_2gpTSeEmiC9aDOqCfCVybI2t4haadeiIcd58gAc9K0GZKJ23t2hBh45N5d_8Fd82XujX_X8eHMCoavKTMzQ-8aUgPMxBI2Sd__Oe_8-1aPstT9iAevfhDhNEcwcADFRQz2tvL71PucX154g_zZeusaXXehDvIMZabdHPbwt-RKXPxSfJHonp2WbOFHT14nDMYFeqLX0krOA6gwSgnI3QMXJI_dt4y4b3627GLdN5UG5fsX60xlh-bIt5_lTi__lWYQf6y5_fYZxrfi0twu5o6CcvOaKpkcomxaXPR_xCNaje_UX00Q_-O6-H7FvgRENyUQM2uYmgUN4l0aaEqRFKzXosghuIr8N9mHQ9QXRCd6Zz2WGM0J8qnKNHZQQofgSfA6iM_7_BeIM7wtQan_PxQfLn5xCyfEowBCow3PRXuixkv8DHATFlYjIFlZ_2a_SyLk5c0DzuuH6uqtjewsBzqUEzzWLgAWvxz9SkTVaUhLeGjfQjuKnEoIHIKNEyGgLIjleBq09UVO5OuCKMKF05Oa5A9DvIEL1FiPdWv3dwKQaghJRE8kz8988HCA0tg0bYjAtgQ9oOwBCwch4zlaANa3Hwm6GYaTcB8tkculqUXXpxYZVMQd-7zoEVCg976wFNvGvtVMGIXr5mLzAKd0CxkEdDoVm8SC68SKwiiy1a2or9mhDfJC7XoPr5Fl-uFpyGGZUpYrMkUFoILd_RQyM44AdwSN4byu0UgAbe9ZeQpv1v8-1DQOZT9Z9ci0nh0N-nd9Vdldd1uT7acx75YFfam-Yn-DjV6iwHQwS2E_1Q05ndeaFqbUnn9MwPqqaWdqDk-toLI_j-BeLvKGvft7HJuk46c-hlHXf0FRg9EslTqaSyVXwwTSoG2bOuHPwe3qVZpoiW_5PycZuk4LB85D0PaLZ-D-H6PFcDV-AJsXA4xilL5t3u3Bx9ZZAP20CKSqXnERAPzPdU_DYWrjgMjKWHaJPDPcxBGTUlzq5h1C84QI_Cj_zhIucy4aqwdp3HfKjmmdyra1cQ-QjunHZTymDTbv7-ICPijBUwapTumJnyId2TqxWNFYhTcDToHYWZhN22lhooFhCyGBSScY7MInhU9kY9MTms-Aa7-W4LopMDUGohknK0hRbxs6v4tDnrLTrrsNEuhA6xc0tyUVu_dydravzPKvoOWGKo8jwC_gLCyP_8qHLW8QbWNqi0zUPJ8dT_Asw3heD01vk-XSgCgygTwfUmvPDZD4RikGuW6ZNRxcy_2plxiil-awZhw01mOYIkKtbyIrDEhYtA7JtkUMImN6OEWUzeawlekmv_DCp0C3nKGGPS8GhhqQSWPSeMSALtEOzXF-Wqe-1QzCNoUTlLthOo9F_rnCn-6aBSJWl9BnKsj9jEYdKyNP_xx-MWddPzTdNGm4Fb3gjTdrpE=\",\"function_call\":null,\"code_execution_result\":null,\"executable_code\":null,\"function_response\":null,\"text\":\"{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Simpson\\\",\\n    \\\"first_name\\\": \\\"Marcus\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Eine Vergleichung der Wieland'schen Shakespeare-\u00dcbersetzung mit dem Originale.\\\",\\n    \\\"year\\\": \\\"1898\\\",\\n    \\\"place\\\": \\\"M\u00fcnchen\\\",\\n    \\\"pages\\\": \\\"133\\\",\\n    \\\"publisher\\\": \\\"Kastner & Lossen\\\",\\n    \\\"format\\\": \\\"8\u00b0\\\",\\n    \\\"editor\\\": null\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"M\u00fcnchen, Phil. Diss. v. 1898\\\",\\n    \\\"subjects\\\": \\\"\\\"\\n  }\\n}\"}],\"role\":\"model\"},\"citation_metadata\":null,\"finish_message\":null,\"token_count\":null,\"finish_reason\":\"STOP\",\"url_context_metadata\":null,\"avg_logprobs\":null,\"grounding_metadata\":null,\"index\":0,\"logprobs_result\":null,\"safety_ratings\":null}],\"create_time\":null,\"model_version\":\"gemini-3-flash-preview\",\"prompt_feedback\":null,\"response_id\":\"gH51adOlDLWLkdUP-dKO-Qo\",\"usage_metadata\":{\"cache_tokens_details\":null,\"cached_content_token_count\":null,\"candidates_token_count\":184,\"candidates_tokens_details\":null,\"prompt_token_count\":1520,\"prompt_tokens_details\":[{\"modality\":\"TEXT\",\"token_count\":454},{\"modality\":\"IMAGE\",\"token_count\":1066}],\"thoughts_token_count\":1399,\"tool_use_prompt_token_count\":null,\"tool_use_prompt_tokens_details\":null,\"total_token_count\":3103,\"traffic_type\":null},\"automatic_function_calling_history\":[],\"parsed\":{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"Simpson\",\"first_name\":\"Marcus\"},\"publication\":{\"title\":\"Eine Vergleichung der Wieland'schen Shakespeare-\u00dcbersetzung mit dem Originale.\",\"year\":\"1898\",\"place\":\"M\u00fcnchen\",\"pages\":\"133\",\"publisher\":\"Kastner & Lossen\",\"format\":\"8\u00b0\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"M\u00fcnchen, Phil. Diss. v. 1898\",\"subjects\":\"\"}}}"}, "scoring": {"f1_micro": 0.8657957244655582, "f1_macro": 0.8561977186311788, "micro_precision": 0.8293515358361775, "micro_recall": 0.9055900621118013, "total_instances": 263, "total_tp": 2187, "total_fp": 450, "total_fn": 228, "cost_summary": {"total_input_tokens": 406240, "total_output_tokens": 38275, "total_tokens": 444515, "input_cost_usd": 0.2031199999999995, "output_cost_usd": 0.11482500000000012, "total_cost_usd": 0.31794500000000003}}, "normalized_score": 85.61977186311788}