{"_id": {"$oid": "6abbb1d87a3eaedeeb34da7d"}, "test_id": "T1055", "benchmark": "library_cards", "date": "2026-05-22", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "genai", "model": "gemini-3.5-flash", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Reference\"},\"author\":{\"last_name\":\"Montaghemi\",\"first_name\":\"Nourhomadeh\"},\"publication\":{\"title\":\"Sonnenbichler - Montaghemi, Nourhomadeh\",\"year\":\"\",\"place\":null,\"pages\":null,\"publisher\":null,\"format\":null,\"editor\":null},\"library_reference\":{\"shelfmark\":null,\"subjects\":null}}", "model": "gemini-3.5-flash", "provider": "genai", "finish_reason": "STOP", "usage": {"input_tokens": 1529, "output_tokens": 147, "total_tokens": 3022, "input_cost_usd": 0.0022935, "output_cost_usd": 0.001323, "estimated_cost_usd": 0.0157305, "reasoning_tokens": 1346, "reasoning_cost_usd": 0.012114}, "duration": 7.663616895675659, "timestamp": "2026-05-22T09:14:59.111990", "parsed": {"type": {"type": "Reference"}, "author": {"last_name": "Montaghemi", "first_name": "Nourhomadeh"}, "publication": {"title": "Sonnenbichler - Montaghemi, Nourhomadeh", "year": "", "place": null, "pages": null, "publisher": null, "format": null, "editor": null}, "library_reference": {"shelfmark": null, "subjects": null}}, "conversation_id": "3b950fa0-e530-45cb-b8b3-e819b8ec5283", "score": {"f1_score": 0.29, "precision": 0.25, "recall": 0.3333333333333333, "true_positives": 1, "false_positives": 3, "false_negatives": 2, "field_scores": {"publication.format": {"response": null, "ground_truth": null, "score": 1.0}, "library_reference.shelfmark": {"response": null, "ground_truth": null, "score": 1.0}, "author.last_name": {"response": "Montaghemi", "ground_truth": "Sonnenbichler", "score": 0.34782608695652173}, "author.first_name": {"response": "Nourhomadeh", "ground_truth": null, "score": 0.0}, "publication.place": {"response": null, "ground_truth": null, "score": 1.0}, "publication.title": {"response": "Sonnenbichler - Montaghemi, Nourhomadeh", "ground_truth": "Montaghemi, Nourhomadeh", "score": 0.7419354838709676}, "publication.publisher": {"response": null, "ground_truth": null, "score": 1.0}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "type.type": {"response": "Reference", "ground_truth": "Reference", "score": 1.0}, "publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.year": {"response": null, "ground_truth": null, "score": 1.0}, "publication.pages": {"response": null, "ground_truth": null, "score": 1.0}}, "total_fields": 12}, "raw_response": "{\"sdk_http_response\":{\"headers\":{\"x-gemini-service-tier\":\"standard\",\"content-type\":\"application/json; charset=UTF-8\",\"vary\":\"Origin, X-Origin, Referer\",\"content-encoding\":\"gzip\",\"date\":\"Fri, 22 May 2026 07:14:59 GMT\",\"server\":\"scaffolding on HTTPServer2\",\"x-xss-protection\":\"0\",\"x-frame-options\":\"SAMEORIGIN\",\"x-content-type-options\":\"nosniff\",\"server-timing\":\"gfet4t7; dur=7577\",\"alt-svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\",\"transfer-encoding\":\"chunked\"},\"body\":null},\"candidates\":[{\"content\":{\"parts\":[{\"media_resolution\":null,\"code_execution_result\":null,\"executable_code\":null,\"file_data\":null,\"function_call\":null,\"function_response\":null,\"inline_data\":null,\"text\":\"{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Reference\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Montaghemi\\\",\\n    \\\"first_name\\\": \\\"Nourhomadeh\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\",\\n    \\\"year\\\": \\\"\\\",\\n    \\\"place\\\": null,\\n    \\\"pages\\\": null,\\n    \\\"publisher\\\": null,\\n    \\\"format\\\": null,\\n    \\\"editor\\\": null\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": null,\\n    \\\"subjects\\\": null\\n  }\\n}\",\"thought\":null,\"thought_signature\":\"EqkjCqYjAQw51sfhzZKlUt7Kqc6RVeatf_RNUumAvPi4cyKAHD2F8fcxydbcKNQI5Fwx2FW_qX3QO77oh7ZUOj71wXd_fhMx8ZhEVBPKuVynGiil-GNh-dmuiOIvAE9eqpnRUVMz_Uy4pbRiBl3DGhiHH4Im_AYRjJkscVm8NQGZCWE35Wk4Jh1mcC7FD6BvYQu5OW2DQ8i-XVGNEuGT0LnfWFj_Ir4R6R8ZcV2FJGFJfghtHPyEwgv8YhyYkY0lqKxd1yzasIitvoakkbagK2GQ5eg_GRAoQdZBedzKWoIsIXJgn5DiyhTKeyuNAaIGfKmbWo80K5qfadgjpZCIDwXr7c5n3S6IwRI3vthzfqfhZePY0BBqseMT8EcGcOSPh4hAqwo8axs9tuQD7NYlLZ1ZTdpLeDToxfXhMlWTTaIVbV6XUeY9ImLxU3KT7evxfqNkL9pIdTCxe_P5QPSvHxHREPW1lqbre7R3V9bFlKQ8-p4IuChNxyXj2gnklFXsanb3nC8mXuEO9me4r11eBpjjS-7SDvM7r442kHYgyoutjc0FwPsxVsx4kxPvIWFepLMqNX8uITGR4IyhW_jMCYC-GTHukGElx7OFoZdxLyKJpNXbrhG_QLIr3GQw8-OyGdt7uFClNMNVfl53nKQvc4FfAIpbPwYnTiGAskDyCeq9J8rES-k85AYkAMvV2KC0bkfeMOg982h-BE6F7CHvIjVWTX7TvByPvYqsPVdF4epc6lfq3i7EExxZIE9R6WLXdAGgJwmneNUe6cOYy2bsnWx-jpImsSls20aQdY7a8qpVVWsKm7kkPIKpmLSV8dwCl6E027K0oURMa-_8yHt6sLpkb4YwXWBkev3wqwV4AdckCoqoa5cfKGEr8rLwa0TF-N7OMoVdcT-zemXx-6TCxud9vt1eytAVu0967RwShIDM8EG_hYbjbx6QUkdjmBJ3cD2t7AZ1g3Mid1ephA49yDtvxYhs1DSzWCXn3QLDxGI2zfZxoECfJLa2hYZRupEs1Z75nULMeFrIQOwNfQRQxzuE9hO_CMyQ-zpxcKmuayzvjw2cTagNzlri19Q1t8SC-jE2n3v3FZHrRTxfnwlsZYACnJHPp5m5frOMdkQ70BNs4bU5L608vo2O6acws5PNwewRZ-XcJ5qD-R8tgmadnIo0kGq-5Q0tZB_yVmPvVagLpy4W-sbQ3Zz9ik63rejc9W_nkghgF6NaHGaJLLcy7-OPaO_E9BkkkFli2odITsArDUEIxdIyBQo7zYJyWOk1FX5nulzKGtcpuAfMSyTFbbYaWKeHnxptspuc_kv-Fkr6rpPoVyahOR2t3q0vl53XSJFQznx27-ExyyiBaERCeZdsg-T_DU2FGx1t1f21jPliQKfDPhN4I0dfCjByChfcamzsgo7eHBjVcLIzmNtDCVpouWADM7clT5PhgDOcBQu5r6lSSjDhTus8aygxjxdhhbtM74s9acvTViRC7WR3lzlBVkr001hHPEKxk8t32Mcezpf57r8uC7fVgU5Pji-qx41axYvAUyreVQlVpvTdWHL9ry1y2CS_2gytO5D-yvzPw7ApUrbEORKcPW5_lRx8mGA3roQhjT7fmpFl-VM61sZ9LGD8SbTFKO-FtKvAQVz-PeMct98NpSOz38nrDVrUICbPhUecfHx8uC8AdKwXk9MIw6dkRrwu7qSB89k1S_FyOpF4txOLsPUhGVVu5HDH0e2uodoxO-vrKSfxOF7XXA3PLgb6PKQuO5ngqN3uRkpGxI3vkozqUtaUYPqguSnyRuOr05zRctOENWRCDcFVk-9xSZCeuFDunX2Gfgd1NhxrRc84FIMaxKq0zW0jt2XX4qzxlA3EcMGR0sC7r5pq0hWHXsGZyIrFn8mDgHJSvDvnNZrADNhwlBUl9ZU3aCFqzaso5LYrBNqmJGh0URz0QBP3hWSK-5t4ftMpFGHBxejI7yVpRjxH1ulS5nKlYFIMU02hbBk7ECZ_4ULkMXeCapwjFicWzZ4TIfoNjjyxjjxwKMNCsGq4GWd1zndZwmLW4yr7_QinSMlaA4oGQwxmvSCCmXX0QYq39AUSCsSCxNyYi5hG0eLo5FSRgILY2IrAPPN9cjeGls6qnXJ7TwU389HPM5svDsMfbU7rIYkbYx0J-p9gg2FRt61x46FerPpo99NGuDkw7UvTxOqOb2tVbWrkqKn6GDRJSI7qLwm9_0Z88pjpL4BcG_5imgK8DnzMATxuqv2ie4VsKf0WXfsAqK74SM9-8xb5EIjvAna0KjyMSakekuzPKhIPSuMa8zN-pkf8aY6T72ZxLM59bBQCUT0SJgmbhKN-fmhb8f4JmLSMMygSNmJa1Gb-pVBXuDBhtKjO-Oau5R10d6IzrgFX8Fe1Z10s8Y9HUE29yooJbQDHT_7DFgOhPvH-zjxCXpUJiagEzwHh8qnw3rwmzpAntevvhqczsf7r0L34HAPrOFsd6t5818liErUK-Re4vtRcVfFaZTFIMNWHsZQkAsFARrEQyL-5-sBYvczdlzBVY4_Qtu7Bc1Cu_ge6JpIOaj-tmZmDpF3xWFWP69PnSyWzYhZzUje_X9w8VzQAIFz5Ti03VOZcyw0DAmwHKTtHGis4m1IM-7DwG4xLok_bWDt8w5zzjTEpcPSHtD8N8y-IWW1qC1ETtM3D0NlF6j1IGeNcr-uBfXPHvEqQp-4gjMru7qdaQOUawV4r0RRUZXFgkNWyoW7g9qDF34QBqpdxvLvLoND7vczY2bIk6rMzzndfgiMFSlRW0x1VBcZ-BhoUwAgyLLHYEtfz2tjE7nC9E6e6asvVXSYG1-CNEjvOkckBqL6fyfJuk3PB3T_6CRotQwEbqtB0VWjsDnn3P8dvP9xjgnrKHDSMGH0ikrckMfQOsbd4ljlWmV0ixGWQKoCNP1-0pyIj1GkY867XQUzguQG9-5TOktwaGZVUudWBeO_PfFUemn3TATSzzg4XoKJ3f5dorT7Evq0KwPmeAxuOjM7ESC1UtfwoXy5qOW-TWEzETqhZQoP-8bjcPT8UVQuUSvZzJiW8ezTOL66rboL_ABUwf86_lHQucPWOOK2PwePNR_VKg2lTyBrjg358uzZ33etOEhxecRef_8Oo6zo_aewUDNPzfXBwmUveOkutDyqAYyZf65tBLpvpfrqfH0F7bEwEyvZVuJNA4IpeENfDD-OqzbBUBmL0JHPvUYles2x2Zu58huMcOqXDyKLck6Hae9VnQHaUBFwlWIPltR1SmvD5bSk7eQnVIWbDa9LpVxIxs9uMuY4WlAX_yxmVZVyWbP0zBiJMJtfFTweqVExjAZlSSMoDyxRseYioqvefa2yL_opx7xuY0eTI20EgTPdfInRPGGcZjupkWRQHbImhmGxxCyoIvAi_pbGMLIHylSfUNGvuVVpYgiWpX6X2f0T220SWIRsQOCupYhZWuX0JT6rQd7Cc-llWwU7B3FKa6-3tLBkX5Lj3p74cnFRVk9X0v0VdkzlCKYE3SqXXn55rEk9JSvo1kOlW78LbrqDRAModAeAnG1-C9_dnhDDxipqmJg_rSdlZJzGXapO9LlMcjbwGGQvN0WOWxNJgd4WEFVagWbMTDJm5EyNaduXPmv1kp7Pj4odfBFAdaDe6ofPt5wmZz7G7djl8CR-nR6V-Mkv8Z-XQGp2PRA1Utmb2rF1T5i-p7tRfDleYs3hNMFjUdSsyZLun5t1WHv85LpBwNb9AlE-iZG13J-y4zYc1Z2oOdfX4-mOHJvdTLXKG429Oao6pks5MS3TKO4txYBRChfV09kk3MoXg5sWCw8J3UZL4kAGcpv06aXdlSlbaZArEhIcg2b1a629sY_Tsi4WId1NNH_ByDTxOlaujFPDEfISUob71p-f2JN_uZtHKj4QQkLcIZCHVeYnUGjXQHx9NObjvDi4_JwbGzxcc70T_OtUMYJ__tdjaOmhUzDat09cTFPIVWZ0Tvg7eaOI03Ax8J3EBSvcFdGs_hPwyLKumCAfXQk1fa0BKeyGT9PQy3wvF_XoFXT-e6A9htslYIi7NKV1gcASnOUWlZyjRr9BCwtjV7tNw-1GspH64ltuKIDlwXDQn21htsidFrKKAiNjprvgJXPc-pDPJNbfhz3CBKX9eHLGl7R25QIJjg4i9hSRTReRKvUGsR2ks7640E-PAU5aLBAo4aQl7rmdH0pZ1Ean04oBp74A8IcXJUPKz1PwwoKSjATHUafImcm4sgA0Jdp7md2HqBhJQf2uUZii93d8afQd810yPRfbBqCCgESR6T15wGZGcL6ouHqwrl6cz6P0qy6R7JZs--Y1Y5rC3pP2sjxEV7CAxM0WlseupaMt3Jzx0rCBtgI0_VBgz_2iw-v3CYBmgKdYnn7tq4ccP98AJd6pWHui4wdGi5kSK2SzjH4sQzDiVEk6H7UNaN1cQf7Nupl40Xvz5WKrRGZX34YWFFuE6KRufl3x3f-IFmcvkBYHaUVbDeMzh0KlLcD0NeK1HkaktJlqQiZhveujq5-X_vhP25dDlUYscWEqCR_XrfLhPI48An6vRkr6Ar8gl1H4CENHnlzC3T78fzcSVhpTt9NF29WqL8wyGQOXY9Om0GH8LxRE9p-OXoVq8wOAX7qbKuVUoXd6IYrGfhE1JtWOp-BCJzqRBbyNKtREem0BZroLUaGX6uNGV0Pxs_I9Nq6LOHltPPXVoNmvvYsxagsdo5wuTQM9WD8lo8UetXOxvxWg_cxj8Gp8EA19635foqgzCAHKcY1pkBQpapk8US4AZ2VrHFkTwUal9KVznMA6OtOWeBhr3s_shPts9ngxSLDWI7n3PIm3mofLxGfYEyRVyzuHeORRaVmNCCVXtPmKQVa_t63-a4OUrCIAr4FSCNyBnzgBbrtcfoLjW1xSc9LJqrrKhj3_oCtJVDA2epPvUy9IymTPyP64CfWAGxldU-RA_zSHS21GsTLV37E3RhP6SlECgjaBk6asOnYlMSL4TRq6LWYUXUlmGvWn_JbhD3mHl-BCsXC81qI4GSXdP7rCpwVHp_2lT1Dzh7GblMBOsZ67PUqDw8nK2iKxqFDBknVaik-Ex8OTFz1n6vSIFA-ywyhjtIPDkrhJQj1YQhJmF5R_x6X4YJgCzWpDH5KJDh4auW1TwIr1DLFeuwUvPvvgX95LjH5RUnZEGm16yr7l19a5nzqhGE6od5R5yCKONUWPNPZUcjRjs4zM6Nr11VSouhGKMUw1UpbRtp0YC_C8wMth0Pqp2sGKRXCByv9OPHFDdzTZfME6tP3av9l0Gy9O49W_Hy9Dxu10P6f7KVszNbtrjZy7r6Lplnc_q7GguezBS-vM3cWgCFtrfUh8yYNSaommkTcnp2zzsaoLbeAzmO1qCVduA4Zm7-QlEf4uw9lEhe33TCVFR0upADCHrFhEzdEqkBr9u_pNMQZsu8e4iA87enEXKJh6MiWcRYU0yeNkkWsY7oYB82EadPPt4eukeU18elV1g_BzZPXVXZ6TbEVnfcZND1hVv3DetmDhHdPAoeXsccIx1AzqJ3sb324Yl6jSg1z5fnOv03mE5-zaXWvqvTSQrRi6k6ya1lXPL_GWO12uHeYSF2oSzpR8kcCAeDxKcbAlz2B-4QDC296u_POkAillyoJDGUSBgrbU0KvfkkVQMCe-091pCL_Q_V3n0IlvY3rKznz393OUgPB-0xjlQAbWl4wmusFeoIujSuWXSFevfXa8kV3oII2db4HbOBUkapzMTz-nTW2b43zQgKsM_O_HW83BVLDjW7rb1e_Nwas_eWRK5JlLHNidl0xTmhh1XHiWxx0_cduApNd9XPIWlAa55eqeGJ77VZ47abGSYgsS83Rmy2oLxX6YALZGe5Xtw1qtzlzP3mwuxOSVKmdRc6MG3MbS7ycboq83eMwEEmgCrZJzJy4iXdKjQdwa_PPU-K-ChE0-cf33D1ma_5I5i5lJ72oeqxAnAQWHUm2nOSfN7ffcm2q_h4SGUFlR36vKsqMglwy5zFowqs0bo0Kkrf7KDa2CnJTAgspTS\",\"video_metadata\":null}],\"role\":\"model\"},\"citation_metadata\":null,\"finish_message\":null,\"token_count\":null,\"finish_reason\":\"STOP\",\"grounding_metadata\":null,\"avg_logprobs\":null,\"index\":0,\"logprobs_result\":null,\"safety_ratings\":null,\"url_context_metadata\":null}],\"create_time\":null,\"model_version\":\"gemini-3.5-flash\",\"prompt_feedback\":null,\"response_id\":\"awIQauPZJZjNkdUPr6iIqAw\",\"usage_metadata\":{\"cache_tokens_details\":null,\"cached_content_token_count\":null,\"candidates_token_count\":147,\"candidates_tokens_details\":null,\"prompt_token_count\":1529,\"prompt_tokens_details\":[{\"modality\":\"TEXT\",\"token_count\":454},{\"modality\":\"IMAGE\",\"token_count\":1075}],\"thoughts_token_count\":1346,\"tool_use_prompt_token_count\":null,\"tool_use_prompt_tokens_details\":null,\"total_token_count\":3022,\"traffic_type\":null},\"automatic_function_calling_history\":[],\"parsed\":{\"type\":{\"type\":\"Reference\"},\"author\":{\"last_name\":\"Montaghemi\",\"first_name\":\"Nourhomadeh\"},\"publication\":{\"title\":\"Sonnenbichler - Montaghemi, Nourhomadeh\",\"year\":\"\",\"place\":null,\"pages\":null,\"publisher\":null,\"format\":null,\"editor\":null},\"library_reference\":{\"shelfmark\":null,\"subjects\":null}}}"}, "scoring": {"f1_micro": 0.8708191468055277, "f1_macro": 0.8603041825095057, "micro_precision": 0.843289371605896, "micro_recall": 0.9002070393374741, "total_instances": 263, "total_tp": 2174, "total_fp": 404, "total_fn": 241, "cost_summary": {"total_input_tokens": 406240, "total_output_tokens": 49594, "total_tokens": 455834, "input_cost_usd": 0.6093599999999991, "output_cost_usd": 0.4463459999999999, "total_cost_usd": 5.18811, "total_reasoning_tokens": 459156, "reasoning_cost_usd": 4.132404}}, "pricing": {"bucket_date": "2026-05-22", "age_days": 0, "input_price_per_million": 1.5, "output_price_per_million": 9.0}, "timing": {"total_response_s": 2346.33, "mean_response_s": 8.92, "slowest_response_s": 227.71, "inputs_timed": 263, "inputs_stored": 263, "span_s": 295.81}, "normalized_score": 86.03041825095058}