{"_id": {"$oid": "6a671d25338dadb0be4e82c9"}, "test_id": "T0962", "benchmark": "library_cards", "date": "2026-04-21", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "openrouter", "model": "qwen/qwen3.5-plus-02-15", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"Sydow\",\"first_name\":\"Harald\"},\"publication\":{\"title\":\"Elektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)\",\"year\":\"\",\"place\":\"Kiel\",\"pages\":\"387-407\",\"publisher\":\"\",\"format\":\"8'\",\"editor\":\"\"},\"library_reference\":{\"shelfmark\":\"Diss.1969,6496\",\"subjects\":\"2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss\"}}", "model": "qwen/qwen3.5-plus-20260216", "provider": "openrouter", "finish_reason": "stop", "usage": {"input_tokens": 1819, "output_tokens": 662, "total_tokens": 2481, "input_cost_usd": 0.00047294000000000004, "output_cost_usd": 0.0010327200000000002, "estimated_cost_usd": 0.0015056600000000002}, "duration": 13.172625541687012, "timestamp": "2026-04-21T13:54:59.409370", "parsed": {"type": {"type": "Dissertation or thesis"}, "author": {"last_name": "Sydow", "first_name": "Harald"}, "publication": {"title": "Elektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)", "year": "", "place": "Kiel", "pages": "387-407", "publisher": "", "format": "8'", "editor": ""}, "library_reference": {"shelfmark": "Diss.1969,6496", "subjects": "2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss"}}, "conversation_id": "438aee09-01b8-4e18-bccc-fbd1916e4ca1", "score": {"f1_score": 0.95, "precision": 1.0, "recall": 0.9, "true_positives": 9, "false_positives": 0, "false_negatives": 1, "field_scores": {"publication.format": {"response": "8'", "ground_truth": "8'", "score": 1.0}, "publication.place": {"response": "Kiel", "ground_truth": "Kiel", "score": 1.0}, "publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.publisher": {"response": null, "ground_truth": null, "score": 1.0}, "publication.year": {"response": null, "ground_truth": 1968, "score": 0.0}, "type.type": {"response": "Dissertation or thesis", "ground_truth": "Dissertation or thesis", "score": 1.0}, "publication.title": {"response": "Elektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)", "ground_truth": "Elektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)", "score": 1.0}, "author.first_name": {"response": "Harald", "ground_truth": "Harald", "score": 1.0}, "library_reference.subjects": {"response": "2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss", "ground_truth": "2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss", "score": 1.0}, "author.last_name": {"response": "Sydow", "ground_truth": "Sydow", "score": 1.0}, "publication.pages": {"response": "387-407", "ground_truth": "387-407", "score": 1.0}, "library_reference.shelfmark": {"response": "Diss.1969,6496", "ground_truth": "Diss.1969,6496", "score": 1.0}}, "total_fields": 12}, "raw_response": "{\"id\":\"gen-1776772486-EbJ1WLz5SCxwFxwQsoOr\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Sydow\\\",\\n    \\\"first_name\\\": \\\"Harald\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Elektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)\\\",\\n    \\\"year\\\":  \\\"\\\",\\n    \\\"place\\\": \\\"Kiel\\\",\\n    \\\"pages\\\": \\\"387-407\\\",\\n    \\\"publisher\\\": \\\"\\\",\\n    \\\"format\\\": \\\"8'\\\"\\n  ,\\n  \\\"editor\\\": \\\"\\\"\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"Diss.1969,6496\\\",\\n    \\\"subjects\\\": \\\"2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss\\\"\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"Sydow\",\"first_name\":\"Harald\"},\"publication\":{\"title\":\"Elektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)\",\"year\":\"\",\"place\":\"Kiel\",\"pages\":\"387-407\",\"publisher\":\"\",\"format\":\"8'\",\"editor\":\"\"},\"library_reference\":{\"shelfmark\":\"Diss.1969,6496\",\"subjects\":\"2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss\"}},\"reasoning\":\"\uff0c\u6211\u73b0\u5728\u9700\u8981\u5904\u7406\u7528\u6237\u63d0\u4f9b\u7684\u8fd9\u4e2a\u7d22\u5f15\u5361\uff0c\u63d0\u53d6\u51fa\u7ed3\u6784\u5316\u7684JSON\u6570\u636e\u3002\u9996\u5148\uff0c\u6211\u5f97\u4ed4\u7ec6\u9605\u8bfb\u7528\u6237\u7ed9\u7684\u6307\u4ee4\u548c\u793a\u4f8b\uff0c\u786e\u4fdd\u81ea\u5df1\u5b8c\u5168\u7406\u89e3\u6bcf\u4e2a\u6b65\u9aa4\u7684\u8981\u6c42\u3002\\n\\n\u9996\u5148\uff0c\u7528\u6237\u8981\u6c42\u6839\u636e\u7d22\u5f15\u5361\u7684\u5185\u5bb9\u5224\u65ad\u662f\u201cDissertation or thesis\u201d\u8fd8\u662f\u201cReference\u201d\u3002\u89c4\u5219\u91cc\u63d0\u5230\uff0c\u5982\u679c\u5361\u7247\u4e0a\u6709\u5355\u72ec\u7684\u201cs.\u201d\u884c\uff0c\u5c31\u662fReference\uff0c\u5426\u5219\u662f\u8bba\u6587\u3002\u73b0\u5728\u770b\u63d0\u4f9b\u7684\u56fe\u7247\u5185\u5bb9\uff0c\u91cc\u9762\u6ca1\u6709\u5355\u72ec\u7684\u201cs.\u201d\u884c\uff0c\u6240\u4ee5\u7c7b\u578b\u5e94\u8be5\u662f\u201cDissertation or thesis\u201d\u3002\\n\\n\u63a5\u4e0b\u6765\u662f\u4f5c\u8005\u90e8\u5206\u3002\u5361\u7247\u4e0a\u5199\u7684\u662f\u201cSydow, Harald.\u201d\uff0c\u6240\u4ee5\u59d3\u6c0f\u662fSydow\uff0c\u540d\u5b57\u662fHarald\u3002\u8fd9\u91cc\u8981\u6ce8\u610f\uff0c\u5982\u679c\u53ea\u6709\u4e00\u4e2a\u540d\u5b57\u7684\u8bdd\uff0c\u59d3\u6c0f\u653elast_name\uff0c\u540d\u5b57\u7559\u7a7a\uff0c\u4f46\u8fd9\u91cc\u6709\u4e24\u4e2a\u540d\u5b57\uff0c\u6240\u4ee5\u6ca1\u95ee\u9898\u3002\\n\\n\u7136\u540e\u662f\u51fa\u7248\u4fe1\u606f\u3002\u6807\u9898\u662f\u201cElektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)\u201d\u3002\u5e74\u4efd\u662f1968\uff0c\u5730\u70b9\u662fKiel\u3002\u9875\u6570\u90e8\u5206\u6709\u201cS.387-407\u201d\uff0c\u9700\u8981\u53bb\u6389\u201cS.\u201d\uff0c\u6240\u4ee5\u662f387-407\u3002\u51fa\u7248\u793e\u8fd9\u91cc\u53ef\u80fd\u6ca1\u6709\u660e\u786e\u63d0\u5230\uff0c\u4f46\u540e\u9762\u6709\u201cZeitschrift f\u00fcr Zellforschung,88\u201d\uff0c\u53ef\u80fd\u5c5e\u4e8e\u671f\u520a\uff0c\u4f46\u6839\u636e\u89c4\u5219\uff0c\u51fa\u7248\u793e\u53ef\u80fd\u7559\u7a7a\u3002\u683c\u5f0f\u662f\u201c8'\u201d\uff0c\u6240\u4ee5\u586b\u8fdb\u53bb\u3002\\n\\n\u56fe\u4e66\u9986\u53c2\u8003\u90e8\u5206\uff0cshelfmark\u662f\u201cDiss.1969,6496\u201d\uff0c\u56e0\u4e3a\u524d\u9762\u6709\u201cStandort:\u201d\u3002\u4e3b\u9898\u5206\u7c7b\u662f\u201c2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss\u201d\uff0c\u6240\u4ee5subjects\u586b\u8fd9\u4e2a\u3002\\n\\n\u9700\u8981\u68c0\u67e5\u662f\u5426\u6709\u9057\u6f0f\u7684\u4fe1\u606f\uff0c\u6bd4\u5982\u51fa\u7248\u793e\u662f\u5426\u5e94\u8be5\u586b\u201cZeitschrift f\u00fcr Zellforschung\u201d\uff0c\u4f46\u6839\u636e\u89c4\u5219\uff0c\u53ef\u80fd\u5c5e\u4e8e\u671f\u520a\u540d\u79f0\uff0c\u800c\u51fa\u7248\u793e\u53ef\u80fd\u6ca1\u6709\u660e\u786e\u7ed9\u51fa\uff0c\u6240\u4ee5\u7559\u7a7a\u3002\u53e6\u5916\uff0c\u683c\u5f0f\u662f\u5426\u6b63\u786e\uff0c\u7528\u6237\u4f8b\u5b50\u4e2d\u7684\u683c\u5f0f\u662f\u201c8'\u201d\uff0c\u6240\u4ee5\u6b63\u786e\u3002\\n\\n\u6700\u540e\uff0c\u786e\u4fdd\u6240\u6709\u5b57\u6bb5\u90fd\u7b26\u5408JSON Schema\u7684\u8981\u6c42\uff0c\u7279\u522b\u662f\u5fc5\u586b\u5b57\u6bb5\u662f\u5426\u5b58\u5728\uff0c\u7c7b\u578b\u662f\u5426\u6b63\u786e\u3002\u6bd4\u5982\u5e74\u4efd\u662f\u6574\u6570\uff0c\u5176\u4ed6\u5b57\u7b26\u4e32\u6216\u7a7a\u5b57\u7b26\u4e32\u3002\u786e\u8ba4\u65e0\u8bef\u540e\uff0c\u5c31\u53ef\u4ee5\u751f\u6210\u6b63\u786e\u7684JSON\u4e86\u3002\",\"reasoning_details\":[{\"type\":\"reasoning.text\",\"text\":\"\uff0c\u6211\u73b0\u5728\u9700\u8981\u5904\u7406\u7528\u6237\u63d0\u4f9b\u7684\u8fd9\u4e2a\u7d22\u5f15\u5361\uff0c\u63d0\u53d6\u51fa\u7ed3\u6784\u5316\u7684JSON\u6570\u636e\u3002\u9996\u5148\uff0c\u6211\u5f97\u4ed4\u7ec6\u9605\u8bfb\u7528\u6237\u7ed9\u7684\u6307\u4ee4\u548c\u793a\u4f8b\uff0c\u786e\u4fdd\u81ea\u5df1\u5b8c\u5168\u7406\u89e3\u6bcf\u4e2a\u6b65\u9aa4\u7684\u8981\u6c42\u3002\\n\\n\u9996\u5148\uff0c\u7528\u6237\u8981\u6c42\u6839\u636e\u7d22\u5f15\u5361\u7684\u5185\u5bb9\u5224\u65ad\u662f\u201cDissertation or thesis\u201d\u8fd8\u662f\u201cReference\u201d\u3002\u89c4\u5219\u91cc\u63d0\u5230\uff0c\u5982\u679c\u5361\u7247\u4e0a\u6709\u5355\u72ec\u7684\u201cs.\u201d\u884c\uff0c\u5c31\u662fReference\uff0c\u5426\u5219\u662f\u8bba\u6587\u3002\u73b0\u5728\u770b\u63d0\u4f9b\u7684\u56fe\u7247\u5185\u5bb9\uff0c\u91cc\u9762\u6ca1\u6709\u5355\u72ec\u7684\u201cs.\u201d\u884c\uff0c\u6240\u4ee5\u7c7b\u578b\u5e94\u8be5\u662f\u201cDissertation or thesis\u201d\u3002\\n\\n\u63a5\u4e0b\u6765\u662f\u4f5c\u8005\u90e8\u5206\u3002\u5361\u7247\u4e0a\u5199\u7684\u662f\u201cSydow, Harald.\u201d\uff0c\u6240\u4ee5\u59d3\u6c0f\u662fSydow\uff0c\u540d\u5b57\u662fHarald\u3002\u8fd9\u91cc\u8981\u6ce8\u610f\uff0c\u5982\u679c\u53ea\u6709\u4e00\u4e2a\u540d\u5b57\u7684\u8bdd\uff0c\u59d3\u6c0f\u653elast_name\uff0c\u540d\u5b57\u7559\u7a7a\uff0c\u4f46\u8fd9\u91cc\u6709\u4e24\u4e2a\u540d\u5b57\uff0c\u6240\u4ee5\u6ca1\u95ee\u9898\u3002\\n\\n\u7136\u540e\u662f\u51fa\u7248\u4fe1\u606f\u3002\u6807\u9898\u662f\u201cElektronenmikroskopische Untersuchung \u00fcber zytosomale Lamellenk\u00f6rper in den Eizellen des Ovars vom Igel (Erinaceus europaeus L.)\u201d\u3002\u5e74\u4efd\u662f1968\uff0c\u5730\u70b9\u662fKiel\u3002\u9875\u6570\u90e8\u5206\u6709\u201cS.387-407\u201d\uff0c\u9700\u8981\u53bb\u6389\u201cS.\u201d\uff0c\u6240\u4ee5\u662f387-407\u3002\u51fa\u7248\u793e\u8fd9\u91cc\u53ef\u80fd\u6ca1\u6709\u660e\u786e\u63d0\u5230\uff0c\u4f46\u540e\u9762\u6709\u201cZeitschrift f\u00fcr Zellforschung,88\u201d\uff0c\u53ef\u80fd\u5c5e\u4e8e\u671f\u520a\uff0c\u4f46\u6839\u636e\u89c4\u5219\uff0c\u51fa\u7248\u793e\u53ef\u80fd\u7559\u7a7a\u3002\u683c\u5f0f\u662f\u201c8'\u201d\uff0c\u6240\u4ee5\u586b\u8fdb\u53bb\u3002\\n\\n\u56fe\u4e66\u9986\u53c2\u8003\u90e8\u5206\uff0cshelfmark\u662f\u201cDiss.1969,6496\u201d\uff0c\u56e0\u4e3a\u524d\u9762\u6709\u201cStandort:\u201d\u3002\u4e3b\u9898\u5206\u7c7b\u662f\u201c2/Elektronen-Mikroskopie/Igel,Igelartige/ Diss\u201d\uff0c\u6240\u4ee5subjects\u586b\u8fd9\u4e2a\u3002\\n\\n\u9700\u8981\u68c0\u67e5\u662f\u5426\u6709\u9057\u6f0f\u7684\u4fe1\u606f\uff0c\u6bd4\u5982\u51fa\u7248\u793e\u662f\u5426\u5e94\u8be5\u586b\u201cZeitschrift f\u00fcr Zellforschung\u201d\uff0c\u4f46\u6839\u636e\u89c4\u5219\uff0c\u53ef\u80fd\u5c5e\u4e8e\u671f\u520a\u540d\u79f0\uff0c\u800c\u51fa\u7248\u793e\u53ef\u80fd\u6ca1\u6709\u660e\u786e\u7ed9\u51fa\uff0c\u6240\u4ee5\u7559\u7a7a\u3002\u53e6\u5916\uff0c\u683c\u5f0f\u662f\u5426\u6b63\u786e\uff0c\u7528\u6237\u4f8b\u5b50\u4e2d\u7684\u683c\u5f0f\u662f\u201c8'\u201d\uff0c\u6240\u4ee5\u6b63\u786e\u3002\\n\\n\u6700\u540e\uff0c\u786e\u4fdd\u6240\u6709\u5b57\u6bb5\u90fd\u7b26\u5408JSON Schema\u7684\u8981\u6c42\uff0c\u7279\u522b\u662f\u5fc5\u586b\u5b57\u6bb5\u662f\u5426\u5b58\u5728\uff0c\u7c7b\u578b\u662f\u5426\u6b63\u786e\u3002\u6bd4\u5982\u5e74\u4efd\u662f\u6574\u6570\uff0c\u5176\u4ed6\u5b57\u7b26\u4e32\u6216\u7a7a\u5b57\u7b26\u4e32\u3002\u786e\u8ba4\u65e0\u8bef\u540e\uff0c\u5c31\u53ef\u4ee5\u751f\u6210\u6b63\u786e\u7684JSON\u4e86\u3002\",\"format\":\"unknown\",\"index\":0}]},\"native_finish_reason\":\"stop\"}],\"created\":1776772486,\"model\":\"qwen/qwen3.5-plus-20260216\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":662,\"prompt_tokens\":1819,\"total_tokens\":2481,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":0,\"reasoning_tokens\":440,\"rejected_prediction_tokens\":null,\"image_tokens\":0},\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":0,\"cache_write_tokens\":0,\"video_tokens\":0},\"cost\":0.00150566,\"is_byok\":false,\"cost_details\":{\"upstream_inference_cost\":0.00150566,\"upstream_inference_prompt_cost\":0.00047294,\"upstream_inference_completions_cost\":0.00103272}},\"provider\":\"Alibaba\"}"}, "scoring": {"f1_micro": 0.8221611334320154, "f1_macro": 0.802851711026616, "micro_precision": 0.8401037165082109, "micro_recall": 0.8049689440993789, "total_instances": 263, "total_tp": 1944, "total_fp": 370, "total_fn": 471, "cost_summary": {"total_input_tokens": 478776, "total_output_tokens": 381523, "total_tokens": 860299, "input_cost_usd": 0.12448176000000023, "output_cost_usd": 0.5951758799999997, "total_cost_usd": 0.71965764}}, "normalized_score": 80.28517110266161}