{"_id": {"$oid": "6a8456c1ef0a19c343227d56"}, "test_id": "T1575", "benchmark": "library_cards", "date": "2026-08-15", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "huggingface", "model": "meta-models/Muse-Glimmer-30B:together", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Dissertation or thesis\"},\"author\":{\"last_name\":\"Teufel\",\"first_name\":\"Georg\"},\"publication\":{\"title\":\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung\",\"year\":\"1888\",\"place\":\"Strassburg\",\"pages\":\"56\",\"publisher\":\"Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)\",\"format\":\"8\u00b0\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\",\"subjects\":\"\"}}", "model": "meta-models/Muse-Glimmer-30B", "provider": "huggingface", "finish_reason": "stop", "usage": {"input_tokens": 1953, "output_tokens": 2383, "total_tokens": 4336, "cached_tokens": 1200, "input_cost_usd": 0.00068355, "output_cost_usd": 0.0035745000000000004, "estimated_cost_usd": 0.00425805}, "duration": 388.45527505874634, "timestamp": "2026-08-15T22:16:22.501772", "parsed": {"type": {"type": "Dissertation or thesis"}, "author": {"last_name": "Teufel", "first_name": "Georg"}, "publication": {"title": "Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung", "year": "1888", "place": "Strassburg", "pages": "56", "publisher": "Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)", "format": "8\u00b0"}, "library_reference": {"shelfmark": "Strassburg, Med. Fak., Inaug.-Diss. v. 1888", "subjects": ""}}, "conversation_id": "ce4c55b6-a436-4637-a5b6-68199c04359e", "score": {"f1_score": 0.95, "precision": 0.9, "recall": 1.0, "true_positives": 9, "false_positives": 1, "false_negatives": 0, "field_scores": {"publication.publisher": {"response": "Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)", "ground_truth": "Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)", "score": 1.0}, "publication.title": {"response": "Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung", "ground_truth": "Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung", "score": 1.0}, "publication.format": {"response": "8\u00b0", "ground_truth": "8\u00b0", "score": 1.0}, "author.last_name": {"response": "Teufel", "ground_truth": "Teufel", "score": 1.0}, "library_reference.shelfmark": {"response": "Strassburg, Med. Fak., Inaug.-Diss. v. 1888", "ground_truth": null, "score": 0.0}, "publication.pages": {"response": "56", "ground_truth": "56", "score": 1.0}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "type.type": {"response": "Dissertation or thesis", "ground_truth": "Dissertation or thesis", "score": 1.0}, "publication.place": {"response": "Strassburg", "ground_truth": "Strassburg", "score": 1.0}, "author.first_name": {"response": "Georg", "ground_truth": "Georg", "score": 1.0}, "publication.year": {"response": "1888", "ground_truth": 1888, "score": 1.0}}, "total_fields": 11}, "raw_response": "{\"id\":\"owLbxgC-7ArivV-a2badfcd0cf3cbf8\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Teufel\\\",\\n    \\\"first_name\\\": \\\"Georg\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung\\\",\\n    \\\"year\\\": \\\"1888\\\",\\n    \\\"place\\\": \\\"Strassburg\\\",\\n    \\\"pages\\\": \\\"56\\\",\\n    \\\"publisher\\\": \\\"Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)\\\",\\n    \\\"format\\\": \\\"8\u00b0\\\"\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\",\\n    \\\"subjects\\\": \\\"\\\"\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":[],\"reasoning\":\"Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\\n\\nWe need parse card.\\n\\nText:\\n\\nTeufel, Georg [app. Arzt]: Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung. . . . Strassburg, Univ.- Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel), 1888; 56 S. 8\u00b0\\n\\nStrassburg, Med. Fak., Inaug.-Diss. v. 1888\\n\\nCard Type: If a card contains the note \\\"s.\\\" on a separate line, it is a \\\"Reference\\\". Otherwise, it is a \\\"Dissertation or thesis\\\". Does card contain \\\"s.\\\" on a separate line? The text has \\\"56 S. 8\u00b0\\\". That's \\\"S.\\\" but not on separate line. The note \\\"s.\\\" on a separate line? No. So type is Dissertation or thesis.\\n\\nAuthor: Teufel, Georg [app. Arzt]. Last_name Teufel, first_name Georg. [app. Arzt] is maybe qualification, ignore? Probably just name.\\n\\nPublication:\\ntitle: Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung. ... The title appears broken across lines: \\\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung.\\\" The text shows \\\"congenitalen\\\" but OCR says \\\"congenitalon\\\"? Actually \\\"congenitalen\\\". We'll transcribe as in image: \\\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung.\\\" Might include ellipsis \\\"...\\\". Probably title without ellipsis. Use main title.\\n\\nyear: 1888\\n\\nplace: Strassburg\\n\\npages: 56 S. -> remove \\\" S.\\\" suffix if present. So pages: \\\"56\\\". Or \\\"56 S.\\\"? Rule: Page count (remove \\\" S.\\\" suffix if present). So \\\"56\\\".\\n\\npublisher: Univ.- Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel) . Might be \\\"Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)\\\". Could be publisher. Place is Strassburg, publisher is Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel). Might need clean.\\n\\nformat: 8\u00b0 . The text shows \\\"8\u00b0\\\" or \\\"8\u00b0\\\"? It says \\\"8\u00b0\\\". So format \\\"8\u00b0\\\".\\n\\nlibrary_reference:\\nshelfmark: Often begins with \\\"Diss.\\\" or \\\"AT\\\", may be marked with \\\"Standort:\\\". Here we have \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\". That's maybe shelfmark? Could be subjects? Hmm.\\n\\nThe library_reference shelfmark: Often begins with \\\"Diss.\\\" or \\\"AT\\\", may be marked with \\\"Standort:\\\". The line \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\" seems like a reference note. Could be shelfmark? Might be subjects? The schema has shelfmark and subjects. Could put shelfmark as \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\". Subjects empty.\\n\\nAlternatively subjects could be? No subjects listed.\\n\\nMaybe shelfmark empty, subjects = \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\". But rule says shelfmark often begins with \\\"Diss.\\\" or \\\"AT\\\", may be marked with \\\"Standort:\\\". This line contains \\\"Inaug.-Diss.\\\" So could be shelfmark.\\n\\nLet's put shelfmark: \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\". subjects empty string.\\n\\nThe JSON format required in prompt: \\n{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\" OR \\\"Reference\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"string\\\",\\n    \\\"first_name\\\": \\\"string\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"string\\\",\\n    \\\"year\\\": integer,\\n    \\\"place\\\": \\\"string or empty string\\\",\\n    \\\"pages\\\": \\\"string or empty string\\\",\\n    \\\"publisher\\\": \\\"string or empty string\\\",\\n    \\\"format\\\": \\\"string or empty string\\\"\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"string or empty string\\\",\\n    \\\"subjects\\\": \\\"string or empty string\\\"\\n  }\\n}\\n\\nBut the schema given later shows type as object with type field? Wait schema: WorkType has property type. Document has property type: {$ref WorkType}. So type is an object with type field. The example format shows \\\"type\\\": { \\\"type\\\": \\\"Dissertation or thesis\\\" }. So we need nested.\\n\\nAlso Publication year type in schema is string? The schema says year: type string. But prompt says year integer. Conflict. The schema definition says year: type string. The prompt says integer. The schema is authoritative? The instruction says return as structured JSON object with following exact format. The format shows year: integer. But schema shows year string. Hmm.\\n\\nThe schema provided in the MUST respond section: Publication properties: title string, year string, place anyOf string/null, pages anyOf string/null, publisher anyOf string/null, format anyOf string/null, editor anyOf string/null. Required title, year.\\n\\nSo year should be string? Might be integer but schema says string. Might need to follow schema. The prompt says year integer. Might be conflict. Usually JSON numbers are fine. Schema says string. Could output as number? Might fail validation. Safer to follow schema? The instruction says return ONLY JSON object, no additional text. The schema is given as MUST respond with valid JSON matching this exact schema. So we must follow schema.\\n\\nSchema: Publication year is string. So year should be \\\"1888\\\" as string? But prompt says integer. Hmm.\\n\\nAlso type is object with type field. So output:\\n\\n{\\n  \\\"type\\\": {\\\"type\\\": \\\"Dissertation or thesis\\\"},\\n  \\\"author\\\": {\\\"last_name\\\": \\\"Teufel\\\", \\\"first_name\\\": \\\"Georg\\\"},\\n  \\\"publication\\\": {\\\"title\\\": \\\"...\\\", \\\"year\\\": \\\"1888\\\"? or 1888? Schema says string. Might be string. Let's output as string? Could output as number and still be string? No.\\n\\nMaybe the schema's year type is string but they want integer. Could be a mistake. The initial format shows year integer. I'd lean to follow initial format description. The MUST respond schema might be auto-generated. The initial format is explicit.\\n\\nThe schema also has editor field not in format. Could omit.\\n\\nLibraryReference shelfmark anyOf string/null default null. subjects anyOf string/null default null.\\n\\nWe need empty string for missing. Schema allows null. Might use empty string.\\n\\nLet's produce JSON with type object.\\n\\nAuthor last_name Teufel, first_name Georg.\\n\\nPublication title: \\\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung\\\". Note the OCR says \\\"congenitalon\\\". Might be \\\"congenitalen\\\". The image shows \\\"congenitalon\\\". Actually \\\"congenitalon\\\" maybe typo in OCR. The word is \\\"congenitalen\\\". Let's assume correct German: \\\"congenitalen\\\". Might keep as in image: \\\"congenitalen\\\". The image text: \\\"congenitalon H\u00fcft-\\\". Could be \\\"congenitalen\\\". I'll use \\\"congenitalen\\\". Might be safe to transcribe exactly as visible: \\\"congenitalon\\\". Hmm.\\n\\nBetter to transcribe as seen: \\\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung\\\". I'll assume correct spelling.\\n\\nPlace: \\\"Strassburg\\\"\\n\\nPages: \\\"56\\\"\\n\\nPublisher: \\\"Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)\\\"\\n\\nFormat: \\\"8\u00b0\\\"\\n\\nLibrary reference shelfmark: \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\"\\nsubjects: \\\"\\\"\\n\\nOr subjects empty.\\n\\nIf shelfmark empty and subjects contains that? The line is not a subject classification. It's a note about dissertation. Could be shelfmark.\\n\\nI'll output shelfmark as that string, subjects as \\\"\\\".\\n\\nMake sure JSON valid.\\n\\nPotential nuance: publication place maybe \\\"Strassburg\\\". Publisher includes \\\"Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)\\\". Good.\\n\\nYear as integer 1888 per prompt. Let's output integer. Schema says string but maybe validator accepts number? Might be strict.\\n\\nThe schema says year type string. Hmm.\\n\\nThe instruction says \\\"year: integer\\\". I'll follow instruction over schema? The MUST respond says MUST respond with valid JSON matching this exact schema. That is higher priority.\\n\\nCould output year as string \\\"1888\\\". That matches schema. The prompt says integer but maybe they accept string.\\n\\nAlso type object: \\\"type\\\": {\\\"type\\\": \\\"Dissertation or thesis\\\"}.\\n\\nLet's produce final JSON.\\n\\nEnsure no extra text.\\n\\nOutput:\\n\\n{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Teufel\\\",\\n    \\\"first_name\\\": \\\"Georg\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Ueber einen Fall von multiplen Missbildungen mit besonderer Ber\u00fccksichtigung der congenitalen H\u00fcftluxation und deren operativer Behandlung\\\",\\n    \\\"year\\\": \\\"1888\\\",\\n    \\\"place\\\": \\\"Strassburg\\\",\\n    \\\"pages\\\": \\\"56\\\",\\n    \\\"publisher\\\": \\\"Univ.-Buchdr. v. J. H. Ed. Heitz (Heitz & M\u00fcndel)\\\",\\n    \\\"format\\\": \\\"8\u00b0\\\"\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"Strassburg, Med. Fak., Inaug.-Diss. v. 1888\\\",\\n    \\\"subjects\\\": \\\"\\\"\\n  }\\n}\\n\\nIf year must be integer, change to 1888 without quotes. Let's decide. The schema says year type string. I'll keep string.\\n\\nMaybe also include editor? No.\\n\\nReturn only JSON.\\n\\nLet's output.\"},\"token_ids\":null,\"routed_experts\":null,\"seed\":null}],\"created\":1786824957,\"model\":\"meta-models/Muse-Glimmer-30B\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"vllm-0.1.dev19059+g66826a22d-tp2-46311a71\",\"usage\":{\"completion_tokens\":2383,\"prompt_tokens\":1953,\"total_tokens\":4336,\"completion_tokens_details\":null,\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":1200,\"created_cache_tokens\":752,\"multimodal_tokens\":{\"image\":772}}},\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null,\"prompt\":[]}"}, "scoring": {"f1_micro": 0.8958034800409417, "f1_macro": 0.8894296577946768, "micro_precision": 0.8858299595141701, "micro_recall": 0.9060041407867495, "total_instances": 263, "total_tp": 2188, "total_fp": 282, "total_fn": 227, "cost_summary": {"total_input_tokens": 503142, "total_output_tokens": 664137, "total_tokens": 1167279, "input_cost_usd": 0.17609970000000022, "output_cost_usd": 0.9962055000000003, "total_cost_usd": 1.1723052000000003}}, "normalized_score": 88.94296577946767}