{"_id": {"$oid": "6a8456c1ef0a19c343227d61"}, "test_id": "T1545", "benchmark": "library_cards", "date": "2026-08-15", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "huggingface", "model": "thinkingmachines/Inkling-Small:deepinfra", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\n\n\n  \"type\": {\n    \"type\": \"Dissertation or thesis\"\n  },\n  \"author\": {\n    \"last_name\": \"Wolfsohn\",\n    \"first_name\": \"Wilhelm\"\n  },\n  \"publication\": {\n    \"title\": \"Ein Fall von Melaena neonatorum\",\n    \"year\": 1900,\n    \"place\": \"Leipzig\",\n    \"pages\": \"31\",\n    \"publisher\": \"B. Georgi\",\n    \"format\": \"8\u00b0\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"Med. Diss. v. (20. Jan.) 1900\",\n    \"subjects\": \"\"\n  }\n}", "model": "thinkingmachines/Inkling-Small", "provider": "huggingface", "finish_reason": "stop", "usage": {"input_tokens": 2663, "output_tokens": 1750, "total_tokens": 4413, "cached_tokens": 1792, "input_cost_usd": 0.00119835, "output_cost_usd": 0.0021, "estimated_cost_usd": 0.00329835}, "duration": 372.1789126396179, "timestamp": "2026-08-16T00:03:25.467463", "parsed": {"type": {"type": "Dissertation or thesis"}, "author": {"last_name": "Wolfsohn", "first_name": "Wilhelm"}, "publication": {"title": "Ein Fall von Melaena neonatorum", "year": 1900, "place": "Leipzig", "pages": "31", "publisher": "B. Georgi", "format": "8\u00b0"}, "library_reference": {"shelfmark": "Med. Diss. v. (20. Jan.) 1900", "subjects": ""}}, "conversation_id": "9c00b7ba-970f-47d7-92e1-80fce795458e", "score": {"f1_score": 0.95, "precision": 0.9, "recall": 1.0, "true_positives": 9, "false_positives": 1, "false_negatives": 0, "field_scores": {"author.last_name": {"response": "Wolfsohn", "ground_truth": "Wolfsohn", "score": 1.0}, "publication.title": {"response": "Ein Fall von Melaena neonatorum", "ground_truth": "Ein Fall von Melaena neatorum", "score": 0.9666666666666667}, "type.type": {"response": "Dissertation or thesis", "ground_truth": "Dissertation or thesis", "score": 1.0}, "publication.format": {"response": "8\u00b0", "ground_truth": "8\u00b0", "score": 1.0}, "publication.publisher": {"response": "B. Georgi", "ground_truth": "B. Georgi", "score": 1.0}, "library_reference.shelfmark": {"response": "Med. Diss. v. (20. Jan.) 1900", "ground_truth": null, "score": 0.0}, "publication.pages": {"response": "31", "ground_truth": "31", "score": 1.0}, "publication.year": {"response": 1900, "ground_truth": 1900, "score": 1.0}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "publication.place": {"response": "Leipzig", "ground_truth": "Leipzig", "score": 1.0}, "author.first_name": {"response": "Wilhelm", "ground_truth": "Wilhelm", "score": 1.0}}, "total_fields": 11}, "raw_response": "{\"id\":\"chatcmpl-RYoK4cubhIVc2zPE7SwniJ6q\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n\\n\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Dissertation or thesis\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Wolfsohn\\\",\\n    \\\"first_name\\\": \\\"Wilhelm\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Ein Fall von Melaena neonatorum\\\",\\n    \\\"year\\\": 1900,\\n    \\\"place\\\": \\\"Leipzig\\\",\\n    \\\"pages\\\": \\\"31\\\",\\n    \\\"publisher\\\": \\\"B. Georgi\\\",\\n    \\\"format\\\": \\\"8\u00b0\\\"\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"Med. Diss. v. (20. Jan.) 1900\\\",\\n    \\\"subjects\\\": \\\"\\\"\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"reasoning_content\":\"We need parse index card text. Let's read carefully.\\n\\nText:\\n\\\"Wolfsohn, Wilhelm, approb. Arzt aus Graudenz i. Westpr.: Aus d. Universit\u00e4tskinderklinik zu M\u00fcnchen. (Prof. H. v. Ranke.) Ein Fall von Melaena neonatorum [!]. Leipzig 1900. B. Georgi. (31 S.) 8\u00b0\\nLeipzig. Med. Diss. v. (20. Jan.) 1900\\\"\\n\\nWe need extract bibliographic info about historical dissertation from index card.\\n\\nCard Type: If contains note \\\"s.\\\" on separate line, it's Reference. Here we have \\\"(31 S.)\\\" but not \\\"s.\\\" on separate line? The rule: \\\"If a card contains the note \\\"s.\\\" on a separate line, it is a \\\"Reference\\\". Otherwise, it is a \\\"Dissertation or thesis\\\".\\\" We have \\\"(31 S.)\\\" within line, not separate line \\\"s.\\\". So type = Dissertation or thesis.\\n\\nAuthor: \\\"Wolfsohn, Wilhelm\\\" => last_name Wolfsohn, first_name Wilhelm.\\n\\nPublication:\\n- title: The main title of the work. Likely \\\"Ein Fall von Melaena neonatorum\\\" or maybe includes subtitle? The text: \\\"Aus d. Universit\u00e4tskinderklinik zu M\u00fcnchen. (Prof. H. v. Ranke.) Ein Fall von Melaena neonatorum [!].\\\" So title is \\\"Ein Fall von Melaena neonatorum\\\". Could also include \\\"Aus d. Universit\u00e4tskinderklinik zu M\u00fcnchen\\\" as part? Usually main title is \\\"Ein Fall von Melaena neonatorum\\\". Let's choose that.\\n- year: 1900 (from Leipzig 1900 and diss v. 1900). So integer 1900.\\n- place: Leipzig.\\n- pages: \\\"31\\\" (remove \\\" S.\\\" suffix). So \\\"31\\\".\\n- publisher: \\\"B. Georgi\\\" is publisher? Yes, B. Georgi is publishing house. So publisher \\\"B. Georgi\\\".\\n- format: \\\"8\u00b0\\\". So \\\"8\u00b0\\\".\\n\\nLibrary Reference:\\n- shelfmark: Often begins with \\\"Diss.\\\" or \\\"AT\\\", may be marked with \\\"Standort:\\\". Here we have \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\". That indicates it's a medical dissertation from Leipzig, but shelfmark? Could be \\\"Med. Diss. v. (20. Jan.) 1900\\\" but shelfmark likely missing. However maybe shelfmark is empty. But we might extract \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\" as shelfmark? The instruction: shelfmark: Often begins with \\\"Diss.\\\" or \\\"AT\\\", may be marked with \\\"Standort:\\\". Here it says \\\"Med. Diss.\\\" which contains Diss. So perhaps shelfmark is \\\"Med. Diss. v. (20. Jan.) 1900\\\" or \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\". Let's think. The line \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\" is likely the dissertation reference info, i.e., library reference shelfmark or note. Since shelfmark often begins with \\\"Diss.\\\", we can put \\\"Med. Diss. v. (20. Jan.) 1900\\\" as shelfmark, or include Leipzig? The place is already Leipzig. Let's put shelfmark: \\\"Med. Diss. v. (20. Jan.) 1900\\\". But maybe better \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\". However shelfmark is usually a call number. The text \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\" indicates location and dissertation info. Since we have place Leipzig already, shelfmark could be \\\"Med. Diss. v. (20. Jan.) 1900\\\". Let's use that.\\n\\n- subjects: Subject classifications or keywords. We have \\\"Melaena neonatorum\\\" maybe. But also \\\"approb. Arzt aus Graudenz i. Westpr.\\\" is author bio. Could extract subjects from content: maybe \\\"P\u00e4diatrie\\\" not present. The card might have no explicit subjects. So empty string.\\n\\nBut we need to follow schema: library_reference has shelfmark and subjects, both strings or null? In the provided schema at top, they are string or empty string. The JSON format requested uses \\\"string or empty string\\\". So use \\\"\\\" if missing.\\n\\nLet's decide shelfmark: \\\"Med. Diss. v. (20. Jan.) 1900\\\" seems appropriate because it contains Diss. Let's include \\\"Leipzig.\\\"? Actually shelfmark might be \\\"Diss.\\\" but here \\\"Med. Diss.\\\" is the type. I think shelfmark = \\\"Med. Diss. v. (20. Jan.) 1900\\\". Or maybe empty because it's not a shelfmark but a note. However rule says \\\"Often begins with 'Diss.' or 'AT', may be marked with 'Standort:'\\\". So if it begins with Diss., it's shelfmark. Here \\\"Med. Diss.\\\" contains Diss. but starts with Med. Could still be considered. Let's put shelfmark: \\\"Med. Diss. v. (20. Jan.) 1900\\\".\\n\\nWait, the first part \\\"Wolfsohn, Wilhelm, approb. Arzt aus Graudenz i. Westpr.: Aus d. Universit\u00e4tskinderklinik zu M\u00fcnchen. (Prof. H. v. Ranke.)\\\" is likely author info and affiliation. The title is \\\"Ein Fall von Melaena neonatorum\\\". Publication info: Leipzig 1900, B. Georgi, (31 S.) 8\u00b0. Then the last line is dissertation info: Leipzig. Med. Diss. v. (20. Jan.) 1900.\\n\\nThus library_reference shelfmark could be empty, and the dissertation info is already captured in type and publication. But the rule explicitly says shelfmark often begins with Diss. So likely we should extract \\\"Med. Diss. v. (20. Jan.) 1900\\\" as shelfmark.\\n\\nLet's craft JSON accordingly.\\n\\nCheck author: last_name \\\"Wolfsohn\\\", first_name \\\"Wilhelm\\\".\\n\\nPublication title: \\\"Ein Fall von Melaena neonatorum\\\". Should we include \\\"[!]\\\"? Probably not, ignore punctuation. So \\\"Ein Fall von Melaena neonatorum\\\".\\n\\nYear: 1900.\\n\\nPlace: \\\"Leipzig\\\".\\n\\nPages: \\\"31\\\".\\n\\nPublisher: \\\"B. Georgi\\\".\\n\\nFormat: \\\"8\u00b0\\\".\\n\\nLibrary reference shelfmark: \\\"Med. Diss. v. (20. Jan.) 1900\\\". Or \\\"Leipzig. Med. Diss. v. (20. Jan.) 1900\\\". Let's choose \\\"Med. Diss. v. (20. Jan.) 1900\\\" to avoid duplicating place.\\n\\nSubjects: \\\"\\\". Could also put \\\"Melaena neonatorum\\\" but that's title. Let's leave empty.\\n\\nMake sure to return ONLY JSON object, no extra text.\\n\\nOne more thing: The schema in the prompt says:\\n{\\n  \\\"type\\\": { ... },\\n  \\\"author\\\": { ... },\\n  \\\"publication\\\": { ... },\\n  \\\"library_reference\\\": { ... }\\n}\\n\\nWhere publication has title, year, place, pages, publisher, format. Year is integer in prompt but schema says string? The prompt says \\\"year: integer\\\". The schema says \\\"year\\\": {\\\"title\\\": \\\"Year\\\", \\\"type\\\": \\\"string\\\"}. We should follow prompt's instruction: year as integer. So 1900 without quotes.\\n\\nLet's produce final.\",\"name\":null}}],\"created\":1786831405,\"model\":\"thinkingmachines/Inkling-Small\",\"object\":\"chat.completion\",\"service_tier\":\"default\",\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":1750,\"prompt_tokens\":2663,\"total_tokens\":4413,\"completion_tokens_details\":null,\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":1792,\"cache_write_tokens\":null},\"estimated_cost\":0.002671149998208,\"reasoning_tokens\":1605}}"}, "scoring": {"f1_micro": 0.7188073394495413, "f1_macro": 0.6231178707224334, "micro_precision": 0.8056555269922879, "micro_recall": 0.6488612836438924, "total_instances": 263, "total_tp": 1567, "total_fp": 378, "total_fn": 848, "cost_summary": {"total_input_tokens": 508942, "total_output_tokens": 410502, "total_tokens": 919444, "input_cost_usd": 0.22902389999999956, "output_cost_usd": 0.49260239999999994, "total_cost_usd": 0.7216262999999995}}, "normalized_score": 62.31178707224334}