{"_id": {"$oid": "6ab1687f1481417a2aae5109"}, "test_id": "T1770", "benchmark": "library_cards", "date": "2026-09-16", "tags": {"document-type": ["index-card"], "writing": ["typed", "printed", "handwritten"], "century": [20, 19], "language": ["de", "fr", "en", "la", "el", "fi", "sv", "pl"], "layout": ["index"], "entry-type": ["bibliographic"], "task": ["information-extraction"]}, "contributors": ["gabriel_muller", "maximilian_hindermann"], "hidden": false, "config": {"provider": "deepseek", "model": "deepseek-flash", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with keyword knowledge", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "Extract the bibliographic information about a historical dissertation from this index card and return it as a structured JSON object with the following exact format:\n\n```json\n{\n  \"type\": {\n    \"type\": \"Dissertation or thesis\" OR \"Reference\"\n  },\n  \"author\": {\n    \"last_name\": \"string\",\n    \"first_name\": \"string\"\n  },\n  \"publication\": {\n    \"title\": \"string\",\n    \"year\": integer,\n    \"place\": \"string or empty string\",\n    \"pages\": \"string or empty string\",\n    \"publisher\": \"string or empty string\",\n    \"format\": \"string or empty string\"\n  },\n  \"library_reference\": {\n    \"shelfmark\": \"string or empty string\",\n    \"subjects\": \"string or empty string\"\n  }\n}\n```\n\nEXTRACTION RULES:\n1. **Card Type**: If a card contains the note \"s.\" on a separate line, it is a \"Reference\". Otherwise, it is a \"Dissertation or thesis\".\n\n2. **Author**: Extract last_name and first_name. If only one name is given, put it in last_name and leave first_name empty.\n\n3. **Publication**:\n   - title: The main title of the work\n   - year: Publication year as integer\n   - place: Publication place\n   - pages: Page count (remove \" S.\" suffix if present)\n   - publisher: Publishing house/institution\n   - format: Usually \"8\u00b0\", \"8'\", or \"4\u00b0\" - single value only\n\n4. **Library Reference**:\n   - shelfmark: Often begins with \"Diss.\" or \"AT\", may be marked with \"Standort:\"\n   - subjects: Subject classifications or keywords\n\n5. **Missing Information**: Use empty string \"\" for missing text fields, omit year fields entirely if not present.\n\n6. **Ignore**: Disregard any information that doesn't fit into these categories.\n\nReturn ONLY the JSON object, no additional text or explanation.", "results": {"text": "{\"type\":{\"type\":\"Reference\"},\"author\":{\"last_name\":\"Montaghemi\",\"first_name\":\"Nourhomadeh\"},\"publication\":{\"title\":\"Sonnenbichler - Montaghemi, Nourhomadeh\",\"year\":\"\",\"place\":\"\",\"pages\":\"\",\"publisher\":\"\",\"format\":\"\",\"editor\":null},\"library_reference\":{\"shelfmark\":\"\",\"subjects\":\"\"}}", "model": "deepseek-flash", "provider": "deepseek", "finish_reason": "stop", "usage": {"input_tokens": 1598, "output_tokens": 559, "total_tokens": 2157, "input_cost_usd": 0.0004794, "output_cost_usd": 0.0006708, "estimated_cost_usd": 0.0011502}, "duration": 3.671879768371582, "timestamp": "2026-09-16T17:06:04.665337", "parsed": {"type": {"type": "Reference"}, "author": {"last_name": "Montaghemi", "first_name": "Nourhomadeh"}, "publication": {"title": "Sonnenbichler - Montaghemi, Nourhomadeh", "year": "", "place": "", "pages": "", "publisher": "", "format": ""}, "library_reference": {"shelfmark": "", "subjects": ""}}, "conversation_id": "814096d4-6419-4325-bfb7-d19e4de6f402", "score": {"f1_score": 0.29, "precision": 0.25, "recall": 0.3333333333333333, "true_positives": 1, "false_positives": 3, "false_negatives": 2, "field_scores": {"publication.place": {"response": null, "ground_truth": null, "score": 1.0}, "library_reference.shelfmark": {"response": null, "ground_truth": null, "score": 1.0}, "author.last_name": {"response": "Montaghemi", "ground_truth": "Sonnenbichler", "score": 0.34782608695652173}, "publication.editor": {"response": null, "ground_truth": null, "score": 1.0}, "publication.publisher": {"response": null, "ground_truth": null, "score": 1.0}, "publication.format": {"response": null, "ground_truth": null, "score": 1.0}, "publication.title": {"response": "Sonnenbichler - Montaghemi, Nourhomadeh", "ground_truth": "Montaghemi, Nourhomadeh", "score": 0.7419354838709676}, "library_reference.subjects": {"response": null, "ground_truth": null, "score": 1.0}, "author.first_name": {"response": "Nourhomadeh", "ground_truth": null, "score": 0.0}, "publication.pages": {"response": null, "ground_truth": null, "score": 1.0}, "publication.year": {"response": null, "ground_truth": null, "score": 1.0}, "type.type": {"response": "Reference", "ground_truth": "Reference", "score": 1.0}}, "total_fields": 12}, "raw_response": "{\"id\":\"fd35023c-fc96-4e74-a752-bcbb9667b37d\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"type\\\": {\\n    \\\"type\\\": \\\"Reference\\\"\\n  },\\n  \\\"author\\\": {\\n    \\\"last_name\\\": \\\"Montaghemi\\\",\\n    \\\"first_name\\\": \\\"Nourhomadeh\\\"\\n  },\\n  \\\"publication\\\": {\\n    \\\"title\\\": \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\",\\n    \\\"year\\\": \\\"\\\",\\n    \\\"place\\\": \\\"\\\",\\n    \\\"pages\\\": \\\"\\\",\\n    \\\"publisher\\\": \\\"\\\",\\n    \\\"format\\\": \\\"\\\"\\n  },\\n  \\\"library_reference\\\": {\\n    \\\"shelfmark\\\": \\\"\\\",\\n    \\\"subjects\\\": \\\"\\\"\\n  }\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"reasoning_content\":\"The card contains \\\"s.\\\" on a separate line, so it's a \\\"Reference\\\".\\n\\nAuthor: Montaghemi, Nourhomadeh \u2014 last name Montaghemi, first name Nourhomadeh.\\n\\nSecond line: \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\" \u2014 this appears to be a reference to another author/publication? Actually the card shows author and then \\\"s.\\\" meaning see reference: \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\". Hmm, the title... For a reference card, the title would be the reference itself.\\n\\nGiven the schema requires a title, I'll use \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\" as title. Year is required as a string \u2014 but no year present. The schema requires year (required field, type string). Rules say omit year fields entirely if not present. But the schema requires it. Hmm, conflict. The rules say \\\"Use empty string \\\"\\\" for missing text fields, omit year fields entirely if not present.\\\" But publication requires title and year. I'll include year as \\\"\\\" to be safe? Actually the JSON schema requires year. I'll put \\\"\\\" \u2014 but wait, the instruction says omit year fields entirely if not present. That would break the schema. Safer to include with empty string, giving valid JSON matching schema. Hmm, but maybe better to follow the required schema. I'll include \\\"year\\\": \\\"\\\".\\n\\nActually, let's reconsider: maybe put year empty string. Yes.\\n\\nTitle: \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\" \u2014 this is a reference. Hmm, could the title be \\\"Montaghemi, Nourhomadeh\\\"? The main line is the author name, then \\\"s.\\\" (siehe = see), then the reference.\\n\\nI'll set title to \\\"Sonnenbichler - Montaghemi, Nourhomadeh\\\".\\n\\nOther fields empty.\\n\\nShelfmark: none visible \u2014 the black dot could be a hole punch. Empty.\\n\\nSubjects: empty.\"}}],\"created\":1789571161,\"model\":\"deepseek-flash\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"usage\":{\"completion_tokens\":559,\"prompt_tokens\":1598,\"total_tokens\":2157,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":null,\"reasoning_tokens\":432,\"rejected_prediction_tokens\":null},\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":0},\"prompt_cache_hit_tokens\":0,\"prompt_cache_miss_tokens\":1598}}"}, "scoring": {"f1_micro": 0.8672459082642958, "f1_macro": 0.859467680608365, "micro_precision": 0.8468823993685872, "micro_recall": 0.8886128364389234, "total_instances": 263, "total_tp": 2146, "total_fp": 388, "total_fn": 269, "cost_summary": {"total_input_tokens": 425594, "total_output_tokens": 391448, "total_tokens": 817042, "input_cost_usd": 0.1276781999999996, "output_cost_usd": 0.46973760000000014, "total_cost_usd": 0.5974157999999997}}, "pricing": {"bucket_date": "2026-09-16", "age_days": 0, "input_price_per_million": 0.3, "output_price_per_million": 1.2}, "timing": {"total_response_s": 2143.98, "mean_response_s": 8.15, "slowest_response_s": 41.99, "inputs_timed": 263, "inputs_stored": 263, "span_s": 136.32}, "normalized_score": 85.9467680608365}