{"_id": {"$oid": "6a8456c1ef0a19c343227d6f"}, "test_id": "T1577", "benchmark": "blacklist_cards", "date": "2026-08-16", "tags": {"document-type": ["index-card"], "writing": ["typed", "handwritten"], "century": [20], "language": ["de"], "entry-type": ["company"], "task": ["information-extraction"]}, "contributors": ["lea_kasper", "sorin_marti"], "hidden": false, "config": {"provider": "huggingface", "model": "meta-models/Muse-Glimmer-30B:together", "dataclass": "Card", "temperature": 0.5, "role_description": "You are a Historian", "prompt_file": "prompt.txt", "legacy_test": false}, "prompt": "You are a meticulous archivist extracting data from an index card image. Analyze the provided image and extract the following information. Return the data ONLY as a valid JSON object.\n\n- \"company\": The primary company name, usually in the top-left. Exclude the location.\n- \"location\": The city or town, often following the company name.\n- \"b_id\": The identifier code, usually in the top-right, starting with \"B.\".\n- \"date\": Any stamped dates on the card in YYYY-MM-DD format. If no date is present, use an empty string.\n- \"information\": A list of text blocks from the main body of the card. Each block should be a separate string in the list. Maintain line breaks with \\\\n.\n\nHere is the required JSON format:\n{\n  \"company\": {\"transcription\": \"\"},\n  \"location\": {\"transcription\": \"\"},\n  \"b_id\": {\"transcription\": \"\"},\n  \"date\": \"\",\n  \"information\": [\n    {\"transcription\": \"\"}\n  ]\n}\n\nIf you cannot find a value for a field, leave its transcription value as an empty string. Do not add any explanatory text outside of the JSON object.", "results": {"text": "{\"company\":{\"transcription\":\"Schneider Max, Dr.,\"},\"location\":{\"transcription\":\"Z\u00fcrich.\"},\"b_id\":{\"transcription\":\"B.51.322.GB.1305.\"},\"date\":\"\",\"information\":[{\"transcription\":\"Britische schwarze Liste,\\nAmendment 6, Mai 1944.\"},{\"transcription\":\"gestrichen:\\nVon der britischen schwarzen Liste.\\nAmendment No. 15, vom 10.April 1946.\"}]}", "model": "meta-models/Muse-Glimmer-30B", "provider": "huggingface", "finish_reason": "stop", "usage": {"input_tokens": 3311, "output_tokens": 1144, "total_tokens": 4455, "cached_tokens": 256, "input_cost_usd": 0.00115885, "output_cost_usd": 0.001716, "estimated_cost_usd": 0.00287485}, "duration": 12.743393421173096, "timestamp": "2026-08-16T02:53:25.340613", "parsed": {"company": {"transcription": "Schneider Max, Dr.,"}, "location": {"transcription": "Z\u00fcrich."}, "b_id": {"transcription": "B.51.322.GB.1305."}, "date": "", "information": [{"transcription": "Britische schwarze Liste,\nAmendment 6, Mai 1944."}, {"transcription": "gestrichen:\nVon der britischen schwarzen Liste.\nAmendment No. 15, vom 10.April 1946."}]}, "conversation_id": "45ddbe6b-e583-4688-add9-2bc516275628", "score": {"fuzzy": 0.9826749826749827}, "raw_response": "{\"id\":\"owN5E3h-7ArivV-a2bc75f57f25b87e\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"company\\\": {\\\"transcription\\\": \\\"Schneider Max, Dr.,\\\"},\\n  \\\"location\\\": {\\\"transcription\\\": \\\"Z\u00fcrich.\\\"},\\n  \\\"b_id\\\": {\\\"transcription\\\": \\\"B.51.322.GB.1305.\\\"},\\n  \\\"date\\\": \\\"\\\",\\n  \\\"information\\\": [\\n    {\\\"transcription\\\": \\\"Britische schwarze Liste,\\\\nAmendment 6, Mai 1944.\\\"},\\n    {\\\"transcription\\\": \\\"gestrichen:\\\\nVon der britischen schwarzen Liste.\\\\nAmendment No. 15, vom 10.April 1946.\\\"}\\n  ]\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"parsed\":{\"company\":{\"transcription\":\"Schneider Max, Dr.,\"},\"location\":{\"transcription\":\"Z\u00fcrich.\"},\"b_id\":{\"transcription\":\"B.51.322.GB.1305.\"},\"date\":\"\",\"information\":[{\"transcription\":\"Britische schwarze Liste,\\nAmendment 6, Mai 1944.\"},{\"transcription\":\"gestrichen:\\nVon der britischen schwarzen Liste.\\nAmendment No. 15, vom 10.April 1946.\"}]},\"reasoning\":\"You are a meticulous archivist extracting data from an index card image. Analyze the provided image and extract the following information. Return the data ONLY as a valid JSON object.\\n\\n- \\\"company\\\": The primary company name, usually in the top-left. Exclude the location.\\n- \\\"location\\\": The city or town, often following the company name.\\n- \\\"b_id\\\": The identifier code, usually in the top-right, starting with \\\"B.\\\".\\n- \\\"date\\\": Any stamped dates on the card in YYYY-MM-DD format. If no date is present, use an empty string.\\n- \\\"information\\\": A list of text blocks from the main body of the card. Each block should be a separate string in the list. Maintain line breaks with \\\\\\\\n.\\n\\nHere is the required JSON format:\\n{\\n  \\\"company\\\": {\\\"transcription\\\": \\\"\\\"},\\n  \\\"location\\\": {\\\"transcription\\\": \\\"\\\"},\\n  \\\"b_id\\\": {\\\"transcription\\\": \\\"\\\"},\\n  \\\"date\\\": \\\"\\\",\\n  \\\"information\\\": [\\n    {\\\"transcription\\\": \\\"\\\"}\\n  ]\\n}\\n\\nIf you cannot find a value for a field, leave its transcription value as an empty string. Do not add any explanatory text outside of the JSON object.\\n\\nImage shows:\\n\\nTop left: Schneider Max, Dr.,   Z\u00fcrich.   B.51.322.GB.1305.\\n\\nCompany? Primary company name, usually top-left. Exclude location. Might be \\\"Schneider Max, Dr.\\\" That is a person. Maybe company is that. So company transcription: \\\"Schneider Max, Dr.,\\\"\\n\\nLocation: \\\"Z\u00fcrich.\\\" maybe \\\"Z\u00fcrich\\\". Underlined. So transcription \\\"Z\u00fcrich.\\\"? Probably \\\"Z\u00fcrich\\\". Keep as appears. Might include period. The text is \\\"Z \u00fc r i c h .\\\" with spaces. Probably transcription \\\"Z\u00fcrich.\\\" Let's keep \\\"Z\u00fcrich.\\\" Might be \\\"Z\u00fcrich\\\". I'll transcribe as \\\"Z\u00fcrich.\\\"? The instruction says location city or town often following company name. So \\\"Z\u00fcrich\\\". I'll output \\\"Z\u00fcrich.\\\" maybe without period? Might be safe to keep as \\\"Z\u00fcrich\\\". Hmm.\\n\\nb_id: \\\"B.51.322.GB.1305.\\\" Transcription.\\n\\ndate: Any stamped dates on the card in YYYY-MM-DD format. If no date is present, use empty string. There are dates in text: Mai 1944, 10.April 1946. Are those stamped? Might be typed. The instruction says any stamped dates. Might be no stamped dates. Might be empty string. Could be no stamp. So date = \\\"\\\".\\n\\ninformation: A list of text blocks from main body. Each block should be separate string. Maintain line breaks with \\\\n.\\n\\nMain body lines:\\n\\n\\\"Brittische schwarze Liste,\\nAmendment 6, Mai 1944.\\\"\\n\\nThen \\\"gestrichen:\\\" underlined.\\n\\n\\\"Von der britischen schwarzen Liste.\\nAmendment No. 15, vom 10.April 1946.\\\"\\n\\nMaybe each block is separate? Could be three blocks: first paragraph, then \\\"gestrichen:\\\", then next paragraph. Or maybe \\\"gestrichen:\\\" is part of block. Might be better to have two blocks: first block is \\\"Britische schwarze Liste,\\\\nAmendment 6, Mai 1944.\\\" Second block is \\\"gestrichen:\\\\nVon der britischen schwarzen Liste.\\\\nAmendment No. 15, vom 10.April 1946.\\\"\\n\\nOr three blocks: one for first, one for \\\"gestrichen:\\\", one for the rest. The instruction says list of text blocks from main body. Each block should be separate string. Might treat each paragraph as block.\\n\\nLet's assume blocks:\\n\\n1. \\\"Britische schwarze Liste,\\\\nAmendment 6, Mai 1944.\\\"\\n2. \\\"gestrichen:\\\"\\n3. \\\"Von der britischen schwarzen Liste.\\\\nAmendment No. 15, vom 10.April 1946.\\\"\\n\\nOr combine gestrichen with following. Hmm.\\n\\nMaybe better to have two blocks: first is the British list entry, second is the struck entry including the heading gestrichen.\\n\\nI'll produce information as array with two items: first block transcription as above, second block as \\\"gestrichen:\\\\nVon der britischen schwarzen Liste.\\\\nAmendment No. 15, vom 10.April 1946.\\\"\\n\\nLet's do that.\\n\\nCompany transcription: \\\"Schneider Max, Dr.,\\\"\\n\\nLocation transcription: \\\"Z\u00fcrich.\\\" maybe without period? I'll keep \\\"Z\u00fcrich\\\". The image shows \\\"Z \u00fc r i c h .\\\" with underline. Might transcribe as \\\"Z\u00fcrich.\\\" I'll output \\\"Z\u00fcrich\\\". Might be okay.\\n\\nb_id transcription: \\\"B.51.322.GB.1305.\\\"\\n\\ndate: \\\"\\\"\\n\\nReturn JSON.\\n\\nMake sure JSON valid. Use double quotes.\\n\\nFinal output.\"},\"token_ids\":null,\"routed_experts\":null,\"seed\":null}],\"created\":1786841593,\"model\":\"meta-models/Muse-Glimmer-30B\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"vllm-0.1.dev19059+g66826a22d-tp2-46311a71\",\"usage\":{\"completion_tokens\":1144,\"prompt_tokens\":3311,\"total_tokens\":4455,\"completion_tokens_details\":null,\"prompt_tokens_details\":{\"audio_tokens\":null,\"cached_tokens\":256,\"created_cache_tokens\":3040,\"multimodal_tokens\":{\"image\":3038}}},\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null,\"prompt\":[]}"}, "scoring": {"fuzzy": 0.9408690199005607, "cost_summary": {"total_input_tokens": 108615, "total_output_tokens": 30606, "total_tokens": 139221, "input_cost_usd": 0.038015249999999993, "output_cost_usd": 0.045909000000000005, "total_cost_usd": 0.08392424999999998}}, "normalized_score": 94.08690199005608}