{"_id": {"$oid": "6ac4c7b6164e0f17c0b23a2f"}, "test_id": "T1801", "benchmark": "medieval_manuscripts", "date": "2026-09-29", "tags": {"document-type": ["manuscript"], "writing": ["handwritten"], "century": [15], "language": ["de"], "layout": ["prose"], "task": ["transcription"]}, "contributors": ["ina_serif", "maximilian_hindermann"], "hidden": false, "config": {"provider": "scicore", "model": "Qwen3.8-Flash-Next-FP8", "dataclass": "Document", "temperature": 0.0, "role_description": "You are a historian with expertise in medieval manuscripts and paleography", "prompt_file": "prompt.txt", "rules": "{\"base_url\": \"https://ai-api.scicore.unibas.ch/v1\"}", "legacy_test": false}, "prompt": "IDENTITY and PURPOSE:\n\nYou are presented with an image from a 15th century medieval manuscript written in Basel in late medieval German. Your task is to extract the text from the manuscript in the specified JSON format. You must extract all text exactly as it appears in the manuscript, maintaining historical spellings, punctuation, and formatting. Do not resolve abbreviations.\n\nThe manuscript may contain:\n1. A main text body (often in one column as continuous text)\n2. A folio number (if present)\n3. Additional notes or text written in the margins (labeled as addition1, addition2, etc.)\n\nYou must:\n- Identify and transcribe the main text\n- Extract the folio number if visible (use empty string \"\" if not visible)\n- Identify and transcribe any marginal additions separately\n- Preserve line breaks with \\n\n- Maintain all historical spellings and abbreviations exactly as written. If a letter is superscribed, normalize it by writing the superscribed letter after the main letter, e.g. \"u with superscribed o\" is spelled as \"uo\". If special characters are used for abbreviations, do not resolve them, but try to transcribe the special character according to the Medieval Unicode Font Initiative. \ua75b for \"er\" and \ua770 for \"us\" or \"em\" might be the most common special characters.\n- Do not modernize or correct the text\n- Do not use OCR or attempt to interpret unclear text - transcribe what you can see\n- If a field has no content, use an empty string \"\" (not null)\n\nTake a deep breath and think step by step about the layout of the page. First identify the folio number, then the main text area, then any marginal additions. Return only a JSON file with no additional commentary.\n\nEXAMPLE OUTPUT:\n\n{\n  \"folios\": [\n    {\n      \"folio\": \"15\",\n      \"text\": \"In disem jare kam der kunig\\n mit grossem here in daz lant\\n vnd belagerte die stat Basel\\n do wertend sich die burger\\n mit grosser kraft vnd tugent\\n vnd triben den kunig dannen\\n mit schanden vnd verlust\",\n      \"addition1\": \"Anno domini 1444\",\n      \"addition2\": \"\",\n      \"addition3\": \"\"\n    }\n  ]\n}\n", "results": {"text": "{\"folios\":[{\"folio\":\"3\",\"text\":\"Vnd ein pferit die wir vnd\\nminen knechten vber hulfend\\nsen do wa\u00df minen vater weg\\ndemme den wir machtend\\nvnd richend die knecht v\u00df\\nvnd vil in vntz an den arb\\nvnd die pferit vnz an die\\npettel vnd wa\u00df ze mol ein grosser\\nnebel dz wir kaum gepaffend\\nvnd als mit grosser arbeit kame\\nwir ze mitten tag zu sant\\nein pffel vff den berg do\\ndo sach ich die burcher do gar\\nvil herren wapen zu stone\\ndie ir f\u00fcr v\u00df im gebey gant\\ndo stund mines vatters flugen\\nwapen och in dem einen\",\"addition1\":\"\",\"addition2\":\"\",\"addition3\":\"\"}]}", "model": "Qwen3.8-Flash-Next-FP8", "provider": "scicore", "finish_reason": "stop", "usage": {"input_tokens": 3600, "output_tokens": 2082, "total_tokens": 5682, "reasoning_tokens": 0, "input_cost_usd": 0.0, "output_cost_usd": 0.0, "reasoning_cost_usd": 0.0, "estimated_cost_usd": 0.0, "attempts": 2, "discarded_input_tokens": 3333, "discarded_output_tokens": 16384, "discarded_cost_usd": 0.0}, "duration": 147.42424416542053, "timestamp": "2026-09-29T21:49:56.583961", "parsed": {"folios": [{"folio": "3", "text": "Vnd ein pferit die wir vnd\nminen knechten vber hulfend\nsen do wa\u00df minen vater weg\ndemme den wir machtend\nvnd richend die knecht v\u00df\nvnd vil in vntz an den arb\nvnd die pferit vnz an die\npettel vnd wa\u00df ze mol ein grosser\nnebel dz wir kaum gepaffend\nvnd als mit grosser arbeit kame\nwir ze mitten tag zu sant\nein pffel vff den berg do\ndo sach ich die burcher do gar\nvil herren wapen zu stone\ndie ir f\u00fcr v\u00df im gebey gant\ndo stund mines vatters flugen\nwapen och in dem einen", "addition1": "", "addition2": "", "addition3": ""}]}, "conversation_id": "75d65de6-f814-4d4b-9d11-2485c6ce1a95", "score": {"fuzzy": 0.936, "cer": 0.06, "field_scores": {"[3r] folio": {"response": "3", "ground_truth": "3", "score": 1.0, "cer": 0.0, "match_found": true}, "[3r] text": {"response": "Vnd ein pferit die wir vnd\nminen knechten vber hulfend\nsen do wa\u00df minen vater weg\ndemme den wir machtend\nvnd richend die knecht v\u00df\nvnd vil in vntz an den arb\nvnd die pferit vnz an die\npettel vnd wa\u00df ze mol ein grosser\nnebel dz wir kaum gepaffend\nvnd als mit grosser arbeit kame\nwir ze mitten tag zu sant\nein pffel vff den berg do\ndo sach ich die burcher do gar\nvil herren wapen zu stone\ndie ir f\u00fcr v\u00df im gebey gant\ndo stund mines vatters flugen\nwapen och in dem einen", "ground_truth": "Vnd ein pferit die mir vnd\n minen knechten vber hulfend\n den do was nienan kein weg\n denne den wir machtend\n vnd vielend die knecht dick\n vnd vil in untz an den ars\n vnd die pferit vntz an die \n settel vnd was ze mol ein grosser\n nebel dz wir kum gesachend\n vnd also mit grosser arbeit kome\n wir ze mittem tag zuo sant\n kristoffel vff den berg Do\n Do sach ich die buecher Do gar\n vil herren wopen in stond\n die ir st\u00fcr do hin geben hand\n do stuond mines vatters seligen\n wopen och in dem einen", "score": 0.873, "cer": 0.12, "match_found": true}}}, "raw_response": "{\"id\":\"chatcmpl-bbef9064f79d498f\",\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"{\\n  \\\"folios\\\": [\\n    {\\n      \\\"folio\\\": \\\"3\\\",\\n      \\\"text\\\": \\\"Vnd ein pferit die wir vnd\\\\nminen knechten vber hulfend\\\\nsen do wa\u00df minen vater weg\\\\ndemme den wir machtend\\\\nvnd richend die knecht v\u00df\\\\nvnd vil in vntz an den arb\\\\nvnd die pferit vnz an die\\\\npettel vnd wa\u00df ze mol ein grosser\\\\nnebel dz wir kaum gepaffend\\\\nvnd als mit grosser arbeit kame\\\\nwir ze mitten tag zu sant\\\\nein pffel vff den berg do\\\\ndo sach ich die burcher do gar\\\\nvil herren wapen zu stone\\\\ndie ir f\u00fcr v\u00df im gebey gant\\\\ndo stund mines vatters flugen\\\\nwapen och in dem einen\\\",\\n      \\\"addition1\\\": \\\"\\\",\\n      \\\"addition2\\\": \\\"\\\",\\n      \\\"addition3\\\": \\\"\\\"\\n    }\\n  ]\\n}\",\"refusal\":null,\"role\":\"assistant\",\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":null,\"reasoning\":\"The user wants a transcription of a 15th-century medieval manuscript in late medieval German. I need to look carefully at the image.\\n\\nFolio number: top right corner shows \\\"3\\\".\\n\\nMain text: Let me transcribe line by line. The script is a cursive Gothic (Kurrent-like / Bastarda). Let me read carefully.\\n\\nLine 1: \\\"Vnd ein pferit die wir vnd\\\"\\nLine 2: \\\"minen knechten vber hulfend\\\"\\nLine 3: \\\"sen do wa\u00df minen vater weg\\\"\\nLine 4: \\\"demme den wir machtend\\\"\\nLine 5: \\\"vnd richend die knecht v\u00df\\\"\\nLine 6: \\\"vnd vil in vntz an den arb\\\"\\nLine 7: \\\"vnd die pferit vnz an die\\\"\\nLine 8: \\\"pettel vnd wa\u00df ze mol ein grosser\\\"\\nLine 9: \\\"nebel dz wir kaum gepaffend\\\"\\nLine 10: \\\"vnd als mit grosser arbeit kame\\\"\\nLine 11: \\\"wir ze mitten tag zu sant\\\"\\nLine 12: \\\"ein pffel vff den berg do\\\"\\nLine 13: \\\"do sach ich die b\u00fcrher do gar\\\"\\nLine 14: \\\"vil herren wapen zu stone\\\"\\nLine 15: \\\"die ir f\u00fcr v\u00df im gebey gant\\\"\\nLine 16: \\\"do stund mines vatters flugen\\\"\\nLine 17: \\\"wapen och in dem einen\\\"\\n\\nLet me look more carefully at each line.\\n\\nLine 1: \\\"Vnd ein pferit die wir vnd\\\" - yes.\\nLine 2: \\\"minen knechten vber hulfend\\\" - \\\"vber\\\" looks like \\\"vber\\\" with abbreviation? It looks like \\\"vber\\\" or \\\"vber\\\". The word after knechten looks like \\\"vber\\\" (v b e r with \ua75b?). Actually it shows \\\"vber\\\" with a \ua75b maybe. Let's look: \\\"vber\\\" - the letters v, b, e, r. The r might be \ua75b. Hard to say. I'll write \\\"vber\\\".\\nLine 3: \\\"sen do wa\u00df minen vater weg\\\" - \\\"sen\\\" maybe \\\"sen\\\" or \\\"seyn\\\"? Looks like \\\"sen\\\". Then \\\"do wa\u00df minen vater weg\\\".\\nLine 4: \\\"demme den wir machtend\\\" - \\\"demme\\\" or \\\"denne\\\"? Looks like \\\"demme\\\". Then \\\"den wir machtend\\\".\\nLine 5: \\\"vnd richend die knecht v\u00df\\\" - \\\"richend\\\" or \\\"rihend\\\"? Looks like \\\"richend\\\". \\\"die knecht v\u00df\\\".\\nLine 6: \\\"vnd vil in vntz an den arb\\\" - \\\"vntz\\\" or \\\"vntz\\\"? Looks like \\\"vntz\\\". \\\"an den arb\\\" maybe \\\"arb\\\" abbreviation for arbeit? It's cut. Looks like \\\"arb\\\".\\nLine 7: \\\"vnd die pferit vnz an die\\\" - \\\"vnz\\\" or \\\"vntz\\\"? Looks like \\\"vnz\\\".\\nLine 8: \\\"pettel vnd wa\u00df ze mol ein grosser\\\" - \\\"pettel\\\" or \\\"ettel\\\"? Looks like \\\"pettel\\\". \\\"ze mol\\\" or \\\"ze mol\\\"? Looks like \\\"ze mol\\\".\\nLine 9: \\\"nebel dz wir kaum gepaffend\\\" - \\\"nebel dz wir kaum gepaffend\\\".\\nLine 10: \\\"vnd als mit grosser arbeit kame\\\" - yes.\\nLine 11: \\\"wir ze mitten tag zu sant\\\" - \\\"zu sant\\\" or \\\"zu sant\\\"? Looks like \\\"zu sant\\\".\\nLine 12: \\\"ein pffel vff den berg do\\\" - \\\"ein pffel\\\" maybe \\\"ein pffel\\\"? Could be \\\"ein pfell\\\"? Looks like \\\"pffel\\\". \\\"vff den berg do\\\".\\nLine 13: \\\"do sach ich die b\u00fcrher do gar\\\" - \\\"b\u00fcrher\\\" or \\\"burcher\\\"? Looks like \\\"b\u00fcrher\\\" or \\\"burcher\\\". The word looks like \\\"burcher\\\" with u. Actually \\\"b\u00fcrher\\\"? Hard. I'll write \\\"burcher\\\". Hmm the letters: b, u, r, c, h, e, r. Looks like \\\"burcher\\\". But there's an umlaut? Not sure. I'll write \\\"burcher\\\".\\nLine 14: \\\"vil herren wapen zu stone\\\" - \\\"vil herren wapen zu stone\\\".\\nLine 15: \\\"die ir f\u00fcr v\u00df im gebey gant\\\" - \\\"die ir f\u00fcr v\u00df im gebey gant\\\".\\nLine 16: \\\"do stund mines vatters flugen\\\" - \\\"do stund mines vatters flugen\\\".\\nLine 17: \\\"wapen och in dem einen\\\" - \\\"wapen och in dem einen\\\".\\n\\nNo marginal additions visible. The left edge has some tiny text from previous folio maybe, but not part of this folio's additions. I'll leave additions empty.\\n\\nLet me double check line 2 \\\"vber hulfend\\\" - the word before hulfend looks like \\\"vber\\\" with \ua75b? It's \\\"vber\\\". Good.\\n\\nLine 3 \\\"sen\\\" - maybe it's the end of \\\"hulfend sen\\\"? Actually line 2 ends \\\"hulfend\\\" and line 3 starts \\\"sen\\\". Could be \\\"hulfend sen\\\" = \\\"helfen sein\\\"? Possibly \\\"sen\\\" is \\\"seyn\\\" abbreviated. I'll keep \\\"sen\\\".\\n\\nLine 4 \\\"demme\\\" - maybe \\\"denne\\\"? The first letter d, then e, then m, m, e. Looks like \\\"demme\\\". Could be \\\"denne\\\". I'll go with \\\"demme\\\".\\n\\nLine 5 \\\"richend\\\" - maybe \\\"rihend\\\"? There's a c-like loop. I'll write \\\"richend\\\".\\n\\nLine 6 \\\"vntz\\\" - the word looks like \\\"vntz\\\" with \ua75b? Actually \\\"vntz\\\" then \\\"an den arb\\\". The \\\"arb\\\" might be \\\"arb\\\" with abbreviation stroke. I'll write \\\"arb\\\".\\n\\nLine 7 \\\"vnz\\\" - looks like \\\"vnz\\\" (without t). I'll write \\\"vnz\\\".\\n\\nLine 8 \\\"pettel\\\" - first letter p? Looks like \\\"pettel\\\" or \\\"ettel\\\". There's a descender on first letter? The first letter looks like \\\"p\\\" with descender? Actually it looks like \\\"pettel\\\" with p. Hmm but context \\\"die pferit vnz an die pettel\\\" = the horses up to the saddles? \\\"pettel\\\" = Sattel? Possibly \\\"sattel\\\" written with p? Unlikely. Could be \\\"ettel\\\"? Hmm. The word after \\\"die\\\" line 7 ends \\\"die\\\", line 8 starts with a letter that has a descender loop like \\\"p\\\". I'll write \\\"pettel\\\".\\n\\nActually maybe it's \\\"settel\\\"? The first letter could be a long s? No, it has a clear descender loop like p. I'll keep \\\"pettel\\\".\\n\\nLine 12 \\\"ein pffel\\\" - maybe \\\"ein pfell\\\"? The word looks like \\\"pffel\\\" with double f? Actually \\\"pffel\\\" = \\\"Pfuhl\\\"? Hmm \\\"zu sant ein pffel vff den berg\\\" = to Saint ... a ... on the mountain. Could be \\\"ein pfell\\\" or \\\"ein pffel\\\". I'll write \\\"pffel\\\".\\n\\nHmm, maybe it's \\\"ein pfell\\\" meaning a pool? Not sure. I'll transcribe as seen: \\\"pffel\\\".\\n\\nLine 13 \\\"burcher\\\" vs \\\"b\u00fcrher\\\". The word: b, then u with maybe two dots? then r, then c/h, then e, r. Looks like \\\"burcher\\\". I'll write \\\"burcher\\\".\\n\\nLine 15 \\\"f\u00fcr v\u00df im gebey gant\\\" - \\\"f\u00fcr\\\" with umlaut? Looks like \\\"f\u00fcr\\\". \\\"v\u00df\\\" with \ua785. \\\"im\\\" or \\\"in\\\"? Looks like \\\"im\\\". \\\"gebey\\\" or \\\"gehey\\\"? Looks like \\\"gebey\\\". \\\"gant\\\".\\n\\nLine 16 \\\"flugen\\\" - \\\"flugen\\\" or \\\"fl\u00fcgen\\\"? Looks like \\\"flugen\\\".\\n\\nOK. I'll produce JSON.\\n\\nNo additions.\\n\\nFolio \\\"3\\\".\\n\\nLet me ensure line breaks with \\\\n.\\n\"},\"stop_reason\":null,\"token_ids\":null,\"routed_experts\":null}],\"created\":1790711380,\"model\":\"Qwen3.8-Flash-Next-FP8\",\"object\":\"chat.completion\",\"service_tier\":null,\"system_fingerprint\":\"vllm-0.29.0-tp4-ep-873e7112\",\"usage\":{\"completion_tokens\":2082,\"prompt_tokens\":3600,\"total_tokens\":5682,\"completion_tokens_details\":{\"accepted_prediction_tokens\":null,\"audio_tokens\":null,\"reasoning_tokens\":1860,\"rejected_prediction_tokens\":null},\"prompt_tokens_details\":null},\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null}"}, "scoring": {"fuzzy": 0.778, "cer": 0.231, "cost_summary": {"total_input_tokens": 40797, "total_output_tokens": 93005, "total_tokens": 133802, "input_cost_usd": 0.0, "output_cost_usd": 0.0, "total_cost_usd": 0.0}}, "pricing": {"bucket_date": "2026-09-27", "age_days": 2, "input_price_per_million": 0.0, "output_price_per_million": 0.0}, "timing": {"total_response_s": 1094.85, "mean_response_s": 91.24, "slowest_response_s": 231.16, "inputs_timed": 12, "inputs_stored": 12, "span_s": 3623.81}, "normalized_score": 76.9}