Open methodology · Version 1.0
Document to Markdown Quality Benchmark
A conversion should be judged on semantic fidelity, reading order, table and link accuracy, and explicit loss reporting—not on whether the output merely contains text. This benchmark publishes three inspectable fixtures and one repeatable review protocol.
Last updated:
What this benchmark does—and does not—measure
Version 1.0 is a public review protocol, not a marketing accuracy score. It makes the source specification, acceptance checks, and known loss boundaries visible so another reviewer can repeat the same assessment. It does not claim an aggregate accuracy rate, a competitive ranking, or validation of every document layout.
Repeatable review method
- 1.Freeze the source. Recreate the selected fixture exactly. Keep the source unchanged for every rerun and record a file checksum when publishing results.
- 2.Preserve raw output. Convert once and save the unedited Markdown before fixing spacing, tables, or OCR errors.
- 3.Check each dimension. Compare structure, order, tables, links, and every critical token against the source.
- 4.Label the result. Use Pass, Review, Not represented, or Critical failure; attach the exact diff or review note.
- 5.Rerun after changes. Keep the same fixture ID and record the converter release, date, and environment so results remain comparable.
Result labels
| Label | Acceptance rule |
|---|---|
| Pass | Meaning and order are retained, Markdown is valid, and critical tokens match. |
| Review | The result is usable after a documented, recoverable cleanup. |
| Not represented | The source feature has no Markdown equivalent or is outside current support. |
| Critical failure | A name, number, date, URL, or code token changes without being reported. |
Do not average these labels into a single percentage unless the fixture set, weighting, and error policy are published with the result.
Quality dimensions
Semantic structure
Headings, paragraphs, and lists keep their meaning and hierarchy; valid Markdown syntax is used.
Reading order
Sections and items follow a usable source order; layout flattening is called out when the source has no single linear order.
Tables and links
Simple tables remain readable and link labels and destinations remain paired. Merged or irregular cells are marked for review.
Critical tokens
Names, dates, numbers, URLs, and code tokens match the source exactly or are reported as a failure.
Unsupported content
Visual-only or unsupported content is not presented as successfully preserved; the review notes identify what needs a source check.
Published fixtures
Each case has a stable ID. Follow its source link for the visible example and format-specific limitations.
Observed example
Word (DOCX)
docx-structure-v1
- Source specification
- One Heading 1, one Heading 2, two bullets, one hyperlink, and a simple two-column Item / Owner table.
- Checks
-
- • Heading levels and bullet order
- • Hyperlink label and destination
- • Table cell values and row order
- • Any extra blank table header row
- Known boundary
- Embedded images, comments, headers, footers, and tracked-change metadata are not retained in the current Markdown output.
Reproducible fixture
PowerPoint (PPTX)
pptx-linearization-v1
- Source specification
- Two slides: a Q3 launch plan with two bullets, followed by an Owners slide with a two-column Work / Owner table.
- Checks
-
- • Slide order and heading levels
- • Bullet order and indentation
- • Table labels, names, and reading order
- • Speaker notes and visual-only meaning against the source
- Known boundary
- Animations, transitions, charts, and spatial layouts cannot be reproduced in plain Markdown and speaker notes may be omitted.
Reproducible fixture
Image (PNG or JPG)
image-ocr-critical-tokens-v1
- Source specification
- A clear image containing a Weekly status heading, two status lines, the date 18 August, and a two-column Owner / Task table.
- Checks
-
- • Heading and list inference
- • Exact date, owner name, and punctuation
- • Table labels and row order
- • OCR differences caused by resolution or compression
- Known boundary
- OCR is probabilistic. Handwriting, formulas, unusual fonts, low resolution, and dense layouts require source-by-source review.
Using the benchmark in a real workflow
For a GitHub, Obsidian, RAG, or documentation migration, add a small sample of your own source documents beside these fixtures. Treat names, dates, URLs, citations, and code as critical tokens. Keep the original available for human review, especially when OCR, charts, merged cells, or spatial layouts carry meaning.
How to cite a result
Include the benchmark version, fixture ID, converter release, run date, source checksum, and raw output. Link to this page and state every local change to the fixture or scoring rule. That keeps a one-document result inspectable without implying it applies to every file.
To-Markdown Quality Benchmark v1.0
Fixture: docx-structure-v1
Converter release: [version]
Run date: [YYYY-MM-DD]
Source checksum: [sha256]
Result: [Pass | Review | Not represented | Critical failure]
Notes: [exact differences]