The LF AI & Data Foundation, operating under the Linux Foundation, has established a working group to steer development of DocLang, an AI-friendly document format. The initiative aims to create an open, standardized format that makes it easier for companies to feed documents into AI systems.
Founding members of the DocLang group include IBM, NVIDIA, Red Hat, ABBYY, HumanSignal and Forgis. The working group argues that common document formats—such as PDF, Markdown, HTML and LaTeX—were designed primarily for visual presentation and are not optimal for AI processing. When AI models tokenize these files, semantic, structural or geometric context can be lost.
What DocLang proposes
The DocLang specification is designed specifically to be optimized for large language model (LLM) tokenizers. It uses a limited XML-based vocabulary that maps DocLang elements directly to LLM tokens. The format aims to be lossless: preserving structural, layout, semantic and governance information, while supporting tables, formulas, diagrams and other multimodal elements.
DocLang builds on IBM’s Docling, an open-source toolkit introduced at the end of 2024. Docling can convert various file formats into structured, AI-compatible data; DocLang seeks to provide a standard for exchanging those structured outputs between systems.
Why a new format is needed
According to the specification authors, Markdown is too limited in functionality, HTML is too verbose, and LaTeX permits too much ambiguity—each of which complicates efficient, cost-effective use of LLMs. The proposed format aims to reduce token counts and processing costs, and to improve model accuracy by freeing tokens from the task of deciphering layout and structure.
Maxime Vermeir, ABBYY’s vice president for AI strategy, said: “DocLang’s goal is to solve one of the most fundamental enterprise AI problems: documents were designed for people, not machines.” Vermeir emphasized that a minimal, standardized, AI-native representation provides a more predictable foundation for modern AI systems.
Token costs and early results
The initiative highlights token-cost concerns: AI Cost Check data cited in the discussion indicates that OCR-processing a PDF with an AI model typically consumes roughly 1,200 input tokens and 150 output tokens. At enterprise scale this can translate into meaningful expense, especially for long or complex documents and when using higher-cost models.
Jon Knisley, ABBYY’s head of AI Value and Enablement, noted: “PDF was designed for presentation, not understanding. Whenever a PDF enters an AI pipeline, the document’s structure, meaning and layout are partially lost.” He said this frequently forces engineering teams to build bespoke, brittle tooling at each integration point, increasing cost and maintenance burden.
ABBYY’s DocLang Interactive Benchmark provides concrete comparisons: the PDF version of IBM’s 2025 annual report generated 8,421 input tokens and 512 output tokens, while the DocLang version of the same document required 5,310 input tokens and 498 output tokens. DocLang also achieved lower latency (2.7 seconds versus 4.2 seconds for the PDF) and produced higher-quality outputs: in the PDF run the AI omitted a subsection and mishandled a table merge.
Knisley added that exact savings depend on the use case and document complexity, but early measurements indicate cost reductions ranging from roughly fourfold up to thirtyfold, depending on the model evaluated.
Open standard, community-driven development
The DocLang specification will be available as an open, freely usable standard, and the working group is actively inviting additional technology providers and companies to join. Participants say initial feedback is encouraging, but they do not want to overstate current adoption while the effort is still in early stages. The stated objective is to create an interoperable, AI-native document format that lowers token usage, improves accuracy and makes it easier to maintain provenance and metadata for compliance and data governance.
The DocLang initiative reflects growing demand among enterprise users for solutions that reduce the cost of AI processing and increase reliability, particularly in large-scale document processing scenarios.
