Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git...
For further actions, you may consider blocking this person and/or reporting abuse
PDFs are brutal because they look like documents but behave like layout artifacts. The token waste is only one symptom. The bigger issue is losing section hierarchy, tables, and page context before the model ever gets a chance to reason.
The PDFs are just one of many, many document format. However, when it comes to the knowledge formatting is what makes the solution fundamentally different.
Here are the best practices with respect to the PDFs
Yes, structure recovery is the real work. Raw text extraction feels simpler, but it moves the hardest part downstream into the model. I would rather make headings, tables, page boundaries, and source locations explicit before the LLM ever sees the document.
Text-only helps when the question is semantic, but PDFs often hide structure in layout. Tables, captions, sections, and reading order can all change the answer if extraction flattens them too aggressively.
Agree with you. Time to rethink on your approaches :)
There a various way of solving the problem. Please deep dive into some of the well known solutions like - pageindex.ai/
A given problem has million ways to solve. However, we just need to run the brain in deep research mode.
Agreed that there are multiple viable approaches. The main thing I would preserve is the diagnosis step: know whether the question needs semantic text, layout, tables, image context, or source provenance before choosing the PDF pipeline.
the "formatting tax" framing is exactly right. we ran into this building a RAG pipeline over a legal document corpus — 400 page PDFs, averaging 80K tokens raw. same docs converted to clean markdown averaged 22K tokens. not MarkItDown but a similar pipeline using pymupdf plus a custom table extractor.
the bigger win was not the token count though, it was retrieval quality. chunking raw PDF text means your chunks inherit layout artifacts (footnote numbers mid sentence, header repeats, two column bleed). clean markdown chunks are way more coherent for the retriever.
does MarkItDown handle multi column layouts cleanly, or does column order still get scrambled in the output?
yeah bro