Turn PDFs and Word files into Markdown, table rows and RAG chunks, priced per page

#ai #llm #rag #python

A lot of the knowledge a RAG system needs lives in PDFs: reports, manuals, filings, papers, price lists. Plain text extraction loses most of what makes them useful. Headings flatten, tables turn into a stream of numbers, and a scanned page comes back empty. And when an answer cites "the PDF", nobody can find the page it came from.

This post converts PDFs and Word files into one Markdown record per page, with tables as rows and chunks you can embed directly.

Disclosure: I'm Aurelix, and I built the PDF to Markdown Converter. It's a paid tool (prices below).

One record per page

Give it public links to PDF or DOCX files (up to 50 MB each). Each page comes back as one record:

markdown: the page's text with headings, lists and paragraphs kept.
tables: each table as a list of rows ([["ITEM", "NEEDED"], ["Books", "1"], ...]), ready for a CSV or a dataframe.
chunks: the page's Markdown already split at paragraph and sentence boundaries, at most chunkSize characters each (2,000 by default), with 200 characters of overlap.
page, pages_total, file_name, title, word_count, and ocr, which says whether the page was read with OCR.

Pages without a text layer (scans, photos) are read with Tesseract OCR automatically, in English, French, Spanish or German.

PDFs to a chunk file with page citations

pip install apify-client

import csv
import json
import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/pdf-to-markdown").call(
    run_input={
        "urls": [
            "https://arxiv.org/pdf/1706.03762",
            "https://calibre-ebook.com/downloads/demos/demo.docx",
        ],
        "maxPagesPerDocument": 100,
        "chunkSize": 1500,
    }
)
pages = list(client.dataset(run["defaultDatasetId"]).iterate_items())

chunks = tables = 0
with open("chunks.jsonl", "w", encoding="utf-8") as out:
    for p in pages:
        for i, text in enumerate(p.get("chunks") or []):
            out.write(json.dumps({
                "id": f"{p['url']}#p{p['page']}-{i}",
                "text": text,
                "source": p["url"],
                "title": p.get("title"),
                "page": p["page"],
                "ocr": p["ocr"],
            }, ensure_ascii=False) + "\n")
            chunks += 1
        for t in p.get("tables") or []:
            name = f"{p['file_name']}-p{p['page']}-t{t['index']}.csv"
            with open(name, "w", newline="", encoding="utf-8") as f:
                csv.writer(f).writerows(t["rows"])
            tables += 1

print(f"{len(pages)} pages -> {chunks} chunks in chunks.jsonl, {tables} tables as CSV")

From chunks.jsonl, embed text and store source, title and page as metadata. Every answer can then cite "page 4 of the report" instead of the whole file, and the stable id lets you re-run a document and upsert without duplicates.

Why per page, not per document:

Retrieval works at page or section level anyway, and a page number is the citation a human can check.
Tables stay separate. Numbers in a table embed badly as prose. Keep the rows as structured data and answer numeric questions from them, or store each table as one Markdown chunk.
OCR is visible per page. Filter on ocr: true to spot-check scanned pages, which are where extraction errors live.

Why this tool

Priced per page ($0.30 per 1,000 on the free plan), so short documents cost little: a 10-page PDF is $0.003.
Scanned pages are read automatically with OCR, at $8.00 per 1,000 pages.
Markdown per page, tables as JSON rows, and chunks ready for embedding.
Word (DOCX) files as well as PDFs.
What it doesn't cover: uploaded files or base64. It reads files from URLs.

For web pages rather than files, the Website to Markdown Crawler does the same job for a whole docs site; see Turn any website into clean Markdown chunks for RAG.

Limits

Multi-column layouts (papers, forms) are read line by line across the page, so columns can interleave.
Tables without ruling lines stay as text rows, not cells; merged cells come back repeated or empty.
OCR quality depends on the scan. No handwriting, and OCR takes about 3 to 10 seconds a page.
Images, charts and formulas are not described; only their horizontal text labels come through.
Skipped without charge: files over 50 MB, password-protected PDFs, legacy .doc files and other formats.
Links only. Files must be reachable at a public URL.

Cost

At the time of writing the Store page lists $0.30 per 1,000 pages on Apify's free plan and $0.25 per 1,000 on paid plans, $8.00 per 1,000 OCR pages ($5.60 on paid plans) and $0.001 per run. An OCR page is charged as an OCR page instead of a normal page, never both. Downloads that fail, files that are not a PDF or DOCX, and pages with no text even after OCR are not charged. Apify's free plan includes $5 of credit a month, about 16,000 text pages.

DEV Community