DEV Community

Cover image for I built an API that turns messy PDFs into clean, readable text
Dani
Dani

Posted on

I built an API that turns messy PDFs into clean, readable text

PDFs are made for printing, not reading. Copy text out of one and you get page numbers in the middle of sentences, the same header on every page, words split in half by hyphens, and every line broken where the page ended.

So I built Clean PDF Reader: send it a PDF link or upload a file, and it hands back the text the way a person would want to read it. It's the second of 3 reading APIs I'm building, after the Public Domain Book Finder.

What it does

Endpoint What you get
GET/POST /pdf/read Clean text, served page by page. Headers, footers and page numbers removed, broken lines and hyphenated words joined back up.
GET/POST /pdf/chapters The chapter list, from the PDF's bookmarks or from detected headings, plus the text of any one chapter.
GET/POST /pdf/info Title, author, dates, page count, page size, and tables as rows.

Every endpoint takes either ?url= (GET) or a file upload (POST), up to 25 MB and 400 pages.

Clean text from a link

import os
import requests

HEADERS = {
    "X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"],
    "X-RapidAPI-Host": "clean-pdf-reader.p.rapidapi.com",
}
PDF = "https://arxiv.org/pdf/1706.03762"  # "Attention Is All You Need"

page = requests.get(
    "https://clean-pdf-reader.p.rapidapi.com/pdf/read",
    params={"url": PDF, "words_per_page": 900},
    headers=HEADERS,
).json()

print(page["title"], "-", page["word_count"], "words,", page["total_pages"], "pages")
print(page["cleaned"])
Enter fullscreen mode Exit fullscreen mode
Attention Is All You Need - 5889 words, 7 pages
{'headers_footers': 0, 'page_numbers': 14, 'hyphens_fixed': 14, 'lines_joined': 245}
Enter fullscreen mode Exit fullscreen mode

The cleaned counts show what was fixed. Ask for format=markdown and headings come back as ## Heading.

Uploading a file

with open("report.pdf", "rb") as f:
    page = requests.post(
        "https://clean-pdf-reader.p.rapidapi.com/pdf/read",
        files={"file": f},
        headers=HEADERS,
    ).json()
Enter fullscreen mode Exit fullscreen mode

Chapters, one at a time

chapters = requests.get(
    "https://clean-pdf-reader.p.rapidapi.com/pdf/chapters",
    params={"url": PDF},
    headers=HEADERS,
).json()

for ch in chapters["chapters"]:
    print(ch["number"], ch["title"], f'(pages {ch["start_page"]}-{ch["end_page"]})')

# Then fetch just the one you want:
one = requests.get(
    "https://clean-pdf-reader.p.rapidapi.com/pdf/chapters",
    params={"url": PDF, "chapter": 3},
    headers=HEADERS,
).json()
print(one["chapter"]["text"][:300])
Enter fullscreen mode Exit fullscreen mode

source tells you where the chapters came from: outline (the PDF's own bookmarks), headings (detected from big or "Chapter 1"-style lines), or none.

How it works

  • Headers and footers are found by repetition. The first and last lines of every page are compared with the numbers swapped out, so "Page 3 of 40" and "Page 4 of 40" count as the same line. Anything that repeats on 40% or more of the pages is removed. Big text is left alone, so a chapter title is never mistaken for a running header.
  • Paragraphs are rebuilt from the layout. Line gaps, indents, short last lines and font size changes mark where a paragraph really ends. Everything else is joined back together, including paragraphs that carry on to the next page.
  • Word spacing scales with the font. My first version glued words together in LaTeX papers ("Thedominantsequence...") because their word gaps are tighter than a fixed limit allows. Measuring gaps relative to font size fixed it: the same paper went from 2,005 words to 5,889.
  • Rotated text is skipped. Labels running sideways along figures were showing up as backwards junk, so only upright text is read.
  • Safe link fetching. Links that point at private or internal addresses are refused, downloads stop at 25 MB, and every redirect is checked too.

It's written in Python with FastAPI and pdfplumber, and runs on Render alongside the Book Finder.

Try it

Clean PDF Reader on RapidAPI (free Basic plan)

Scanned PDFs (images with no text layer) and password-protected files aren't supported yet. You'll get a clear error instead of empty text.

Next up in this series: a Doc Summarizer / Chapter Splitter. If you build something with this one, I'd love to see it!

Top comments (5)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

For a model, a running head is pure cost. The same line goes in as input on every page and tells it nothing new. Comparing first and last lines with the numbers stripped finds it without a layout model at all.

Collapse
 
dfrausto9 profile image
Dani •

Thanks Paul, that's exactly the reasoning behind it. A running head is tokens you pay for on every page with nothing new in them.

The reader works the way you describe: it takes the first and last couple of lines of each page, swaps the numbers out, and drops anything that keeps repeating. No layout model, just text comparison plus font size so real headings don't get removed by mistake.

Two cases needed extra handling:

  • Books that alternate headers (book title on one side, chapter title on the other), so odd and even pages are counted separately.
  • Short chapters, where the header only appears on a few pages and never reaches the repeat threshold. Those get caught by comparing each page with the next two.

The response also includes a count of the headers and page numbers removed, so you can see what was stripped.

Have you run into header styles this approach misses? Multi-line headers are the one I'm watching.

Collapse
 
kolmopdf profile image
KolmoPDF •

Comparing first and last lines with numbers stripped across a 40% repetition threshold is a practical heuristic for running headers. In technical reports, one edge case is alternating verso/recto headers where even pages carry the document title on the left and odd pages carry the chapter title on the right. If a chapter only spans three pages, odd headers can fall below 40%. Partitioning candidate lines by page parity (page_num % 2) handles that margin asymmetry cleanly.

Scaling word spacing by font size is spot on for LaTeX papers. Another subtle issue in Computer Modern and TeX fonts is ligature decoding (fi, fl, ffi). In pdfplumber, check whether those glyphs map into private use area Unicode or decode into standard characters, which prevents broken word stems during search.

For hyphenation, separating soft hyphens from genuine compound nouns (like state-of-the-art) avoids accidentally collapsing hyphenated terms that belong together.

Collapse
 
dfrausto9 profile image
Dani •

Thank you, this is really helpful! The even/odd page headers and compound words like state-of-the-art splitting across lines are both great catches. I’m adding both to the next update. Ligatures should already be covered by Unicode normalization, but I’ll keep an eye out for fonts that use private-use glyphs.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dfrausto9 Multi-line headers are the one I'd worry about too. The check only sees the lines it's given, so a header that wraps and changes on each page can slip through as body text.