Before You Index 17,000 Research PDFs, Test the Collection

A folder of 300 research papers can become a useful local library. A folder of 17,000 can become an impressive-looking way to retrieve the wrong paper.

The difference is rarely the vector database. It is whether you know what is actually in the collection: scans and text PDFs, preprints and publisher versions, duplicate downloads, missing pages, broken reading order, multilingual titles, and references that were extracted as if they were body text.

Before building a large local RAG system, run a collection integrity test on a representative sample. The result should tell you whether the collection is ready to index, where citations will break, and which problems deserve engineering time.

Start with the decisions the library must support

Write down real questions before choosing a stack. A literature collection may need to answer very different kinds of queries:

• Find the paper that introduced a method.
• Compare two definitions without merging them.
• Trace a claim back to the exact version and page.
• Follow a citation from one paper to another.
• Search an English collection with a Chinese or Japanese query.
• Tell a primary experiment from a review that merely cites it.

For each question, name one or more documents that a good search should find. Keep 20–50 of these queries as a small evaluation set. It does not need to be statistically grand; it needs to reflect the work you actually do.

Without this set, every architecture decision is judged by how fluent the demo looks. With it, you can ask the more useful question: did the new component recover evidence that the simpler system missed?

Make a source ledger before making chunks

Give every input file a stable local identifier and record enough information to recognize it again:

{
  "document_id": "doc_004281",
  "original_path": "papers/attention-is-all-you-need.pdf",
  "sha256": "…",
  "bytes": 2201700,
  "page_count": 15,
  "source_url": "…",
  "retrieved_at": "2026-09-04",
  "declared_language": "en",
  "rights_note": "research copy supplied by collection owner"
}

The hash catches exact duplicates even when filenames differ. The original path and retrieval record preserve where the file came from. Keep originals read-only; extraction results, OCR text, embeddings, and graph data belong in separate derived-data directories.

Scientific collections also contain version families. A preprint, accepted manuscript, and publisher PDF may describe the same work while differing in pagination, figures, corrections, or supplementary material. Do not silently collapse them into one document. Link them as related versions and let the citation point to the version the reader can actually open.

DOIs are useful identifiers, but they are not a complete provenance system. The Crossref REST API can return deposited scholarly metadata as JSON, including fields such as funding, licences, updates, ORCID and ROR identifiers when available. Store the returned metadata beside the source ledger and record when it was retrieved; do not overwrite what the PDF itself says.

Audit extraction by document class

A single “text extracted successfully” flag is too weak. A PDF can yield thousands of characters while losing columns, captions, equations, or the relationship between a table and its heading.

First divide a representative sample into classes that matter in your collection:

• born-digital text PDFs;
• scanned pages;
• two-column articles;
• equation-heavy papers;
• table-heavy reports;
• multilingual or non-Latin documents;
• theses, proceedings, supplements, and unusually long files.

For each sampled document, inspect the title, author list, section order, page boundaries, references, one figure caption, one table, and any equations that matter to retrieval. Record failure types instead of averaging them into one score.

PyMuPDF’s text recipes document several extraction forms, including plain text, blocks, words, HTML and structured dictionaries. Blocks and word coordinates are useful when reading order matters. For scientific structure, GROBID is designed to turn technical and scientific PDFs into structured TEI documents. Neither removes the need to inspect the classes your collection actually contains.

OCR should remain a measured exception. If only one slice of the collection is scanned, routing that slice through OCR is easier to test and maintain than treating every PDF as an image.

Define a citation contract

Before choosing chunk size, decide what every search result must be able to show. A useful minimum is:

document ID + exact version + page or section + quoted snippet + extraction method

If the PDF has stable printed page numbers, keep both the PDF page index and printed label. If a passage crosses a page boundary, say so. If OCR produced the text, expose that fact. If page geometry was unavailable, do not invent a page number from chunk position.

This contract should survive every later layer: lexical search, embeddings, reranking, generation, and graph traversal. A generated answer is only as inspectable as the evidence object that reached it.

Establish a lexical baseline first

Index the sample with a straightforward lexical system and run the evaluation queries. BM25, SQLite FTS5, or another ordinary full-text engine gives you an understandable baseline for titles, author names, technical terms, identifiers, and exact phrases.

The BEIR benchmark evaluated retrieval systems across 18 diverse datasets and found BM25 to be a robust baseline; reranking and late-interaction approaches improved average performance at higher computational cost. That is a good reason to measure the baseline, not a reason to assume one method will win on your collection.

For each query, record:

• whether an expected document appears near the top;
• whether the correct passage can be opened;
• whether the displayed citation identifies the right version;
• which failure class explains a miss.

Add semantic retrieval when the measured misses are genuinely semantic—for example, the query and paper use different vocabulary for the same concept. Keep exact search available for debugging. Hybrid retrieval should earn its place by improving the saved query set, not by making the architecture diagram larger.

Build the knowledge graph after identifiers are stable

A graph is valuable when it answers questions that ranked passages cannot answer cleanly: which papers use a method, which edition introduced a correction, how a term changed across languages, or which claim depends on which experiment.

But an edge such as paper A —supports→ claim B is not self-justifying. Store provenance on the edge:

source document + version + page/section + supporting span + extraction method + review state

Keep observed facts separate from inferred links. An author list parsed from a title page, a reference recovered by GROBID, and a relationship proposed by a language model have different evidence strength. If they all become identical graph edges, the graph will look complete while hiding its weakest assumptions.

For multilingual collections, preserve the original label and script. Translations, aliases, pinyin, furigana, roots, and normalized spellings can be attached as additional forms; they should not replace the source wording. This makes cross-language discovery possible without erasing provenance.

Test updates before the first full build

Large libraries are not static. Papers are added, corrected, renamed, and replaced. Before indexing all 17,000 files, prove that the sample pipeline can:

1. add a new document without rebuilding everything;
2. recognize an unchanged file;
3. preserve an older version when a new one arrives;
4. remove derived data when a source is withdrawn;
5. rebuild an index from the ledger and recorded settings.

Also test the offline boundary you intend to promise. After required tools and models are installed, disconnect the network and rebuild the sample. Watch extraction, embedding, reranking, telemetry, and model-loading paths rather than assuming that “local UI” means every stage is local.

What the integrity report should say

A useful report is short enough to make a decision. It should contain:

• the collection classes and representative sample;
• exact duplicates and suspected version families;
• extraction failures by class;
• citation coverage and known weak points;
• results from the real query set;
• privacy and derived-data locations;
• the smallest architecture that meets the acceptance criteria;
• a go/no-go boundary for OCR, semantic retrieval, generation, and graph work.

There is no universal pass percentage. A historian tracing one disputed quotation and a team surveying thousands of abstracts have different tolerances. The collection owner should set the acceptance criteria; the integrity test should make the trade-offs visible.

A bounded way to test one collection

I maintain Local Knowledge Terminal for local, multilingual, source-linked collections. Its sample report shows the kind of provenance map, browser proof, and go/no-go decision described here.

If you have a collection and an existing machine, the free fit check is the quiet first step. If the material is suitable and we agree on the written scope, the optional USD 250 sprint tests one representative sample and delivers the map, small browser proof, and recommendation. It does not include hardware, shipping, custom OCR, or production deployment.

Whatever stack you eventually choose, keep the order simple: know the files, prove the extraction, preserve the citation, measure retrieval, then add intelligence.