How to Search Confidential PDFs Locally on a 12 GB Laptop—Without Overbuilding RAG

Suppose you have a daily log in plain-text files, a folder full of PDFs, 12 GB of RAM, no discrete GPU, and one non-negotiable requirement: the collection must remain private.

It is tempting to begin with a local LLM, a vector database, an orchestration framework, and a chat interface. That is often the expensive way to discover that your real need was simpler: type an idea, find the right file, and see the passage that matched.

For that job, start with retrieval that still works when the model is switched off.

First decide what “search” means

These are different products:

1. File finding: return the most likely files and matching snippets.
2. Passage finding: return the best pages or sections with their source paths.
3. Question answering: compose an answer from retrieved passages.
4. Document analysis: compare, classify, or extract structured facts from many sources.

If you only need the first two, you may not need generation at all. A deterministic search result is faster, easier to audit, and much more comfortable on a small CPU-only machine.

Write down 20–30 questions before installing anything. For each question, record the file or page you expect to find. This small set becomes your retrieval test. Without it, every new component can make the demo look more sophisticated while making the actual search worse.

Step 1: inventory the collection

Make a table with at least these columns:

Field	Why it matters
File path	Lets every result return to the original source.
Format	TXT, Markdown, text PDF, scanned PDF, DOCX, and images need different extraction paths.
Language	Tokenization and semantic models are language-sensitive.
Text or scan	A PDF may contain real text, page images, or both.
Rights	You should have the right to process the material.
Sensitivity	Logs, contracts, medical records, and client documents need different controls.

Do not copy the only version of a collection into an experimental pipeline. Keep the originals read-only and write extracted text, OCR output, indexes, and caches into separate derived-data folders.

That separation matters because a search index is sensitive too. Even if it contains no original PDF, it may contain most of the document text.

Step 2: extract text before adding AI

For PDFs that already contain selectable text, Poppler’s pdftotext utility is a small and predictable starting point:

pdftotext -layout input.pdf output.txt

The -layout option attempts to preserve the physical layout. It is useful for visual inspection, although a simpler reading-order extraction may work better for some indexes. Test both on a representative sample instead of assuming one setting fits the entire collection. See the Poppler project and the `pdftotext` manual.

For scanned pages, OCR is a separate stage. OCRmyPDF adds a searchable text layer to scanned PDFs. Keep the original and write a new file:

ocrmypdf --skip-text input.pdf searchable.pdf
pdftotext -layout searchable.pdf searchable.txt

--skip-text is useful for mixed PDFs because pages that already contain text do not need to be OCRed again. Still inspect the output: names, dates, tables, uncommon characters, and multilingual pages are common failure points.

Custom OCR can easily become the largest part of a document project. Measure how many pages actually need it before designing the rest of the system around OCR.

Step 3: try ordinary full-text search

If you want a ready-made desktop interface, Recoll indexes document contents and returns relevant files. Its user manual documents the supported search modes and the external helpers required for formats such as PDF.

If you are building your own small tool, SQLite FTS5 is often enough. FTS5 is SQLite’s full-text-search virtual table module and supports phrases, prefixes, proximity queries, Boolean combinations, ranking, highlights, and snippets.

A minimal schema can keep the evidence attached to every searchable passage:

CREATE VIRTUAL TABLE docs USING fts5(
  path UNINDEXED,
  page UNINDEXED,
  text,
  tokenize = 'unicode61'
);

Then retrieve both the source and a compact excerpt:

SELECT
  path,
  page,
  snippet(docs, 2, '[', ']', ' … ', 24) AS excerpt,
  bm25(docs) AS rank
FROM docs
WHERE docs MATCH ?
ORDER BY rank
LIMIT 20;

This is not a complete ingestion program, but it establishes the contract your later system should preserve: every hit has a file, a page or section, a snippet, and a reproducible query.

Run the 20–30 test questions now. Record which expected files appear in the first five results. If ordinary search works, stop. A smaller system that solves the real task is not an unfinished RAG system; it is a finished search system.

Step 4: add semantic retrieval only for measured misses

Full-text search struggles when the query and the document use different words for the same idea. If your test set exposes that problem, add embeddings as a second retrieval signal—not as a replacement for exact search.

A practical hybrid sequence is:

1. retrieve exact or lexical matches;
2. retrieve semantic matches from short, overlapping passages;
3. merge the two result lists;
4. show the source path and passage for every result;
5. keep an “exact only” switch for debugging.

On a small machine, embed the collection in batches and persist the index. Do not re-embed everything on every launch. If you download a model once for offline use, verify that later searches make no external requests.

Language matters here. An English-only model may be a poor choice for Chinese, Japanese, or a mixed collection. Test retrieval using the languages people will actually type, including names, abbreviations, and domain-specific terms.

Use the context window as a budget, not as document storage

An offline model does not need the whole corpus in its prompt. The collection stays in the search index; only a small set of passages for the current question enters the context. That separation is the core idea behind retrieval-augmented generation: retrieve from an explicit external memory, then let the model work with the retrieved evidence.

More context is not automatically better. *Lost in the Middle* found that language models can use relevant information less reliably when it sits in the middle of a long input. A large advertised context window is therefore not a reason to paste in every hit.

Use this as a starting experiment, not a universal constant:

1. retrieve perhaps 20 lexical and semantic candidates;
2. remove duplicates and cap how many passages come from one document;
3. rerank, then send only the best 4–8 short passages that fit comfortably;
4. label them with stable source IDs such as P1, P2, and P3;
5. ask for an answer supported only by those passages, with an explicit “insufficient evidence” result.

Measure retrieval recall, citation correctness, and abstention on the 20–30 test questions. Increase the passage count only when the expected evidence is being omitted; decrease it when irrelevant passages distract the answer. The useful context size is the smallest one that reliably contains the required evidence, not the model’s maximum token number.

Where a small language model can help

A small language model can fill three different roles. Test them separately:

Role	Use it when	Main failure to measure
Query rewriter	Short queries, acronyms, or vocabulary mismatch cause retrieval misses.	The rewrite drifts away from the user’s actual question.
Reranker	Search finds the right passage, but it appears too low in the result list.	Relevant exact matches are demoted and latency grows.
Answer generator	Readers need a comparison or synthesis across a few passages.	Unsupported claims or citations that do not support the sentence.

Research on rewrite–retrieve–read shows that a small trainable model can be used as the query rewriter. For a private local system, keep the original query, retrieve with both the original and rewritten forms, log the rewrite locally, and compare their results. Do not let a fluent rewrite silently replace the user’s wording. If rewriting does not improve the measured retrieval set, remove it.

This architecture makes a modest context window manageable: retrieval reduces the corpus to evidence, while the small model performs one bounded job.

Step 5: add an LLM only when generation has a job

An LLM is useful when you want a short summary, a comparison, or an answer assembled from several passages. It should not become the only way to see the evidence.

Keep these invariants:

• retrieval works without the LLM;
• every generated claim can expose its source file and page;
• the user can open the original passage;
• “not found” is an acceptable result;
• generated text is never silently written back into the source collection.

On CPU-only hardware, a small quantized model may be adequate for phrasing, but model size is not the first question. Retrieval quality and source traceability usually matter more.

A privacy checklist that includes derived data

“Local” is not a complete privacy design. Check the whole path:

• Keep original documents and derived indexes under access controls appropriate to the material.
• Bind any browser interface to 127.0.0.1 unless remote access is deliberately secured.
• Confirm that extraction, embedding, reranking, and generation do not call cloud APIs.
• Treat logs, thumbnails, OCR sidecars, vector indexes, and backups as sensitive data.
• Encrypt backups where the threat model requires it.
• Record which model and extraction settings produced each index so it can be rebuilt.
• Delete a derived index when the source collection must be removed.

A simple offline test is to disconnect the network after installation and rebuild a small sample. That does not prove perfect security, but it catches many accidental dependencies.

The smallest useful architecture

For many personal collections, the first useful version is only:

read-only originals
        ↓
text extraction / selective OCR
        ↓
passages with path + page metadata
        ↓
Recoll or SQLite FTS5
        ↓
ranked snippets that open the source

Add a semantic index only when your test questions show vocabulary mismatch. Add generation only when users need synthesis. This order keeps the system understandable and gives every new layer a measurable reason to exist.

When a collection-fit check is useful

The difficult part is often not choosing a fashionable RAG stack. It is deciding whether the collection is extractable, what must remain private, which languages and readers matter, where citations should point, and what the existing machine can realistically support.

Before sharing any material, you can inspect a complete sample fit report built from LKT’s own documented reference collection. It shows the data/privacy map, representative browser proof, and go/no-go boundary. It is project-owned evidence, not a customer result or testimonial.

I maintain Local Knowledge Terminal, and I offer a free, private-by-design collection fit check for that decision. The page prepares an email draft in your browser; it does not automatically upload, store, or send your answers.

If the collection is suitable and we both accept the scope, the optional founding sprint is USD 250. It covers one customer-provided collection, one language goal, and one existing machine. The deliverables are a written data/privacy/citation map, a small browser proof from a representative sample when usable, and a go/no-go recommendation.

Hardware, shipping, custom OCR, and production deployment are not included. The fit check comes before payment, and you should have the right to use the source material.

Whether or not you use that service, the core advice is the same: make the files searchable first, measure the misses, and add intelligence only where it earns its place.