Table of Contents

How I’d Build a Local PubMed/PMC Search and QA Workstation

Suppose the goal is a private literature explorer that runs on one workstation: PubMed-style fielded search, PMC full-text passages when they are legally available, and short answers that point back to the exact evidence.

I would not begin by downloading everything or choosing an LLM. I would build a useful search engine on a small, representative slice, measure it, and only then scale the parts that earned their place.

PubMed and PMC are different source layers

This distinction shapes the whole system:

PubMed supplies citation records and, where available, abstracts. NLM publishes an annual XML baseline plus daily update files. Its instructions say to load the baseline first, apply updates in numerical order, and replace revised or deleted citations. See Download PubMed Data.
PubMed Central (PMC) supplies full text for a subset of the literature. Free-to-read does not automatically mean reusable. The PMC Open Access Subset says that not every PMC article is available for text mining, licenses vary by article, and automated retrieval must use the approved PMC services.

For a prototype, choose a subject and time window, then fetch a bounded set through an official API. For a durable local PubMed mirror, switch to the annual baseline and ordered daily updates. For PMC full text, use only an approved OA-subset route and retain the article-level license with every record.

Do not scrape the public article pages.

Define the first useful result before ingestion

Write 40–60 queries that the workstation must answer. Include several kinds:

an exact PMID, PMCID, DOI, author, or title phrase;
an acronym and its expanded form;
a biomedical concept expressed with different wording;
a query with year, journal, publication-type, or MeSH filters;
a question whose answer appears in one known passage;
a question with conflicting papers;
a question for which the collection contains no adequate answer.

For each query, mark the relevant article or passage before tuning retrieval. This becomes the evaluation set. Without it, a semantic-search demo can look impressive while exact identifiers, filters, or important negative results quietly break.

The first milestone should be narrow:

On one topic slice, return the expected article in the top ten and open a result at the supporting abstract or PMC section.

Question answering comes after that works.

Keep one canonical article ledger

Use a relational database for identity, versions, rights, and ingestion state. Keep the original XML and derived indexes separate. A practical record might contain:

article_key
pmid | pmcid | doi | manuscript_id
title | abstract | journal | publication_date
authors | publication_types | mesh_descriptors
source_dataset | source_version | retrieved_at
license_code | license_url | reuse_class
xml_sha256 | parser_version | indexed_at

PMID, PMCID, and DOI are aliases, not interchangeable primary keys. The official PMC ID Converter API maps the identifiers for articles present in PMC and can return version information. For bulk work, PMC also documents a downloadable ID table; do not make millions of one-record API calls.

Store three states separately:

a PubMed citation exists;
a PMC full-text record exists;
that full text has terms permitting the intended use.

This prevents a search result from implying that every abstract has local full text or that every PMC page can be redistributed.

Parse structure, not just a bag of text

For PubMed XML, preserve title, abstract sections, authors, journal, dates, publication types, chemicals, and MeSH headings. NLM publishes the current PubMed DTD and element documentation rather than treating the XML layout as an accidental format.

For eligible PMC full text, preserve article title, section hierarchy, paragraph order, figure and table captions, references, and stable article identifiers. Give each passage an address such as:

PMCID: PMC1234567
section: Results > Adverse events
paragraph: 4
source_sha256: ...

Chunk on section and paragraph boundaries first. Token-count windows are a fallback for unusually long sections, not the source of truth. A passage should remain readable when shown alone and should always open back to its article and section.

Import MeSH as a versioned vocabulary. Keep descriptor IDs, preferred labels, entry terms, and tree numbers. Use it for query suggestions and filters; do not silently expand every query into every descendant.

Build lexical retrieval first

Biomedical search contains exact strings that embeddings should not be asked to guess: gene symbols, trial identifiers, drug names, dosages, PMIDs, and quoted phrases.

I would use a compact inverted index such as Tantivy with fields along these lines:

pmid, pmcid, doi       exact + stored
title                  text + positions + stored
abstract               text + positions + stored
body                    text + positions
mesh_id                 exact, repeated
publication_type        exact, repeated
journal, year, language filters
passage_id, section     stored

Index PubMed records at article level and eligible PMC text at passage level. Search the two layers together, but label them clearly in the result list: “citation/abstract” is not “full text.”

Before adding vectors, check the evaluation set. Exact ID lookup should be deterministic. Phrase search and filters should survive punctuation and case differences. Each result should show a title, source identifier, date, matching snippet, and the fields that caused the match.

Add MedCPT as a second retrieval channel

Lexical search will miss genuine paraphrases. That is the point at which I would test MedCPT—not replace the working lexical index.

NCBI publishes a paired MedCPT Query Encoder and MedCPT Article Encoder. The model cards use the query encoder for short questions or search queries and the article encoder for title-and-abstract pairs; both produce vectors in the same space. The accompanying Bioinformatics paper describes training from PubMed search-log query/article pairs and evaluates biomedical retrieval tasks.

Follow that input contract first:

query_vector = encode_query(user_query)
article_vector = encode_article([title, abstract])

Store vectors with stable internal article IDs. Faiss is enough for an in-process first version: it adds fixed-dimension vectors to an index and returns nearest neighbours. Keep metadata and rights filters in the article ledger; never let a vector ordinal become the only identity of a paper.

MedCPT’s article model card demonstrates titles and abstracts. Treat arbitrary full-text passage embeddings as a new experiment, not a documented guarantee. If passage-level semantic search matters, build a passage-labelled evaluation subset and measure it separately.

Fuse ranks, not incomparable scores

For each query:

handle PMID, PMCID, DOI, and quoted-phrase lookups directly;
retrieve perhaps 100 lexical candidates;
retrieve perhaps 100 MedCPT candidates;
remove records that fail rights or user-selected metadata filters;
combine the two ranked lists with reciprocal-rank fusion;
optionally rerank only the top 20–30;
show the top results before generating anything.

A small rank-fusion function is enough:

def rrf(*ranked_lists, k=60):
    score = {}
    for rows in ranked_lists:
        for rank, article_id in enumerate(rows, start=1):
            score[article_id] = score.get(article_id, 0.0) + 1.0 / (k + rank)
    return sorted(score, key=score.get, reverse=True)

Do not average raw BM25 and cosine scores: their scales do not mean the same thing. Tune candidate counts and fusion only against held-out queries, not the examples used while building the system.

Make question answering a view over evidence

The QA layer should receive a small evidence packet, not the corpus:

[E1] PMID ... — title + abstract sentence(s)
[E2] PMCID ... — Methods > Eligibility, paragraph 2
[E3] PMCID ... — Results > Primary outcome, paragraph 1

Ask the model for:

a short answer limited to those passages;
a citation after each material claim;
separate treatment of disagreement;
“not enough evidence in this collection” when support is absent.

Then validate mechanically that every cited evidence ID exists. Keep the evidence panel visible beside the prose, and let the reader open the source passage. A fluent paragraph with three links at the bottom is not provenance.

For biomedical literature, I would also keep “Search only” as the default mode. The QA view is for literature exploration and synthesis, not direct diagnosis or treatment decisions. That boundary matches the limitation stated on NCBI’s own MedCPT model cards.

Measure before scaling to the full feeds

Record four groups of numbers:

Layer	Useful checks
Ingestion	parsed, rejected, updated, deleted, rights-eligible, hashes changed
Retrieval	Recall@10, MRR, filter correctness, exact-ID success
QA	supported-claim rate, citation accuracy, abstention on no-answer questions
Operations	index time, peak RAM, disk per article/passage, query latency

Run the pipeline on a sample large enough to include ugly XML, missing abstracts, duplicate identifiers, corrections, and long PMC sections. Measure bytes per record and seconds per record, then extrapolate. Do not buy hardware or promise a full-corpus build from a toy demo.

A 768-dimensional float32 vector alone uses 768 × 4 = 3,072 bytes before index and metadata overhead. Record the actual on-disk cost of your chosen index. Try float16 or quantization only after the full-precision run establishes a retrieval baseline.

Treat updates as part of the product

An initial index is easy; a trustworthy changing index is the real system.

Save the source file name, checksum, and processing result for every ingestion batch.
Apply PubMed updates in published order and process revised and deleted citations explicitly.
Rebuild from the new annual baseline instead of accumulating years of unverified patch state.
Re-embed only records whose canonical title or abstract changed.
Keep parser and model versions in the manifest.
Build a new index generation beside the active one, test it, then switch an atomic pointer.
Preserve a rollback generation until the next one is verified.

The user interface should show the local dataset date. “Local” should never quietly mean “stale and impossible to audit.”

The smallest architecture I would ship

approved NCBI downloads
        ↓
immutable XML + batch manifest
        ↓
canonical article / license / ID ledger
        ├── Tantivy article + passage index
        └── MedCPT article vectors → Faiss
                    ↓
          filtered rank fusion
                    ↓
        search results + evidence cards
                    ↓
          optional cited QA view

This fits one workstation because every layer can be built and tested independently. Start with a subject slice, keep the source ledger stable, and scale ingestion only after search quality and storage measurements are real.

I maintain LazyingArt’s local, provenance-heavy document tooling and have worked with multilingual book and research collections. I do not have a PMC-specific customer deployment or customer outcome to present. The LKT sample report shows the kind of data, rights, citation, and go/no-go map I use on a bounded project-owned collection; it is not a biomedical-system result.

Any collection-fit work would begin with metadata and a representative rights-cleared sample on one existing machine. Hardware, full PubMed/PMC ingestion, production deployment, medical validation, and direct clinical use are outside that bounded check.