One Fact, Two Sources: Deduplicating Study Cards Across Chapters and Lecture Transcripts

A textbook says that angular momentum is conserved when external torque vanishes. A later lecture says the same thing in different words. If each upload generates a fresh flashcard, the learner now has two nearly identical prompts, two review histories, and no clear way to see that one claim came from two sources.

The tempting fix is to compare the new cards with the cards from the current upload. That works only until the next chapter or transcript arrives. The durable fix is to stop treating the rendered card as the canonical object.

Keep one accepted fact. Attach every supporting source span to it. Render or update the study card from that fact without changing its identity.

A card, a fact, and a source are different things

Anki already separates notes from the cards generated by their templates. Its ordinary duplicate warning, however, mainly compares the first field within the same note type. That is useful at edit time, but it is not a collection-wide model of semantic identity. The Anki manual’s key concepts and duplicate check are worth reading before designing an importer.

For cross-source generation, use three layers:

  1. Fact: the accepted claim or question-answer unit.
  2. Evidence span: an exact location in a source, such as textbook page 74 or transcript timestamp 00:31:20.
  3. Rendered note: the learner-facing fields exported to Anki or another review system.

One fact may have several evidence spans. One fact may also be rendered with more than one template. Neither case requires creating a second canonical fact.

textbook p.74 ─┐
               ├─> accepted fact ─> stable Anki note ─> review history
lecture 31:20 ─┘

This separation is also what makes deletion safe. Removing one uploaded transcript should detach its evidence relation; it should not silently delete the fact, the textbook citation, or the learner’s scheduling history.

Preserve the original; normalize only for matching

Never rewrite the displayed front and back merely to make deduplication easier. Store the original text, then build a versioned matching projection beside it.

A conservative first projection can:

  • apply Unicode NFC normalization;
  • case-fold text where the language permits it;
  • collapse repeated whitespace;
  • keep negation, numbers, units, formulas, and mathematical symbols.

Do not blindly remove punctuation, stop words, or accents. “Increases” and “does not increase” are close in vocabulary and opposite in meaning. Likewise, 5 mg and 50 mg must never become one key. Unicode’s Normalization Forms specification explains why compatibility normalization such as NFKC can erase distinctions that some applications need to preserve.

Give the projection a name such as match_text_v1. If its rules change later, old decisions remain reproducible because the matcher version travels with them.

For an exact signature, hash the scope and both normalized fields:

SHA-256(owner_id || language || note_type || normalized_front || normalized_back)

Scope matters. Do not compare private notes across accounts. In many systems, language, note type, course, and user-selected collection boundaries also belong in the key.

Use fuzzy search to find candidates, not to authorize a merge

Exact signatures catch retries and lightly reformatted duplicates. They do not catch paraphrases such as:

  • “When is angular momentum conserved?”
  • “State the condition for conservation of angular momentum.”

Use a second stage only after exact matching misses:

new draft
  -> exact signature
       -> hit: attach the new evidence span
       -> miss: retrieve a small candidate set
                  -> score exact resemblance and containment
                  -> duplicate / review / distinct

For a small local collection, SQLite FTS5’s documented trigram tokenizer can shortlist candidates. PostgreSQL users can make the same kind of shortlist with `pg_trgm`. At a larger scale, fixed-seed MinHash or locality-sensitive hashing can reduce the search space; Broder’s work on document resemblance and containment provides the underlying set-based model.

The shortlist is not the decision. Recompute a deterministic score for each candidate, keep the component scores, and route ambiguous cases to review. Until a labeled test set shows otherwise, a fuzzy match should never merge two facts automatically. A missed duplicate is annoying; a false merge can corrupt the meaning of both cards.

Make exact collisions transactional

A minimal SQLite schema can keep the boundaries visible:

CREATE TABLE facts (
  fact_id     TEXT PRIMARY KEY,
  owner_id    TEXT NOT NULL,
  language    TEXT NOT NULL,
  note_type   TEXT NOT NULL,
  exact_key   TEXT NOT NULL,
  front       TEXT NOT NULL,
  back        TEXT NOT NULL,
  status      TEXT NOT NULL DEFAULT 'accepted',
  UNIQUE (owner_id, language, note_type, exact_key)
);

CREATE TABLE evidence_spans (
  span_id       TEXT PRIMARY KEY,
  document_id   TEXT NOT NULL,
  locator       TEXT NOT NULL,
  source_hash   TEXT NOT NULL,
  excerpt       TEXT NOT NULL,
  UNIQUE (document_id, locator, source_hash)
);

CREATE TABLE fact_evidence (
  fact_id  TEXT NOT NULL REFERENCES facts(fact_id),
  span_id  TEXT NOT NULL REFERENCES evidence_spans(span_id),
  PRIMARY KEY (fact_id, span_id)
);

The unique constraint prevents two concurrent retries from creating two canonical rows. Within one transaction:

  1. insert the proposed fact;
  2. if the exact key already exists, select the established fact_id;
  3. insert the new evidence span;
  4. attach it with INSERT ... ON CONFLICT DO NOTHING;
  5. commit both decisions together.

SQLite documents unique constraints, transactions, and UPSERT directly. Use those guarantees rather than a fragile “search, then insert” sequence in application code.

Keep the learner’s review history attached

Deduplication is incomplete if every export still creates a new Anki note. Keep a stable external identity for each accepted fact and update the existing note’s fields or evidence list.

Anki’s text import documentation describes duplicate handling through the first field or GUID and explains how updating existing notes can preserve scheduling. Test the exact import path you use; do not assume a regenerated package will retain identity merely because the visible question is similar.

If a fact gains a second source, the learner-facing result might change from:

Source: Chapter 4, p.74

to:

Sources: Chapter 4, p.74 · Lecture 6, 00:31:20

The note identity and review history stay the same.

Test the cases that similarity scores hide

Before tuning thresholds, build fixtures for the failures you cannot accept:

  1. The same upload retried concurrently produces one fact and one evidence relation.
  2. Identical chapter and transcript wording produces one fact with two locators.
  3. Case, whitespace, and canonically equivalent Unicode variants match exactly.
  4. A paraphrase becomes a review candidate, not an automatic merge.
  5. Negation, different quantities, units, populations, or conditions remain distinct.
  6. Conflicting answers create a conflict for review; neither answer overwrites the other.
  7. A changed textbook edition keeps its new locator and source hash.
  8. Multilingual statements remain distinct unless a reviewed equivalence connects them.
  9. Deleting one source leaves the other evidence and review history intact.
  10. Re-running the same fixtures yields the same keys, candidate order, scores, and decisions.

Measure candidate recall separately from final merge precision. A candidate generator may be deliberately generous because the final decision stage is conservative.

Patterns already useful in public projects

Video2Book retains timestamped transcript structure and source paths. The Susskind archive shows a stable course, run, and lecture hierarchy across subtitles, transcripts, and generated notes. Local Knowledge Terminal treats cards as views over accepted entities and keeps source identifiers, hashes, and locators with the evidence.

None of those repositories is a finished cross-upload flashcard deduplicator. Together, however, they demonstrate the useful boundary: sources keep their identity, accepted knowledge has its own identity, and a card is a replaceable view rather than the database itself.

If you are deciding whether a mixed book-and-transcript collection is ready for this design, the LKT sample report shows the source, privacy, citation, and go/no-go checks I would run before building.

Leave a Reply