Add research workflow guide

README.md

@@ -1,3 +1,75 @@
-# reference-deduplication-workflow
+# Deduplicating References Before Screening: A Practical Workflow
 
-A practical, auditable workflow for deduplicating references before screening in a literature review
+Every multi-database literature search produces duplicates. Search PubMed, Scopus, Web of Science and a discipline-specific index for the same question and it is normal for 30 to 50 percent of the combined records to be copies of each other. If those duplicates reach the screening stage, reviewers waste hours making the same decision twice, and worse, inconsistent decisions on two copies of one paper can let an excluded study slip back in. This guide describes a deduplication workflow that is fast, auditable and conservative, and explains where an AI research agent such as [Yila](https://yila.ai) can help without hiding what happened to your records.
+
+## Why automatic deduplication is not enough
+
+Reference managers such as Zotero, EndNote and Mendeley all include a "find duplicates" function, and screening tools such as Rayyan and Covidence run their own matching on import. These are useful first passes, but each has blind spots:
+
+- **Title variations.** One database stores "COVID-19" and another "Covid 19"; one keeps a subtitle after a colon and another drops it; one uses British spelling and another American.
+- **Author formatting.** "Smith J.", "Smith, John" and "J. Smith" are the same person to a human but different strings to a matcher.
+- **Missing or inconsistent identifiers.** Not every record has a DOI, some DOIs are stored with a `https://doi.org/` prefix and some without, and conference papers frequently have none at all.
+- **Different versions of the same work.** A preprint, a conference paper and the final journal article may be genuinely different documents that report the same study.
+
+Relying on one tool's default settings typically removes most exact duplicates and leaves a long tail of near-duplicates. Being too aggressive is also dangerous: two different papers by the same group with similar titles can be merged by mistake, silently losing a study.
+
+## Step 1: Export everything in one consistent format
+
+Export the results from every database in RIS or BibTeX, with abstracts and all available identifiers included. Keep the raw exports untouched in a folder named by date and database. These files are the record of what each search returned, and you will need them if a reviewer asks how you got from 2,300 records to 1,450.
+
+Record the count from each database in your search log before you do anything else. PRISMA 2020 asks for the number of records identified from each source and the number removed as duplicates, so the counts must be captured at this point.
+
+## Step 2: Normalise before matching
+
+Most missed duplicates are caused by superficial formatting differences. Before matching, create normalised copies of key fields:
+
+1. **DOI:** lower-case, strip any URL prefix, trim whitespace.
+2. **Title:** lower-case, remove punctuation and HTML entities, collapse multiple spaces, and optionally remove a trailing subtitle for a secondary comparison.
+3. **First author surname:** lower-case, strip accents.
+4. **Year:** four digits only.
+5. **Journal or venue:** lower-case, expand common abbreviations if possible.
+
+You do not overwrite the original fields; you add normalised versions used only for matching. This keeps the audit trail intact.
+
+## Step 3: Match in tiers, from certain to uncertain
+
+A tiered approach keeps the process both fast and safe.
+
+**Tier 1 – exact identifier match.** Records with the same normalised DOI or the same PubMed ID are duplicates. Merge them automatically, keeping the most complete record.
+
+**Tier 2 – strong bibliographic match.** Records with the same normalised title, first author and year are almost always duplicates. Merge automatically, but keep a list.
+
+**Tier 3 – fuzzy match.** Records whose titles are very similar (for example, a string similarity above 0.9) and that share an author or a year are probable duplicates. Do not merge these automatically. Put them in a review queue.
+
+**Tier 4 – related reports.** Records that look like different versions of the same study, such as a conference abstract and a later journal article with a different title, are not duplicates in the strict sense. Keep both and link them as reports of one study at the data-extraction stage.
+
+The distinction between tiers 3 and 4 is important. Deduplication removes copies of the same record; linking reports of the same study is a separate step that happens later, with full text in hand.
+
+## Step 4: Review the uncertain pairs by hand
+
+The tier 3 queue is usually small, often a few dozen pairs in a search of a few thousand records. Show the pairs side by side with title, authors, year, venue, volume, pages and abstract. Decide for each pair: duplicate, not duplicate, or related report. A second reviewer should check a sample, particularly for any pair where the two records have different page numbers or sample sizes, which often signals two distinct papers.
+
+This is where an AI research agent earns its keep. Yila can run the normalisation, apply the tiered matching rules and present the uncertain pairs with the differing fields highlighted, so a human reviewer can work through the queue quickly. Because every merge is logged with the rule that triggered it, the result can be checked rather than taken on trust.
+
+## Step 5: Keep a deduplication log
+
+For every merge, store the IDs of the records involved, the tier or rule that matched them, the record that was kept and, for manual decisions, who made the decision. A simple table with those columns is enough. The log lets you:
+
+- report the number of duplicates removed in the PRISMA flow diagram,
+- reverse a mistaken merge without re-running the whole search,
+- show reviewers and editors exactly how the screening set was produced.
+
+## Step 6: Deduplicate again when you update
+
+Living reviews and review updates add new search results months or years after the original search. New records must be checked against the entire existing library, including records that were already screened and excluded, otherwise previously excluded papers come back as "new". Keep the normalised fields from the first round so the comparison is quick, and repeat the same tiered process.
+
+## Common pitfalls
+
+- **Merging errata and retractions into the original paper.** Notices of correction or retraction are separate records and can be important; do not delete them as duplicates.
+- **Losing abstracts.** When merging, keep the record with the fullest abstract and the most identifiers, not simply the first one imported.
+- **Deduplicating after screening has started.** Always finish deduplication before the first screening decision, or decisions may be made twice and disagree.
+- **Not recording counts.** If the numbers are not written down at each step, the PRISMA diagram becomes guesswork.
+
+## Summary
+
+Good deduplication is boring, careful work: consistent exports, normalised fields, tiered matching, a human check of uncertain pairs and a complete log. Done well, it shrinks the screening workload considerably and protects the review from inconsistent decisions. An AI research agent such as Yila can automate the normalisation, matching and logging so that the human effort goes into the small number of genuinely ambiguous cases, and the full history of every record remains visible for anyone who needs to check it.