Skip to content
Researcher

Cookbook: Verify the citations

A fabricated citation is the one mistake that turns a good paper into a retraction notice, and it is exactly the mistake a fluent language model is happy to make on your behalf. Researcher treats citation integrity as a gate, not a suggestion. This recipe runs two checks on the running scenario (self-supervised learning for ECG arrhythmia classification, evaluated on PTB-XL): an existence-plus-retraction audit over the whole bibliography, and a state-of-the-art table that refuses to print a number it cannot trace to a real paper.

Everything below comes from the worked examples in examples/research-verification/.

What runs where: the baseline and the kernel gate

Section titled “What runs where: the baseline and the kernel gate”

Two mechanical checks are always available, with no kernel install at all:

  • python scripts/bib-validator.py references/library.bib resolves each entry’s DOI against Crossref, compares the title (similarity threshold 0.70), first-author surname, and year against the Crossref record, flags retractions via Crossref update-to metadata, and reports duplicate keys, duplicate DOIs, and missing required fields. It exits nonzero when errors are found; network failures are warnings, never entry errors.
  • The citation commit guard blocks any Claude-run git commit whose \cite keys have no matching entry in the bibliography (scripts/citation-check-hook.py; scripts/install-git-hooks.py installs the same check as a real git pre-commit for terminal commits).

The fuller gate walked through below shipped in 0.3.0 as part of the optional evidence kernel (core/): multi-index verification with a four-state verdict, verified / mismatch / unresolvable / inconclusive, checked against OpenAlex, Crossref, and DataCite by default (Semantic Scholar and arXiv are selectable with --sources). unresolvable and mismatch are refusal-grade, and inconclusive never is (an index timeout stalls the gate, it does not fail your bibliography). The /researcher:verify-citations command runs this gate when the kernel is installed and degrades to bib-validator.py when it is not. The worked example in citation-audit.md shows the gate’s report on a real bibliography.

Audit the bibliography for existence

Section titled “Audit the bibliography for existence”

Verify every reference in references/library.bib actually exists in the literature, flag anything that cannot be resolved, and check for retractions.

The input is a 10-entry library.bib. Nine are the real papers from the shared example. One, kessler2021cmae, is a synthetic entry planted deliberately so the gate has something to reject. It is labeled (synthetic, for demonstration) in the source and is the only invented reference anywhere in the examples.

With the kernel installed, each reference is checked against multiple independent indexes, and a verdict of verified requires confirmation by at least two of them, with title similarity of 0.70 or better and a matching year and first author. The table below is the worked example’s report. Without the kernel, bib-validator.py runs the Crossref column of this table: DOI resolution plus title, first-author, and year comparison against a single index.

KeyVerdictConfirmed by
mehari2022sslverifiedOpenAlex, Crossref, S2
diamant2022pclrverifiedOpenAlex, Crossref, S2
wagner2020ptbxlverifiedOpenAlex, Crossref, S2
strodthoff2021benchmarkverifiedOpenAlex, Crossref
ribeiro2020automaticverifiedOpenAlex, Crossref, S2
hannun2019cardiologistverifiedOpenAlex, Crossref, S2
lai2023practicalverifiedOpenAlex, Crossref
liu2023reviewverifiedOpenAlex, Crossref
sarkar2022sslverifiedOpenAlex, Crossref
kessler2021cmaeunresolvablenone

The nine real entries clear the gate cleanly. The planted fake does not: its DOI 10.1109/TBME.2021.3098765 does not resolve in Crossref, and no title match appears in any index. Verdict, unresolvable, likely fabricated. This particular failure is caught even without the kernel: bib-validator.py reports the Crossref 404 as an error and exits nonzero.

Gate result: 9 verified, 0 mismatch, 1 unresolvable.

Sweep for retractions and corrections

Section titled “Sweep for retractions and corrections”

Retraction checking runs even without the kernel: bib-validator.py --check-retracted reads Crossref update-to metadata for every entry with a DOI. The kernel’s publication-status axis adds OpenAlex is_retracted as a second signal. Either way, the point is to tell a genuine retraction apart from the routine post-publication housekeeping that panics people who conflate the two.

KeyRetracted?On record
all 9 verified entriesNotwo routine corrections, noted below

Two of the real papers carry corrections. hannun2019cardiologist has a Publisher Correction (Nature Medicine, 2019, 10.1038/s41591-019-0359-9), and ribeiro2020automatic has an Author Correction (Nature Communications, 2020, 10.1038/s41467-020-16172-1). Neither is a retraction. Both are ordinary errata, so they are noted for your awareness and not flagged as problems.

Sweep result: 0 retractions, 2 routine corrections noted.

Audit verdict

Section titled “Audit verdict”

NOT CLEAN. 1 of 10 references is unresolvable (kessler2021cmae) and must be resolved or removed before this bibliography can be certified. No retractions found. Two corrections noted.

That single verdict line is the whole product. A green checkmark you cannot trust is worse than no checkmark, so the design would rather say “not clean” than launder a fake into your reference list. The kernel-free bib-validator.py reaches the same conclusion on this bibliography (one error, exit code 1), just from Crossref alone.

Verify the SOTA numbers before you cite them

Section titled “Verify the SOTA numbers before you cite them”

The second half of integrity is the numbers you quote, not just the papers you cite. Ask for a state-of-the-art comparison and the sota-finder skill reads each source’s own reported metric before any figure enters the table.

What is the state of the art for deep learning on the PTB-XL ECG benchmark, and how do the self-supervised results compare? Give me a comparison table for a related-work section, and be honest about what is and is not directly comparable.

Method (paper)YearBenchmark / taskReported metric (as stated by source)Verified
Supervised CNNs (Strodthoff et al.)2021PTB-XL, all statement tasksresnet/inception CNNs “show the strongest performance across all tasks”partial
Contrastive predictive coding, SSL (Mehari and Strodthoff)2022PTB-XL, linear evaluationlinear-eval “only 0.5% below supervised”; fine-tuned “roughly 1%” aboveyes
PCLR, patient-contrastive SSL (Diamant et al.)20224 clinical tasks, large private cohort”51% performance increase, on average” over from-scratchyes
12-lead supervised DNN (Ribeiro et al.)2020CODE test set, 6 abnormalitiesF1 “above 80%” and specificity “over 99%“yes
Single-lead supervised DNN (Hannun et al.)2019Ambulatory single-lead, 12 rhythm classes”cardiologist-level” detection and classificationyes
SSL at scale (Lai et al.)2023Wearable 12-lead, 60 diagnostic termsaverage AUROC 0.975, average F1 0.575 (offline test)yes
”xresnet1d101 ~0.94 macro-AUROC on PTB-XL”n/aPTB-XL, allcirculates on public leaderboards; not traced to a specific published numberunverified

Two habits make this table trustworthy. First, every number is copied from the source’s own abstract or reported results in that source’s own metric, so no row is silently harmonized into a single misleading ranking. The studies use different datasets (PTB-XL, CODE, a proprietary wearable corpus) and different metrics (macro-AUROC vs. F1 vs. relative improvement), and only the two PTB-XL rows are actually comparable to each other. Second, the last row exists to show the flag working: a number seen on a leaderboard is not admitted as fact until it is traced to a real source, so it stays marked unverified rather than quietly promoted to a citation.

What you get

Section titled “What you get”
  • Always, kernel or not: a DOI audit (scripts/bib-validator.py) that resolves every entry against Crossref, compares title, first author, and year, reports duplicates and missing fields, and exits nonzero on failures; plus a commit guard that blocks dangling \cite keys.
  • Always, kernel or not: a retraction sweep via Crossref update-to that separates genuine retractions (none here) from routine corrections and errata (present, noted calmly).
  • With the optional kernel (shipped in 0.3.0): the multi-index audit with the four-state verdict shown above, where unresolvable and mismatch refuse certification and inconclusive never does.
  • A SOTA table where each figure is attributed to the paper that reported it, cross-dataset non-comparability is stated up front, and untraceable claims are flagged rather than included.

Keep going

Section titled “Keep going”