Skip to content

How to revalidate a clinical pipeline after a tool upgrade

Short answer: identify exactly what changed, re-run the same benchmark you validated with (same reference material, truth set, comparator and region), compare the new metrics with the validated baseline in percentage points, review the result against the acceptance criteria your laboratory set, and record the decision with its evidence. A tool upgrade such as GATK 4.5.0.0 → 4.6.0.0 is one change. Measure it on its own rather than bundling it with others.

Does every change need a full revalidation?

That is your laboratory's decision, made under its own quality procedures. Aver does not make it. A practical approach is to grade changes by kind in advance:

  • A container rebuilt with identical tool versions is usually documented, not re-run.
  • A tool version or parameter change usually needs review and a re-run of the benchmark.
  • A reference genome change, or a metric change beyond your threshold, usually needs full revalidation.

These are Aver's default rules (see severity). They are versioned, and your laboratory can adjust them.

What are the steps?

  1. Snapshot before and after. Fingerprint the pipeline before the upgrade and after it, then diff the two. Confirm the upgrade is the only change. An unrelated parameter or container change makes the result impossible to attribute.
  2. Re-run the same benchmark. Use the same reference material (for example GIAB HG002), the same truth-set release, the same comparator version and the same region or target BED as the baseline. Changing any of these at the same time as the tool makes the comparison meaningless.
  3. Compare with the baseline in percentage points. A drop from 99.43% to 98.56% recall is −0.87 pp. Percentage points are unambiguous, while relative percentages hide the scale.
  4. Look at where the numbers moved. Stratified metrics show whether a change sits in difficult regions such as homopolymers or low-mappability sequence, or across the genome.
  5. Review against your criteria. Compare each delta with the thresholds your laboratory defined in advance, for example "SNV recall may not fall by more than 0.5 pp".
  6. Record the decision. Write down who reviewed it, the justification, and whether the new version becomes the baseline. Keep the evidence with the decision.

What should the evidence contain?

Enough for someone else to reproduce and check it:

  • the before and after dependency snapshots, with their hashes, and the diff between them;
  • the reference material, truth-set release and checksums, and the comparator version;
  • recall, precision and F1 per variant class for both runs, with the deltas;
  • the acceptance thresholds in force and the outcome against each;
  • the reviewer, the justification, the decision and the date.

Your quality system defines the exact form. The useful property to aim for is verifiability: the document should let a reader confirm that it has not been edited since it was signed.

How do I do this with Aver?

aver resolve -o before.json
# upgrade the tool, then:
aver resolve -o after.json
aver diff before.json after.json --clinical   # confirm the upgrade is the only change
aver benchmark                                # same material, truth set and comparator as the baseline
aver report --format pdf                      # deterministic report bundle
aver verify BUNDLE                            # recompute every hash

Acceptance thresholds live in aver.yaml under policy (for example snv_recall_delta_pp: -0.5). See the CLI reference. On the platform, the re-run happens automatically when a change calls for it. Promoting a new baseline requires a written justification, and evidence packs are signed after re-authentication and backed by a hash-chained audit history.

Aver produces validation evidence. Decisions about whether a pipeline is fit for clinical use remain with the laboratory.