What is pipeline drift, and how do you catch it?

Short answer: pipeline drift is any change to what a validated bioinformatics pipeline actually runs after it was validated: a new tool version, a container rebuilt under the same tag, a changed parameter or a different reference file. Some drift is deliberate and some happens with nobody touching the pipeline. You catch it by fingerprinting the pipeline's full dependency set, comparing fingerprints over time, and re-running a benchmark when something material changes.

Why does a pipeline change when nobody touches it?

Because many of its dependencies are references to things other people control:

• Mutable container tags. A tag such as :latest, or even a version tag, can be pointed at a new image upstream. The pipeline code is unchanged, but the software it runs is not.
• Rebuilt images. An image can be rebuilt with the same tool version but different system libraries underneath.
• Loose conda specifications. In conda, samtools=1.21 means "any 1.21.x". A fresh environment can resolve to a different build than the one you validated.
• Configuration layering. Profiles, institutional configs and defaults combine at run time, so a change in any layer changes the effective parameters.
• Reference and annotation data. A re-downloaded reference or a new annotation database release changes inputs without changing any code.
• Vendor upgrades. For platforms such as DRAGEN, a software or configuration update changes behaviour even though the lab's own scripts are untouched.

Why does drift matter for clinical pipelines?

A validation measures one specific configuration. When that configuration changes, the validation evidence describes a pipeline that is no longer the one running. Most changes have no measurable effect, and some do. The only way to know which is to detect the change and measure it.

How do you detect pipeline drift?

1. Fingerprint the pipeline. Record the engine version, every container by its digest (not its tag), tool versions, every effective parameter, and the checksum of each reference file.
2. Hash the fingerprint. Serialise it as canonical JSON and hash it. If the hash is the same, nothing the pipeline depends on has changed.
3. Compare fingerprints over time. When the hash changes, diff the two fingerprints to see exactly what moved.
4. Grade each change with fixed rules. Decide in advance which kinds of change need review, a re-run or full revalidation. Rules a person can read are easier to defend to an inspector than a judgement made case by case.
5. Re-run the benchmark when it matters, and compare the result with the validated baseline in percentage points.

How severe is a given change?

Aver's default rules grade changes like this. The full table is in Concepts.

Change	Severity
Documentation-only change	informational
Container rebuilt, identical tool versions	documentation
Tool version change	review required, with an automatic re-run
Reference genome change	revalidation required
Metric change beyond your policy threshold	revalidation required
Unpinned dependency in a clinical pipeline	critical

Severity comes from versioned rules, never from a model, and every classification lists the rules that fired.

How do you prevent drift in the first place?

• Pin containers by digest (image@sha256:…), never by tag.
• Pin conda packages exactly (==X or =X=BUILD), or keep an explicit pin file.
• Pin the pipeline to a release or commit, not a branch.
• Verify reference files by checksum before every run.

Pinning makes drift deliberate. It does not remove the need to detect it, because a deliberate upgrade still has to be measured.

How do I do this with Aver?

aver resolve -o before.json
# … upgrade a tool, change a parameter, or simply wait …
aver resolve -o after.json
aver diff before.json after.json --clinical

aver resolve prints the snapshot hash and lists any unpinned dependencies. aver diff shows each change with its rule-derived severity. On the platform, the same checks run continuously and re-run the benchmark when a change calls for it. See Concepts and the worked example, where two resolutions a day apart produced the same hash.

---

Aver produces validation evidence. Decisions about whether a pipeline is fit for clinical use remain with the laboratory.