Skip to content

Quickstart

About 30 minutes of your time; the first benchmark then runs for one to two hours on a laptop.

Requirements

  • Linux or macOS, or Windows with WSL2 (Ubuntu 24.04 recommended)
  • Docker (or Apptainer, from M2), with at least 12 GB of memory and 60 GB of free disk
  • Java 17 or newer (Nextflow needs it)
  • Python 3.11 or newer

1. Install

pipx install aver-cli            # or: uv tool install aver-cli
aver version

2. Initialise

In an empty directory (or your pipeline repository):

aver init --preset nf-core/sarek --name germline-chr20 --clinical

This writes aver.yaml. The defaults benchmark nf-core/sarek 3.10.0 (HaplotypeCaller) on HG002, GRCh38, chromosome 20, against the published GIAB v4.2.1 truth set. Set pipeline.intended_use to one or two sentences describing what the pipeline is for; it appears in every report.

3. Fingerprint the pipeline

aver resolve -o snapshot.json

Aver asks Nextflow for the fully layered configuration (nextflow config) and every process's container (nextflow inspect), resolves every container tag to an immutable digest, and prints the snapshot hash. Same pipeline, same hash, on any machine.

Anything that is not pinned (an image tagged latest, a tag that cannot be resolved) is listed as UNPINNED. In a pipeline marked clinical, an unpinned dependency is a critical finding.

4. Benchmark

aver benchmark

The first run downloads and verifies the reference data (the GRCh38 analysis set, the HG002 truth set and about 1.5 GB of HG002 reads for chromosome 20), runs the pipeline, then compares its VCF with the truth set:

germline-chr20  ·  HG002  ·  giab-hg002-v4.2.1-grch38  ·  region chr20
          Recall  Precision        F1        TP      FN      FP
SNV     99.4294%   98.9941%  99.2113%     70926     407     721
INDEL   97.8323%   98.7786%  98.3031%     11012     244     141

These are the numbers from the worked example (sarek 3.10.0 defaults). Yours will differ with your pipeline and configuration.

Re-running with an identical snapshot, truth set, reference and comparator reuses the cached result. Use --force to run again.

5. Report and verify

aver report --format pdf        # or html; writes ./aver-report/
aver verify aver-report

aver-report/ contains payload.json (the canonical, machine-readable record), report.html, report.pdf and manifest.json (the content hash and every file's SHA-256). aver verify recomputes all of them and re-renders the HTML from the payload. Anyone can run it to show the report reflects the recorded results.

6. After a change

Resolve again and compare the snapshots:

aver resolve -o snapshot-new.json
aver diff snapshot.json snapshot-new.json --clinical

Every change is listed with its old and new value and the processes it affects, followed by the severity and every rule that produced it. Severity is always rule-derived, never model-derived. See How severity is decided.