All chapters
Cloud & HPC Genomics
advancedWorkflow Management Systems
Workflow Managers Compared
💧Nextflow DSL2Dataflow paradigm | HPC + cloud | nf-core community | Docker/Singularity | Most popular in genomics
🐍SnakemakePython-based rules | Familiar for Python users | Conda integration | Bioinformatics standard
🔬WDL/CromwellBroad/Google standard | GATK workflows | Terra platform | JSON inputs
🌊CWLVendor-neutral | JSON/YAML | Seven Bridges | Portable description language
Workflow Management Systems
- Nextflow (DSL2) - dataflow paradigm; runs on HPC (SLURM/LSF) and cloud (AWS/GCP/Azure); nf-core community pipelines
- Snakemake - Python-based rules; familiar for Python users; conda/singularity integration
- WDL/Cromwell - Broad/Google standard; GATK best practices workflows use WDL
- CWL (Common Workflow Language) - vendor-neutral; JSON/YAML; used by Seven Bridges, Rabix
- All support: parallelism, checkpointing, container execution, cluster/cloud dispatch
Nextflow DSL2
code
// main.nf - DSL2 pipeline
nextflow.enable.dsl = 2
process BWA_MEM {
tag "$meta.id"
container "biocontainers/bwa-mem2:2.2.1"
cpus 16
memory "32 GB"
input:
tuple val(meta), path(reads)
path reference
output:
tuple val(meta), path("*.bam")
script:
"""
bwa-mem2 mem -t ${task.cpus} ${reference} ${reads} \
| samtools sort -@ 8 -o ${meta.id}.bam
"""
}
workflow {
reads_ch = Channel.fromFilePairs(params.reads)
ref_ch = file(params.reference)
BWA_MEM(reads_ch, ref_ch)
}
// Run: nextflow run main.nf --reads "data/*_{R1,R2}.fastq.gz" --reference hg38.fa -profile slurmnf-core Pipelines
- nf-core/sarek - WGS/WES germline + somatic; GATK4 best practices; widely used in clinical research
- nf-core/rnaseq - RNA-seq; STAR + HISAT2 + Salmon; MultiQC; strand detection
- nf-core/scrnaseq - scRNA-seq; Cell Ranger + STARsolo + Salmon Alevin
- nf-core/atacseq - ATAC-seq; peak calling with MACS2/HMMRATAC
- nf-core/methylseq - Bismark or bwa-meth; bisulfite sequencing
- Install: nextflow pull nf-core/sarek; nextflow run nf-core/sarek -profile docker
Cloud Genomics Platforms
- AWS (Amazon Web Services): EC2 for compute, S3 for storage, Batch for job queuing; AWS Genomics CLI
- Google Cloud Life Sciences (Batch): native WDL/Cromwell; BigQuery for variant analytics
- Microsoft Azure Genomics: GATK-certified; batch variant calling service
- Terra (Broad/Google): analysis platform; TCGA/GTEx data access; WDL workflows
- DNAnexus: managed genomics cloud; clinical-grade; used by UK Biobank and Regeneron
- Nextflow Tower (Seqera): deploy Nextflow pipelines to any cloud/HPC with GUI monitoring