Script to Pipeline: Turning Your Scripts into a Nextflow Workflow¶
This is the companion guide to Genomics Exchange Fall 2026, Session 5 ("Script to pipeline", November 24, 2026). It walks through converting a working shell script into a small Nextflow workflow that runs many samples in parallel on Negishi and that a collaborator can run unchanged with their own samplesheet.
The complete workflow is in the rcac-bioinformatics/script-to-pipeline repository (tag v1.0.0).
-
Prerequisites
- An RCAC account with access to Negishi and a SLURM account to charge
- Familiarity with Nextflow channels, processes, and
-profile(Session 3, Nextflow fundamentals) - Experience running an nf-core pipeline on a cluster (Session 4, Running nf-core pipelines on clusters)
- A shell, R, or Python analysis you already run by hand
-
Objectives
- Break an existing script into Nextflow processes without rewriting the commands
- Connect processes with channels and run many samples in parallel with one command
- Change tool options and inputs through configuration, not code
- Package the software with containers or modules and hand the workflow to a collaborator
When to write your own workflow¶
If an nf-core pipeline already does your analysis, use it; that was Session 4. Write your own workflow when your analysis does not match an existing pipeline, when you have a sequence of scripts you run on every new batch of samples, or when someone else needs to run your analysis and get the same result.
The aim is not to rewrite your analysis. The commands you trust stay the same; Nextflow takes over the parts you currently do by hand: looping over samples, submitting jobs, waiting for them, noticing failures, and rerunning only what changed.
The example analysis¶
The session uses a small variant-calling analysis on Escherichia coli clones from the Lenski Long-Term Evolution Experiment (population Ara-3, Tenaillon et al. 2016, BioProject PRJNA294072), mapped to the REL606 reference (GenBank CP000819.1). Each sample is subsampled to 300,000 read pairs, so the whole run takes a few minutes.
| Step | Tool | Runs | Pattern |
|---|---|---|---|
| Index the reference | bwa index, samtools faidx |
once | singleton, output reused |
| Map and sort each sample | bwa mem, samtools sort |
once per sample | fan-out |
| Call variants jointly | bcftools mpileup, bcftools call |
once, needs every BAM | fan-in |
| Count variants per sample | a Python script | once | your own script |
The data is staged on Negishi under /depot/workshop/data/script-to-pipeline/.
The starting point: a linear script¶
Most analyses start as one script with a loop. This one works, and for six small samples it finishes in one job:
Its limits appear as the project grows: samples run one after another, and a failure in sample 5 means restarting from the top.
Doing it in plain SLURM¶
The usual next step is one script per step, submitted per sample or as a job array, chained with dependencies:
| submit_arrays.sh (excerpt) | |
|---|---|
This is a real improvement, and if you already work this way you are most of the way to a workflow. What it still leaves to you:
- If one array task fails, the dependent job stays
PENDINGwith reasonDependencyNeverSatisfied, and nothing tells you. - Recovering means finding the failed task, resubmitting it, cancelling the stuck jobs, and resubmitting them with new job IDs.
- Changing one option downstream means deciding by hand which steps to rerun.
- The next project means editing paths, sample lists, and file-name patterns in every script.
Partial outputs look like real outputs
If bwa mem dies partway through a sample, samtools sort still reaches the end of its input and writes a valid BAM containing only some of the reads. The job fails, but the file is there. Nextflow marks the task as failed and does not pass or publish its outputs.
From script to process¶
A process is your commands plus a declaration of what they read and what they write. The conversion has three steps:
- Copy the commands into a
script:block. - Declare the files the commands read as
input:and the files they write asoutput:. - Replace hardcoded file names with the input variables, and thread counts with
task.cpus.
Here is the align step after conversion. The commands are the same lines as in the loop above:
Three things to notice:
- The loop is gone. A process handles one sample; the channel decides how many samples there are.
- Backslashes are doubled inside the triple-quoted script (
\\for a line continuation,\\tfor a tab), because Groovy reads the string before Bash does.${...}is filled in by Nextflow; use\$for a Bash variable. - No paths. The task runs in its own work directory, and Nextflow links the declared inputs into it.
Your own R or Python scripts do not need converting at all. Put them in the workflow's bin/ directory, make them executable with a shebang line, and call them by name; Nextflow puts bin/ on the PATH of every task:
| modules/summarize.nf | |
|---|---|
Connecting processes with channels¶
The workflow block wires the processes together. Three patterns cover most analyses:
| Pattern | Where | What it replaces |
|---|---|---|
| Fan-out | reads_ch into BWA_MEM |
the for loop, or a job array |
| Singleton | BWA_INDEX called with a single file, so its output is a value channel reused by every task |
copying the index path into every script |
| Fan-in | .collect() before BCFTOOLS_CALL |
watching squeue, or --dependency=afterok |
The samplesheet is a plain CSV with one row per sample. The first two of the six rows in project_a.csv:
| project_a.csv (first two rows) | |
|---|---|
Use absolute paths in samplesheets
Relative paths are resolved from the directory you launch Nextflow in, not from the samplesheet's location. Absolute paths make a samplesheet usable from anywhere.
Configuration: what, where, and how¶
Keeping three kinds of settings out of the .nf files is what lets the same code run on a new project or a new cluster.
| Setting | Lives in | Changed with |
|---|---|---|
| What data to run on | params in nextflow.config |
--input, --fasta, --outdir, --min_qual on the command line |
| Where tasks run | the nf-core purdue_negishi profile |
-profile purdue_negishi --cluster_account <account-name> |
| How each tool behaves | ext.args in conf/modules.config |
editing that file, or -c my.config |
The workflow does not define its own cluster settings. It loads the nf-core institutional configs and uses the purdue_negishi profile, the same profile nf-core pipelines use on Negishi. That profile sets the SLURM executor, routes tasks to the right partition, charges the account you pass with --cluster_account, and enables Apptainer.
Tool options use the same ext.args convention as nf-core modules: ext.args goes to the first tool in a process, ext.args2 to the second.
| conf/modules.config | |
|---|---|
Containers or modules¶
Each process names a pinned container image, and with -profile purdue_negishi Nextflow pulls it with Apptainer and runs the task inside it. Containers are what make the workflow portable: a collaborator on another cluster gets the same tool versions.
| Tool | Version | Image |
|---|---|---|
| bwa | 0.7.17-r1188 | quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:219b6c272b25e7e642ae3ff0bf0c5c81a5135ab4-0 |
| samtools | 1.16.1 | same image |
| bcftools | 1.21 | quay.io/biocontainers/bcftools:1.21--h3a4d415_1 |
| Python | 3.12 | docker.io/library/python:3.12-slim |
Containers are not required. A process can load environment modules instead with the Nextflow module directive, which is convenient on campus but ties the workflow to RCAC module names. This workflow uses containers so it runs the same everywhere.
Running on Negishi¶
Run slist to see the SLURM accounts you can charge. Apptainer is available on Negishi without loading a module, and the purdue_negishi profile caches the container images under $RCAC_SCRATCH/.apptainer/cache, so each image is pulled once and reused by later runs.
Purge modules before launching Nextflow
Start with module --force purge. The sticky XALT module sets LD_PRELOAD, SLURM passes your environment to every job Nextflow submits, and that LD_PRELOAD breaks programs inside containers. A plain module purge does not remove sticky modules.
While it runs, squeue -u $USER shows the jobs Nextflow submitted: one index job, then six BWA_MEM jobs at once, then the calling and summary jobs. Results land in:
Open results/pipeline_info/timeline.html in a browser to see the six alignment tasks running side by side.
Where the Nextflow process runs
The Nextflow head process only submits and watches jobs, so for short runs it can run on a front-end inside tmux or screen. For long workflows, put the same nextflow run command in a 1-core sbatch job with a time limit longer than the whole workflow.
Resuming and changing options¶
Nextflow caches every task by its inputs and its script. Add -resume and anything whose inputs and commands are unchanged is reused.
To change a tool option for one run, write the override in a small config file:
| stricter_mapq.config | |
|---|---|
BWA_INDEX and all six BWA_MEM tasks are reported as cached; only BCFTOOLS_CALL and SUMMARIZE run again, because only the calling command changed. Changing a parameter such as --min_qual 50 reruns only SUMMARIZE.
Resume needs the same launch directory
Nextflow records each run in .nextflow/ in the directory you launched from, and keeps task outputs in the work directory (-w). Run the -resume command from the same launch directory with the same -w, and do not delete the work directory until you no longer need to resume.
Handing it to a collaborator¶
A collaborator needs four things from you: the code in a Git repository, a version tag, pinned software (the container images), and a README that names the inputs. With those, they run your exact workflow with their own samplesheet:
Nextflow downloads the tagged version from GitHub; nobody edits a script. If their organism is diploid, they pass a config file that changes ext.args2 for BCFTOOLS_CALL, still without touching the code. On another cluster they use that cluster's nf-core institutional profile, if one exists, or write their own, and keep everything else.
Adapting it to your own scripts¶
- List your steps and mark each as once, per sample, or needs all samples. That gives you singletons, fan-out, and fan-in.
- Make one process per step. Paste the commands, declare inputs and outputs, and replace file names with variables. Keep processes small enough that rerunning one is cheap.
- Write a samplesheet with one row per sample and the columns your first step needs.
- Move tool options to
ext.argsand project-level choices toparams. - Pin your software with a container per process.
- Test on two or three samples with a short samplesheet, then run the full set.
- Commit, tag, and write the README before you hand it over.
Check syntax before you submit anything
nextflow lint main.nf checks your scripts against the strict syntax that Nextflow 26.04 uses by default, without running anything.
Troubleshooting¶
A task failed: where to look
Nextflow prints the failing process, its exit status, and its work directory. In that directory, .command.sh is the exact script that ran, .command.err and .command.out hold its output, and .command.run is the SLURM wrapper. Fix the cause, then rerun the same command with -resume.
Jobs stay pending
squeue -u $USER shows the reason. Check that --cluster_account is a SLURM account you can charge (slist lists yours).
Image pulls are slow or time out
The first run pulls each image once into the cache that the purdue_negishi profile sets, $RCAC_SCRATCH/.apptainer/cache. Later runs reuse it. If pulls time out, rerun the same command with -resume; images that finished downloading are not pulled again.
Related guides¶
- Nextflow and nf-core on Negishi: installing Nextflow and running nf-core pipelines.
- Reproducible Bioinformatics Using Nextflow: Nextflow fundamentals on Gautschi.
- Reproducible Genomics on RCAC: the Session 1 companion guide.
- Session 6: nf-core on Open OnDemand: launching nf-core pipelines from a web form, the next session in the series.