What Is a Computation Cluster and How Does Ruffus Run Pipelines on One?

Updated

A computation cluster is a group of networked computers that run jobs in parallel, and Ruffus can send pipeline jobs to one through its multiprocessing and DRMAA support (Chapter 14, "Multiprocessing, drmaa and Computation Clusters"). You need this if your Ruffus pipeline has independent tasks — one file per sample, for example — that would take too long to run sequentially on a single machine. If your pipeline is small or its stages are strictly serial, a cluster adds setup cost without much benefit.

What a cluster gives a pipeline

A single machine runs your Python process on a fixed number of cores. A cluster adds two things:

  • More workers than one machine has cores. Jobs are spread across nodes, so many independent tasks run at once.
  • A scheduler that owns the queue. You don't start jobs directly; you submit them, and the scheduler decides when and where they run.

Ruffus fits this model because its decorators already express dependencies between tasks. Ruffus knows which jobs are ready, which are waiting on inputs, and which are already up to date — the same information a scheduler needs to keep a queue full.

How Ruffus submits jobs

Ruffus documents two mechanisms in Chapter 14:

Mechanism What it does When it fits
Multiprocessing Runs jobs in parallel using local processes One machine with multiple cores
DRMAA Submits jobs to a cluster scheduler through the DRMAA API A real cluster with a queue

DRMAA is the bridge: instead of Ruffus spawning a local process, it hands the job to the cluster's scheduler, which places it on a node. From the pipeline's point of view the task is still just a function with inputs and outputs — only the execution location changes.

What you need before it works

  1. A pipeline whose tasks are actually independent. Ruffus can only parallelise jobs that don't depend on each other's output. If every stage waits on the previous one, the cluster won't help.
  2. A DRMAA-capable scheduler on the cluster side, and the DRMAA library available to the Python environment running Ruffus.
  3. A working pipeline first. Run it locally and confirm the dependency graph is correct — pipeline_printout(...) (Chapter 5) shows what Ruffus thinks will run, and pipeline_printout_graph(...) (Chapter 7) draws it. Debugging job submission and pipeline logic at the same time is much harder.
  4. Shared storage for inputs and outputs. Cluster nodes are separate machines; a job that writes to a local path on one node may be invisible to the next. Output paths need to be reachable from every node that might run a dependent job.

Common pitfalls

  • Assuming more nodes means faster. If the pipeline is mostly serial, or if jobs are tiny, scheduler overhead dominates and total runtime can get worse.
  • Local paths that aren't shared. The most common failure: a job succeeds on one node, and the next job can't find its input.
  • Losing track of what's up to date. Ruffus checks whether outputs exist and are newer than inputs. On a cluster, stale or partially written files from a killed job can make Ruffus skip work it should redo. Chapter 10 covers checkpointing and interrupted pipelines.
  • Debugging on the cluster. Get the pipeline correct locally first, then move execution to the cluster.