简体中文
EnglishWebsite review
This review may include AI-generated content and is not individually verified. Domain, certificate and other facts come from the observed data on this page.
What is Ruffus?
Ruffus is a Python library for building and running computational pipelines. You write ordinary Python functions, attach decorators to describe how each step depends on the ones before it, and Ruffus works out the order to run them, which steps are already up to date, and how to spread the work across processors or a compute cluster. Its documentation presents it as open-source, powerful and user-friendly, with wide use in science and bioinformatics Ruffus.
What it actually does
- Describes dependencies between tasks. Decorators such as
@transform,@merge,@split,@subdivideand@collateexpress common pipeline shapes — one file to one file, many files to one, one file split into many, and so on. - Tracks what is up to date. Ruffus compares input and output files so an interrupted or repeated run skips completed work instead of redoing it. Chapter 10 of the manual covers checkpointing after interruptions and exceptions.
- Runs work in parallel. Chapter 14 covers multiprocessing, DRMAA and computation clusters, so the same pipeline can run locally or be dispatched to a scheduler.
- Lets you inspect before you run.
pipeline_printout(...)shows what would execute, andpipeline_printout_graph(...)draws the pipeline as a flow chart. - Supports command-line use. The
ruffus.cmdlinemodule turns a pipeline script into a command-line tool with logging and job control.
Who it suits
It fits researchers and engineers who already have a set of scripts — alignment, simulation, image processing, model fitting — and want them chained with reruns, parallelism and a record of what happened, without adopting a heavyweight workflow engine. Version 2.6 introduces an object-oriented syntax alongside the original decorator style, which matters if you prefer classes over module-level functions.
Trade-offs to weigh
Ruffus is a library inside your Python process, not a server: there is no central scheduler UI, no database of run history, and pipelines are code you maintain yourself. That keeps setup light but means provenance, retries across machines and multi-user scheduling are your responsibility. File-based dependency tracking also assumes your steps produce files; steps that only pass data in memory need the @parallel or custom-uptodate routes described in the later "esoteric" chapters.
A concrete next step
Take one manual step you currently rerun by hand — say, converting raw instrument output into a summary table — and rewrite it as a single decorated task. Run pipeline_printout(...) to confirm Ruffus sees the dependency, then add a second task with @transform and rerun; if the second run skips the first task, the file-based up-to-date logic is working and you can add parallel execution.
How do I install Ruffus and run my first pipeline?
Install Ruffus with pip, then write a small Python script that chains tasks with decorators such as @transform or @merge, and run it from the command line. The documentation lists an Installation chapter and a command-line chapter (ruffus.cmdline), so both routes are supported.
Install
pip install ruffus
If you prefer the source, the project points to a git checkout; the documented "Get Ruffus" and "Installation" sections are the places to confirm the current method. Ruffus is a pure-Python library, so no compiler or cluster is needed to get started.
Your first pipeline
A pipeline is just Python functions decorated to declare dependencies. A minimal pattern:
from ruffus import *
@originate("hello.txt")
def create_file(output_file):
with open(output_file, "w") as f:
f.write("hello\n")
@transform(create_file, suffix(".txt"), ".upper")
def uppercase(input_file, output_file):
with open(output_file, "w") as out:
out.write(open(input_file).read().upper())
pipeline_run([uppercase])
Run it with python my_pipeline.py. Ruffus checks which output files already exist and only runs the stages that are out of date, so re-running is cheap. Use pipeline_printout() to see what would run before committing, and pipeline_printout_graph() to get a visual diagram of the task graph.
Command-line route
The documented ruffus.cmdline module lets you expose the same pipeline as a shell command with options for parallel jobs, logging and dry runs. This is the usual choice when the pipeline is shared with collaborators or submitted to a cluster.
Practical checks
- Start small. Two or three tasks are enough to learn the decorator style; the manual's early chapters build up from there.
- Name outputs explicitly. Most confusion comes from suffix and formatter rules, covered in the formatter/regex chapter.
- Inspect before running.
pipeline_printout()is the fastest way to catch a wrong dependency. - Scale later. Multiprocessing and DRMAA cluster execution are separate chapters; you do not need them for a first pipeline.
For background on the scientific workflow ecosystem Ruffus sits in, see Snakemake and Nextflow.
How does Ruffus decide which pipeline tasks need to be rerun?
Ruffus decides what to rerun by checking, for each task, whether the files it is supposed to produce are missing or older than the files it depends on. In Ruffus terms, this is dependency and up-to-date checking: a task is considered out of date when its output does not exist or when its inputs are newer than its outputs. Only out-of-date tasks, and the tasks that depend on them, are executed in a run. The manual devotes a chapter to this behavior (Chapter 10, "Checkpointing: Interrupted Pipelines and Exceptions") and an appendix to "How dependency is checked."
The practical rules
- Missing output means rerun. If a task's declared output file is absent, that task runs.
- Newer input means rerun. If an input file has a later modification time than the output, the task runs.
- Propagation through the graph. When an upstream task reruns and rewrites its output, downstream tasks see changed inputs and rerun in turn.
- Unchanged tasks are skipped. A task whose inputs and outputs are all present and timestamps look consistent is left alone.
- Custom logic is possible. Ruffus supports writing your own function to decide whether jobs are up to date, via the
@check_if_uptodatedecorator (described as "esoteric" in the manual). This lets you replace timestamp comparison with checksums, database lookups or other criteria. - Interrupted runs resume. Because decisions are file-based, a pipeline stopped partway can be restarted and will pick up from the first incomplete stage.
Why this design matters
The approach is deliberately lightweight: Ruffus does not maintain a separate state database. Everything it needs is inferred from the filesystem and the decorators that describe each task's inputs and outputs. That makes pipelines portable and easy to inspect, and it means you can reason about reruns just by looking at file timestamps. The trade-off is that timestamp-based checking can be fooled — for example, by a file copied with a preserved old timestamp, or by an input that changes without its mtime updating. Where that matters, @check_if_uptodate is the escape hatch.
A concrete scenario
Suppose you have a pipeline that trims reads, aligns them and calls variants. You edit the trimming parameters and rerun. Ruffus sees the trim task's inputs are newer than its outputs, so trimming reruns and writes new trimmed files. The aligner then finds its inputs newer than its BAM files, so it reruns, and variant calling follows. If instead you only added a new sample at the start, the existing samples' tasks stay up to date and only the new sample's path through the graph executes.
Next step
Before running a long job, use pipeline_printout(...) to see which tasks Ruffus considers out of date, and pipeline_printout_graph(...) to visualize the flow. If timestamp checking is not reliable for your data, look at Chapter 23 on writing custom up-to-date functions. The project documentation is at Ruffus.
How can I run Ruffus pipelines in parallel or on a computation cluster?
Ruffus handles parallelism and cluster execution through its decorators and its command-line module, not through a separate scheduler you configure up front. You mark tasks so they can run concurrently, then choose where the work executes when you start the pipeline.
Parallelism within one machine
Ruffus can run independent tasks in parallel using multiple processes. The documentation covers this under multiprocessing, and the practical effect is that stages which do not depend on each other can be dispatched at the same time. The decorators describing pipeline topology — for example @transform, @split, @merge, @subdivide, @collate — determine what can run concurrently, because Ruffus derives dependencies from the input and output files you declare. If your task declarations are accurate, the scheduler can safely overlap work; if they are vague, it will serialize more than necessary.
Running on a computation cluster
Chapter 14 of the manual, "Multiprocessing, drmaa and Computation Cluster", is the relevant reference. Ruffus integrates with DRMAA, the standard API for submitting jobs to cluster schedulers, so instead of running each job as a local process it submits them to the cluster's queueing system. You still describe the pipeline in Python; the change is in how jobs are launched. This is why Ruffus is common in bioinformatics, where pipelines mix many small tasks with a few large ones and the cluster is the natural place to run them.
Practical route
- Write the pipeline with explicit input/output files for each task.
- Verify the topology with
pipeline_printout(...)before running anything — this shows the order and grouping Ruffus intends. - Inspect it visually with
pipeline_printout_graph(...)if the dependencies are hard to follow in text. - Run locally with multiprocessing first, on a small subset of data, to confirm the tasks are genuinely independent.
- Switch to cluster submission via the DRMAA route described in Chapter 14, and use
ruffus.cmdline(Chapter 6) to control the run from the shell. - Add logging (Chapter 15) and checkpointing (Chapter 10) so an interrupted cluster run can resume rather than restart.
Trade-offs and decision criteria
| Situation | Reasonable choice |
|---|---|
| Few tasks, small data, one workstation | Serial or simple multiprocessing |
| Many independent tasks, one multi-core machine | Multiprocessing |
| Long-running jobs, queue-managed resources, shared cluster | DRMAA submission |
| Pipeline interrupted often | Checkpointing plus logging |
The main cost of cluster execution is latency: queue wait times can exceed the runtime of short tasks, so grouping small jobs with @merge, @collate or @subdivide often matters more than raw parallelism. A useful rule is to parallelize at the level where tasks are already independent and substantial, and to keep the dependency declarations honest so Ruffus does not over-schedule.
For the authoritative details, start with the Ruffus manual at Ruffus and read Chapter 14 alongside Chapter 6.
How do I visualize or debug a Ruffus pipeline before running it?
Ruffus provides two main tools for inspecting a pipeline without executing it: pipeline_printout(...) and pipeline_printout_graph(...). Both let you see what the pipeline would do, which tasks depend on which, and whether the flow of data matches your intent.
Text output with pipeline_printout
pipeline_printout(...) writes a textual description of the pipeline to standard output. It lists tasks in dependency order, shows input and output files, and indicates whether each task is up to date or needs to run. This is the quickest way to check that your decorators (@transform, @merge, @split, @collate, etc.) are wired together as expected before committing compute time.
A typical use is to call it near the end of a pipeline script, or from the command line via ruffus.cmdline, passing a target task or output file. You can narrow the printout to a subset of the pipeline, which helps when a full graph is large.
Visual output with pipeline_printout_graph
pipeline_printout_graph(...) renders the pipeline as a flow chart, typically to a file such as a PDF, SVG or PNG. Nodes represent tasks and edges represent dependencies, so you can spot orphaned tasks, unexpected fan-in or fan-out, and tasks that never connect to the rest of the pipeline. The documentation includes an appendix on flow-chart colours, which encode task states and types, so the diagram is more informative than a plain dependency graph.
This is especially useful for the topologies covered in the manual — @split, @merge, @subdivide/@collate, and the all-versus-all @combinations, @permutations and @product decorators — where the shape of the pipeline is hard to verify by reading code alone.
Choosing between them
| Goal | Tool | Why |
|---|---|---|
| Check task order and file names quickly | pipeline_printout(...) |
Fast, textual, easy to diff or grep |
| Understand complex fan-in/fan-out | pipeline_printout_graph(...) |
Shows topology visually, colour-coded |
| Verify a subset of the pipeline | Either, with a target task | Limits noise in large pipelines |
| Share a pipeline design with collaborators | pipeline_printout_graph(...) |
A diagram communicates structure at a glance |
Practical next step
Run pipeline_printout(...) first on a small target task. If the printed order and file names look wrong, fix the decorators before generating a graph. Once the text output is sensible, produce a graph for the full pipeline and check that every branch terminates in the outputs you actually need. This two-step check catches most wiring mistakes before any jobs are submitted, which matters when the pipeline would otherwise run on a cluster via multiprocessing or DRMAA.
For background on pipeline design and the full list of decorators, see the Ruffus documentation at Ruffus.
What is the new object-oriented syntax in Ruffus 2.6 and how does it differ from the decorator syntax?
Ruffus 2.6 introduced an object-oriented syntax as an alternative way to define pipeline tasks, alongside the decorator-based style that earlier versions used. The documentation lists "New Object Orientated Syntax" and a "Worked Example for New Object orientated syntax for Ruffus in Version 2.6" as prominent manual entries, which indicates the feature was treated as a significant addition rather than a minor variation.
What the decorator syntax does
In the classic Ruffus style, you write a plain Python function and attach behaviour by stacking decorators above it. Each decorator declares a dependency or an action: @transform derives new jobs from existing outputs, @originate creates files from nothing, @split divides a large task, @merge combines many inputs into one result, and @mkdir prepares directories. The function body contains only the work for a single job. The manual's early chapters build up exactly this vocabulary, and later chapters add @active_if, @posttask, @subdivide, @collate, @combinations, @permutations and @product.
This style is compact and reads top-to-bottom: decorators state the pipeline topology, and the function states the computation. It is also the style most existing Ruffus code uses, so examples, answers and older pipelines you encounter will overwhelmingly be decorator-based.
What the object-oriented syntax changes
The object-oriented form expresses the same pipeline through objects and methods instead of decorator annotations. The practical consequences are the interesting part:
- Discoverability. Methods on a task object can be listed by an editor or an interactive session, whereas decorators are invisible at runtime until applied. If you are learning the library or returning to a pipeline after months away, an object interface is easier to inspect.
- Programmatic construction. Building pipelines in loops, from configuration, or from another data structure is more natural when tasks are objects you can pass around, store in lists and configure conditionally. Decorator syntax can do this too, but it becomes awkward.
- Composition and reuse. Shared setup can live in a base class or a helper that returns a configured task, rather than being copied as a decorator stack.
- Familiarity. Programmers coming from workflow tools with explicit task objects, or from object-oriented Python generally, may find the mapping more immediate.
The trade-off is the reverse of these: the object syntax is newer, so fewer external examples exist, and the enormous body of Ruffus documentation, forum answers and third-party pipeline code is written in decorators. Mixing the two styles in one project is possible but makes the pipeline harder to read as a whole.
Which to choose
| Situation | Better fit |
|---|---|
| Adapting an existing Ruffus pipeline | Decorator syntax |
| Learning from the manual and community examples | Decorator syntax |
| Generating tasks dynamically from config or data | Object-oriented syntax |
| Large pipelines needing shared setup and reuse | Object-oriented syntax |
| Team already comfortable with explicit task objects | Object-oriented syntax |
A reasonable next step is to open the manual's "Worked Example for New Object orientated syntax for Ruffus in Version 2.6" and rewrite one small pipeline you already understand — for instance a @transform followed by a @merge — in both styles. Comparing the two versions of a pipeline whose behaviour you can predict is the fastest way to judge which one you want to maintain. The official documentation is at Ruffus.
IONOS SE
Germany
User reviews (0)