Running Pipelines

Available pipelines

A list of available pipelines known to the scheduler can be obtained with:

owl pdef list

Currently we have the STPT mosaic, the IMC segmentation and the MerFISH segmentation and decoding pipelines to process data from three of the main modalities.

There are also two general purpose pipelines that allow the user to execute custom code.

Pipeline definition file

In order to submit a pipeline you need to write a pipeline definition file (PDeF). This is a file in YAML format that specifies the pipeline to run and the arguments for the pipeline to run as well as the resources needed in the cluster.

A default version of the PDeF file for a pipeline can be obtained with:

owl pdef get shell

e.g. for the shell pipeline (see Shell Pipeline for a description of this pipeline). This looks like the following:

version: 1.2
name: shell

command: ["sleep", "600"]

resources:
  workers: 1
  memory: 8
  threads: 1

The resources block defines the number of workers, memory per worker and number of CPUs per worker.

Submitting a pipeline

Given a PDeF file, say pipeline.yaml, the pipeline can be submitted with:

owl submit pipeline.yaml

The progress can be then monitored in the archive web interface.