.. _piperun:

Running Pipelines
=================


Available pipelines
-------------------

A list of available pipelines known to the scheduler can be obtained with:

.. code-block:: sh

    owl pdef list

Currently we have the STPT mosaic, the IMC segmentation and the
MerFISH segmentation and decoding pipelines to process data from three
of the main modalities.

There are also two general purpose pipelines that allow the user to execute custom code.

Pipeline definition file
------------------------

In order to submit a pipeline you need to write a pipeline definition file (PDeF).
This is a file in YAML format that specifies the pipeline to run and the arguments
for the pipeline to run as well as the resources needed in the cluster.

A default version of the PDeF file for a pipeline can be obtained with:

.. code-block:: sh

    owl pdef get shell

e.g. for the ``shell`` pipeline (see :ref:`shellpipe` for a description of this pipeline).
This looks like the following:

.. code-block:: yaml

    version: 1.2
    name: shell

    command: ["sleep", "600"]

    resources:
      workers: 1
      memory: 8
      threads: 1


The ``resources`` block defines the number of workers, memory per worker and number of CPUs per worker.

Submitting a pipeline
---------------------

Given a PDeF file, say ``pipeline.yaml``, the pipeline can be submitted with:

.. code-block:: sh

    owl submit pipeline.yaml

The progress can be then monitored in the archive web interface.