Data Pipelines

Note

The most recent documentation on using the Owl pipeline scheduler is in https://eddienko.github.io/owl-pipeline/

Pipelines allow to run non-interactive jobs in the cluster with larger resource allocation that Jupyter notebooks or remote desktops.

Pipelines are run submitting them to our custom scheduler named Owl. The technical implementation is described elsewhere. Here we describe how to run pipelines in the cluster.