Long Running Processes

Warning

This feature is in development.

While notebooks are a great resource for explanatory analysis, they do not allow to execute very long running processes. The notebook environment is stopped when a user logs out.

For this reason we have made available the following functions that allow restricted submission of long running processes that continue runing until completion or termination even if the user logs out.

%lrp run <cmd>

Run the command specified as a long running process in the cluster. If the job is correctly scheduled it outputs the name that can be used to identify the process (e.g. lrp-eglez-1e2553836ebc).

A simple command that simply sleeps 60 seconds

%lrp run sleep 60

In order to run a script in /data/meds1_a/jimaxt/eglez/run.py

%lrp run python /data/meds1_a/jimaxt/eglez/run.py

Note that files need to be available in one of the /data directories, they will need to read and output data to the shared disks as well. Notebook home disks are currently not mounted.

Commands are not limited to Python scripts, a bash script can be run as shown below, but please make sure that the script is executable (chmod +x /data/meds1_a/jimaxt/eglez/run.sh)

%lrp run /data/meds1_a/jimaxt/eglez/run.sh

%lrp logs <name>

Display the output of the running job identified by name

For example:

%lrp logs lrp-eglez-1e2553836ebc

%lrp status <name>

Display the status of the running job identified with name.

For example:

%lrp logs lrp-eglez-1e2553836ebc

STATUS(lrp-eglez-1e2553836ebc): Succeeded

%lrp delete <name>

Delete the running job identified with name.

For example:

%lrp delete lrp-eglez-1e2553836ebc

STATUS(lrp-eglez-1e2553836ebc): Deleted