Using Singularity with Slurm

Created: July 21, 2023

Last update: August 6, 2023

In the previous article we saw how to install Singularity and now it’s time to make some practical example of its usage.

it allows to use most of the Docker containers available on DockerHub.

Let’s try to use a command that’s not available in the current machine:

orangepi@orange-1 ~$ parallel
Command 'parallel' not found, but can be installed with:
sudo apt install moreutils
sudo apt install parallel 
orangepi@orange-1 ~$

The Debian I’ve installed on the OrangePis doesn’t include the program “parallel” and the system invites me to install it.

I wanted to keep the operating system as simple as possible and avoid to install any program other then Singularity.

orangepi@orange-1 ~$ singularity --version
singularity-ce version 3.11.3

First run

The easiest manner to user Singularity with a Docker container is to pull it directly from DockerHub at every call.

Because of the current configuration with the shared home folder it’s better to generate a SIF file on the local storage (ex: ‘/media/singularity/’) to be able to use the container in the most efficient way.

We can choose a container that has the program we want to execute.

orangepi@orange-1 ~$ SINGULARITY_TMPDIR=/media/tmp singularity build /media/singularity/restoration.sif docker://lbecchi/restoration:latest
INFO:    Starting build...
2023/08/03 17:12:01  info unpack layer: sha256:a39c84e173f038958d338f55a9e8ee64bb6643e8ac6ae98e08ca65146e668d86
2023/08/03 17:12:03  info unpack layer: sha256:61c10baa037f179b93e728a17cac1c69a0534f2797111c9741f7d3c7efd9b11f
2023/08/03 17:12:09  info unpack layer: sha256:64c16b50b0bc25ac294da68726573a656370eee439570d46258ec3fb2960ea4f
2023/08/03 17:12:10  info unpack layer: sha256:255027cfe6b5bb30c76ad3307b536b0c0a9392067efb7033c31a7a71e3395ef2
2023/08/03 17:12:10  info unpack layer: sha256:f503e4beba86e4116fb7c4c08aa704d11ca2eacef306a18733e665a4ee2afb9e
2023/08/03 17:12:14  info unpack layer: sha256:8f59957e2a7d21c438d19f961367f03aab3b3bf0a5d8c08c40d2db819fd9dde2
2023/08/03 17:12:14  info unpack layer: sha256:26a7d361035fef10ebe8cd1bc0ff6985e413fd8e07d84bbd34fabb304575b22d
2023/08/03 17:12:14  info unpack layer: sha256:d19be554a04fbf5f202c43dc7ac07e1a54395548529d879ddcafe571d9ebab9c
2023/08/03 17:12:14  info unpack layer: sha256:a4f8fad9f22ff89fe4c9f633dbe6654909c172af515bb4cceff1679e2d037dbe
2023/08/03 17:12:14  info unpack layer: sha256:29b713fa28c599fe9a36ce53636ebe307fd819016bec5643a83175e0ae0c8d1c
2023/08/03 17:12:14  info unpack layer: sha256:43ade19dc1b1c9e827970922bb46e1d4e047a37aef04b111afb137d206ccf14d
2023/08/03 17:12:14  info unpack layer: sha256:b0cf0d69dbf292f4a0d97895294f658f268f8e36e993e0839bd71a191174a0fa
2023/08/03 17:12:14  info unpack layer: sha256:5aeeb4e2ff578377ddac351334ae7b393debc63739d2db02a0e30aa4d1542103
2023/08/03 17:12:14  info unpack layer: sha256:420c4441e6622ee3bb5747348b6c4d7ba895793021008df608a94f69fffac358
2023/08/03 17:12:15  info unpack layer: sha256:a968148337d0f7e3e2454b8c5cbce7c645274fe668221f88c59009fdc929bc94
2023/08/03 17:12:15  info unpack layer: sha256:56b433c4da44343cd204e79c624d0f0668657caaaed00dcbae94da0239e433c0
2023/08/03 17:12:15  info unpack layer: sha256:857320bd5833891d605ee64eb279aa4d17c7760cb848d307da5cb313011dc907
2023/08/03 17:12:16  info unpack layer: sha256:62c2a95ed0eec1f2ad011192308da3c0c8efa212dc52650c1232cae56cd27993
2023/08/03 17:12:16  info unpack layer: sha256:142b9cea6bec6f4796a205b7741409ae593c3ce2a05fde3ec97d13aed8648d40
2023/08/03 17:12:17  info unpack layer: sha256:3c09fdcc06c4eeb84b89f46fa0bfe2bf072797f4bfe81d2673473fb208278c18
2023/08/03 17:12:22  info unpack layer: sha256:2167d5c0def312b78474f50dca38319a1539c1a739cbf306eb75f6e2f1fbf02f
2023/08/03 17:12:22  info unpack layer: sha256:34548aa4009564d9d447ecc662ba815a03adcbdf7dc3449f45ed611d74467db4
INFO:    Creating SIF file...
INFO:    Build complete: /media/tmp/restoration.sif

It takes a bit of time to synchronize all the layers and generate the SIF package.

The final product “/media/singularity/restoration.sif” can be used without having to pass from DockerHub every time we launch a command through Singularity.

orangepi@orange-1 ~$ singularity run /media/singularity/restoration.sif parallel --version
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

Now “GNU parallel” is recognized by the system if it now installed on it.

Shell

Singularity offers the possibility to open a shell on the downloaded container:

orangepi@orange-1 ~$ singularity shell /media/tmp/restoration.sif
Singularity> parallel --version
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel
Singularity>

Using it with the cluster

The syntax of Singularity calls might become quit long, to make it easy we can create some shortcut variables:

orangepi@slurm-boss ~$ parallel="singularity run /media/singularity/restoration.sif parallel"
orangepi@slurm-boss ~$ srun $parallel --version
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

In this case the command is launched from the master node, where Singularity is not installed, and it’s executed on the first available node where Singularity is available and the SIF file has been already created or copied.

As example we can define a Bash function that print a sequence of integers and the host name of the machine who’s computing the command:

orangepi@slurm-boss ~$ hostname_and_numbers(){ echo $1 $(hostname);}
orangepi@slurm-boss ~$ export -f hostname_and_numbers

Now we can run it on the first node available:

orangepi@slurm-boss ~$ srun $parallel hostname_and_numbers $1 ::: {1..3}
1 orange-1
2 orange-1
3 orange-1

And finally trying it on every node:

orangepi@slurm-boss ~$ srun --nodes 5 $parallel hostname_and_numbers $1 ::: {1..3}
1 orange-4
2 orange-4
3 orange-4
1 orange-2
2 orange-2
3 orange-2
1 orange-5
2 orange-5
3 orange-5
1 orange-3
2 orange-3
3 orange-3
1 orange-1
2 orange-1
3 orange-1

Notes

From this simple exercise we can see how we can define a shortcut to our favorite commands from a Docker container and use them on every node of our cluster without reinstalling the software again and again.

The maintenance of the version of a software is managed directly into the Docker container.

The same container can be prepared to run both in ARM and X86 infrastructure and it allows for great consistency in heterogeneous environments.