Skip to content

Running containers with apptainer

Learning outcomes

After having completed this chapter you will be able to:

  • Login to a remote machine with ssh
  • Use apptainer pull to convert an image from dockerhub to the ‘apptainer image format’ (.sif)
  • Execute a apptainer container
  • Explain the difference in default mounting behaviour between docker and apptainer
  • Use apptainer shell to generate an interactive shell inside a .sif image
  • Search and use images with both docker and apptainer from bioconda

Material

Download the presentation

Exercises

Login to remote

If you are enrolled in the course, you have received an e-mail with an IP, username, private key and password. To do the Apptainer exercises we will login to a remote server. Below you can find instructions on how to login.

VScode is a code editor that can be used to edit files and run commands locally, but also on a remote server. In this subchapter we will set up VScode to work remotely.

If not working with VScode

If you are not working with VScode, you can login to the remote server with the following command:

ssh -i key_username.pem username@123.456.789.123

If you want to edit files directly on the server, you can mount a directory with sshfs.

Required installations

For this exercise it is easiest if you use VScode. In addition you would need to have followed the instructions to set up remote-ssh:

  • OpenSSH compatible client. This is usually pre-installed on your OS. You can check whether the command ssh exists.
  • The Remote-SSH extension. To install, open VSCode and click on the extensions icon (four squares) on the left side of the window. Search for Remote-SSH and click on Install.

Open a PowerShell and cd to the directory where you have stored your private key. After that, move it to ~\.ssh:

mv .\key_username.pem ~\.ssh

Open a terminal, and cd to the directory where you have stored your private key. After that, change the file permissions of the key and move it to ~/.ssh:

chmod 400 key_username.pem
mv key_username.pem ~/.ssh

Open VScode and click on the green or blue button in the bottom left corner. Select Connect to Host..., and then on Configure SSH Host.... Specify a the location for the config file. Use the same directory as where your keys are stored (so ~/.ssh/config). A skeleton config file will be provided. Edit it, so it looks like this (replace username with your username, and specify the correct IP at HostName):

Host sib_course_remote
    User username
    HostName 123.456.789.123
    IdentityFile ~\.ssh\key_username.pem

Host sib_course_remote
    User username
    HostName 123.456.789.123
    IdentityFile ~/.ssh/key_username.pem

Save and close the config file. Now click again the green or blue button in the bottom left corner. Select Connect to Host..., and then on sib_course_remote. You will be asked which operating system is used on the remote. Specify ‘Linux’.

Pulling an image

Apptainer can take several image formats (e.g. a docker image), and convert them into it’s own .sif format. Unlike docker this image doesn’t live in a local image cache, but it’s stored as an actual file.

Exercise: On the remote server, pull the docker image that has the adjusted default CMD that we have pushed to dockerhub in this exercise (ubuntu-figlet-df:v3) with apptainer pull. The syntax is:

apptainer pull docker://[USER NAME]/[IMAGE NAME]:[TAG]
Answer 

apptainer pull docker://[USER NAME]/ubuntu-figlet:v3
This will result in a file called ubuntu-figlet_v3.sif

Note

If you weren’t able to push the image in the previous exercises to your docker hub, you can use geertvangeest as username to pull the image.

Executing an image

These .sif files can be run as standalone executables:

./ubuntu-figlet_v3.sif

Note

This is shorthand for:

apptainer run ubuntu-figlet_v3.sif

And you can overwrite the default command like this:

apptainer run [IMAGE NAME].sif [COMMAND]

Note

In this case, you can also use

./[IMAGE NAME].sif [COMMAND]

However, most applications require apptainer run. Especially if you want to provide options like --bind (for mounting directories).

Exercise: Run the .sif file without a command, and with a command that runs figlet. Do you get expected output? Do the same for the python or R image you’ve created in the previous chapter.

Entrypoint and apptainer

The daterange image has an entrypoint set, and apptainer run does not overwrite it. In order to ignore both the entrypoint and cmd use apptainer exec.

Answer

Running it without a command (./ubuntu-figlet_v3.sif) should give:

__  __         _                                                 _        _
|  \/  |_   _  (_)_ __ ___   __ _  __ _  ___  __      _____  _ __| | _____| |
| |\/| | | | | | | '_ ` _ \ / _` |/ _` |/ _ \ \ \ /\ / / _ \| '__| |/ / __| |
| |  | | |_| | | | | | | | | (_| | (_| |  __/  \ V  V / (_) | |  |   <\__ \_|
|_|  |_|\__, | |_|_| |_| |_|\__,_|\__, |\___|   \_/\_/ \___/|_|  |_|\_\___(_)
       |___/                     |___/
Which is the default command that we changed in the Dockerfile.

Running with a another figlet command:

./ubuntu-figlet_v3.sif figlet 'Something else'

Should give:

____                       _   _     _                    _
/ ___|  ___  _ __ ___   ___| |_| |__ (_)_ __   __ _    ___| |___  ___
\___ \ / _ \| '_ ` _ \ / _ \ __| '_ \| | '_ \ / _` |  / _ \ / __|/ _ \
___) | (_) | | | | | |  __/ |_| | | | | | | | (_| | |  __/ \__ \  __/
|____/ \___/|_| |_| |_|\___|\__|_| |_|_|_| |_|\__, |  \___|_|___/\___|
                                             |___/

Pulling the search_biomart_datasets image:

apptainer pull docker://[USER NAME]/search_biomart_datasets:v1

Running it without command:

./search_biomart_datasets_v1.sif

Running with a command:

./search_biomart_datasets_v1.sif --pattern sapiens

To overwrite both entrypoint and the command:

apptainer exec search_biomart_datasets_v1.sif search_biomart_datasets.R --pattern "(R|r)at"

Pulling the daterange.py image:

apptainer pull docker://[USER NAME]/daterange:v1

Running it without command:

./daterange_v1.sif

Running with a command:

./daterange_v1.sif --date 20221005

To overwrite both entrypoint and the command:

apptainer exec daterange_v1.sif daterange.py --date 20221005

Mounting with Apptainer

Apptainer is also different from Docker in the way it handles mounting. By default, Apptainer binds your home directory and a number of paths in the root directory to the container. This results in behaviour that is almost like if you are working on the directory structure of the host.

If your directory is not mounted by default

It depends on the apptainer settings whether most directories are mounted by default to the container. If your directory is not mounted, you can do that with the --bind option of apptainer exec:

apptainer exec --bind /my/dir/to/mount/ [IMAGE NAME].sif [COMMAND]

Running the command pwd (full name of current working directory) will therefore result in a path on the host machine:

./ubuntu-figlet_v3.sif pwd

Exercise: Run the above command. What is the output? How would the output look like if you would run a similar command with Docker?

Hint

A similar Docker command would look like (run this on your local computer):

docker run --rm ubuntu-figlet:v3 pwd
Answer

The output of ./ubuntu-figlet_v3.sif pwd is the current directory on the host: i.e. /home/username if you have it in your home directory. The output of docker run --rm ubuntu-figlet:v3 pwd (on the local host) would be /, which is the default workdir (root directory) of the container. As we did not mount any host directory, this directory exists only within the container (i.e. separated from the host).

Interactive shell

If you want to debug or inspect an image, it can be helpful to have a shell inside the container. You can do that with apptainer shell:

apptainer shell ubuntu-figlet_v3.sif

Note

To exit the shell type exit.

Exercise: Can you run figlet inside this shell?

Answer

Yes: 

Apptainer> figlet test
 _            _
| |_ ___  ___| |_
| __/ _ \/ __| __|
| ||  __/\__ \ |_
 \__\___||___/\__|

During the lecture you have learned that apptainer takes over the user privileges of the user on the host. You can get user information with command like whoami, id, groups etc.

Exercise: Run the figlet container interactively. Do you have the same user privileges as if you were on the host? How is that with docker?

Answer

A command like whoami will result in your username printed at stdout:

Apptainer> whoami
myusername
Apptainer> id
uid=1030(myusername) gid=1031(myusername) groups=1031(myusername),1001(condausers)
Apptainer> groups
myusername condausers

With apptainer, you have the same privileges inside the apptainer container as on the host. If you do this in the docker container (based on the same image), you’ll get output like this:

root@a3d6e59dc19d:/# whoami
root
root@a3d6e59dc19d:/# groups
root
root@a3d6e59dc19d:/# id
uid=0(root) gid=0(root) groups=0(root)

A bioinformatics example (extra)

All bioconda packages also have a pre-built container. Have a look at the bioconda website, and search for fastqc. In the search results, click on the appropriate record (i.e. package ‘fastqc’). Now, scroll down and find the namespace for the latest fastqc image.

Exercise: Check out the container image at quay.io, by following quay.io/biocontainers/fastqc. Choose a tag, and pull it with apptainer pull.

Answer

It’s up to you which tag you choose. The tag with the latest version is 0.12.1--hdfd78af_0.

apptainer pull docker://quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0

Apptainer images at galaxy.org

Most (if not all) biocontainer images are available as apptainer (singularity) image at https://depot.galaxyproject.org/singularity/. You can simply download them with wget or curl, e.g.:

wget https://depot.galaxyproject.org/singularity/fastqc%3A0.12.1--hdfd78af_0

Let’s test the image. Download some sample reads first:

mkdir reads
cd reads
wget https://introduction-containers.s3.eu-central-1.amazonaws.com/ecoli_reads.tar.gz
tar -xzvf ecoli_reads.tar.gz
rm ecoli_reads.tar.gz

Now you can simply run the image as an executable preceding the commands you would like to run within the container. E.g. running fastqc would look like:

cd
./fastqc_0.12.1--hdfd78af_0.sif fastqc ./reads/ecoli_*.fastq.gz

This will result in html files in the directory ./reads. These are quality reports for the sequence reads. If you’d like to view them, you can download them with scp or e.g. FileZilla, and view them with your local browser.

Questions & Answers