Working with dockerfiles
Learning outcomes
After having completed this chapter you will be able to:
- Build an image based on a dockerfile
- Use the basic dockerfile syntax
- Change the default command of an image and validate the change
- Map ports to a container to display interactive content through a browser
Material
Exercises
To make your images shareable and adjustable, it’s good practice to work with a Dockerfile. This is a script with a set of instructions to build your image from an existing image.
Basic Dockerfile
You can generate an image from a Dockerfile using the command docker build. A Dockerfile has its own syntax for giving instructions. Luckily, they are rather simple. The script always contains a line starting with FROM that takes the image name from which the new image will be built. After that you usually want to run some commands to e.g. configure and/or install software. The instruction to run these commands during building starts with RUN. In our figlet example that would be:
FROM ubuntu:focal-20210401
RUN apt-get update
RUN apt-get install figlet
On writing reproducible Dockerfiles
At the FROM statement in the the above Dockerfile you see that we have added a specific tag to the image (i.e. focal-20210401). We could also have written:
FROM ubuntu
RUN apt-get update
RUN apt-get install figlet
This will automatically pull the image with the tag latest. However, if the maintainer of the ubuntu images decides to tag another ubuntu version as latest, rebuilding with the above Dockerfile will not give you the same result. Therefore it’s always good practice to add the (stable) tag to the image in a Dockerfile. More rules on making your Dockerfiles more reproducible here.
Exercise: Create a file on your computer called Dockerfile, and paste the above instruction lines in that file. Make the directory containing the Dockerfile your current directory. Build a new image based on that Dockerfile with:
docker build .
docker build --platform amd64 .
If using an Apple M1 chip (newer Macs)
If you are using a computer with an Apple M1 chip, you have the less common ARM system architecture, which can limit transferability of images to (more common) x86_64/AMD64 machines. When building images on a Mac with an M1 chip (especially if you have sharing in mind), it’s best to specify the --platform amd64 flag.
The argument of docker build
The command docker build takes a directory as input (providing . means the current directory). This directory should contain the Dockerfile, but it can also contain more of the build context, e.g. (python, R, shell) scripts that are required to build the image.
What has happened? What is the name of the build image?
Answer
A new image was created based on the Dockerfile. You can check it with: docker image ls, which gives something like:
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 92c980b09aad 7 seconds ago 101MB
ubuntu-figlet latest e08b999c7978 About an hour ago 101MB
ubuntu latest f63181f19b2f 30 hours ago 72.9MB
It has created an image without a name or tag. That’s a bit inconvenient.
Exercise: Build a new image with a specific name. You can do that with adding the option -t to docker build. Before that, remove the nameless image.
Hint
An image without a name is usually a “dangling image”. You can remove those with docker image prune.
Answer
Remove the nameless image with docker image prune.
After that, rebuild an image with a name:
docker build -t ubuntu-figlet:v2 .
docker build --platform amd64 -t ubuntu-figlet:v2 .
Using CMD
As you might remember the second positional argument of docker run is a command (i.e. docker run IMAGE [CMD]). If you leave it empty, it uses the default command. You can change the default command in the Dockerfile with an instruction starting with CMD. For example:
FROM ubuntu:focal-20210401
RUN apt-get update
RUN apt-get install figlet
CMD figlet My image works!
Exercise: Build a new image based on the above Dockerfile. Can you validate the change using docker image inspect? Can you overwrite this default with docker run?
Answer
Copy the new line to your Dockerfile, and build the new image like this:
docker build -t ubuntu-figlet:v3 .
docker build --platform amd64 -t ubuntu-figlet:v3 .
The command docker inspect ubuntu-figlet:v3 will give:
"Cmd": [
"/bin/sh",
"-c",
"figlet My image works!"
]
So the default command (/bin/bash) has changed to figlet My image works!
Running the image (with clean-up (--rm)):
docker run --rm ubuntu-figlet:v3
Will result in:
__ __ _ _ _
| \/ |_ _ (_)_ __ ___ __ _ __ _ ___ __ _____ _ __| | _____| |
| |\/| | | | | | | '_ ` _ \ / _` |/ _` |/ _ \ \ \ /\ / / _ \| '__| |/ / __| |
| | | | |_| | | | | | | | | (_| | (_| | __/ \ V V / (_) | | | <\__ \_|
|_| |_|\__, | |_|_| |_| |_|\__,_|\__, |\___| \_/\_/ \___/|_| |_|\_\___(_)
|___/ |___/
And of course you can overwrite the default command:
docker run --rm ubuntu-figlet:v3 figlet another text
Resulting in:
_ _ _ _
__ _ _ __ ___ | |_| |__ ___ _ __ | |_ _____ _| |_
/ _` | '_ \ / _ \| __| '_ \ / _ \ '__| | __/ _ \ \/ / __|
| (_| | | | | (_) | |_| | | | __/ | | || __/> <| |_
\__,_|_| |_|\___/ \__|_| |_|\___|_| \__\___/_/\_\\__|
Two flavours of CMD
You have seen in the output of docker inspect that docker translates the command (i.e. figlet "my image works!") into this: ["/bin/sh", "-c", "figlet 'My image works!'"]. The notation we used in the Dockerfile is the shell notation while the notation with the square brackets ([]) is the exec-notation. You can use both notations in your Dockerfile. Altough the shell notation is more readable, the exec notation is directly used by the image, and therefore less ambiguous.
A Dockerfile with shell notation:
FROM ubuntu:focal-20210401
RUN apt-get update
RUN apt-get install figlet
CMD figlet My image works!
A Dockerfile with exec notation:
FROM ubuntu:focal-20210401
RUN apt-get update
RUN apt-get install figlet
CMD ["/bin/sh", "-c", "figlet My image works!"]
Exercise: Now push our created image (with a version tag) to docker hub. We will use it later for the singularity exercises.
Answer
docker tag ubuntu-figlet:v3 [USER NAME]/ubuntu-figlet:v3
docker push [USER NAME]/ubuntu-figlet:v3
Build image for your own script
Often containers are built for a specific purpose. For example, you can use a container to ship all dependencies together with your developed set of scripts/programs. For that you will need to add your scripts to the container. That is quite easily done with the instruction COPY. However, in order to make your container more user-friendly, there are several additional instructions that can come in useful. We will treat the most frequently used ones below.
In the exercises will use a simple script called daterange.py. You can download it here. After you have downloaded it, make sure to set the permissions to executable:
chmod +x daterange.py
Note
Have a look at daterange.py. It is a simple script that uses pandas. It takes a date (in the format YYYYMMDD) as provided by the option --date, and returns a list of all dates in the week starting from that date. An example for execution would be:
./daterange.py --date 20220226
Giving a list of dates starting from 26-FEB-2022:
2022-02-26 00:00:00
2022-02-27 00:00:00
2022-02-28 00:00:00
2022-03-01 00:00:00
2022-03-02 00:00:00
2022-03-03 00:00:00
2022-03-04 00:00:00
In the Dockerfile below we give the instruction to copy daterange.py to /opt inside the container:
FROM python:3.9.4-buster
RUN pip install pandas
COPY daterange.py /opt
Note
In order to use COPY, the file that needs to be copied needs to be in the same directory as the Dockerfile or one of its subdirectories.
Exercise: Download the daterange.py and build the image with docker build. After that, execute the script inside the container.
Hint
Make an interactive session with the options -i and -t and use /bin/bash as the command.
Answer
Build the container:
docker build -t own_script .
docker build --platform amd64 -t own_script .
Run the container:
docker run -it --rm own_script /bin/bash
Inside the container we look up the script:
cd /opt
ls
This should return daterange.py.
Now you can execute it from inside the container:
./daterange.py --date 20220226
That’s kind of nice. We can ship our python script inside our container. However, we don’t want to run it interactively every time. So let’s make some changes to make it easy to run it as an executable. For example, we can add /opt to the global $PATH variable with ENV.
The $PATH variable
The path variable is a special variable that consists of a list of path seperated by colons (:). These paths are searched if you are trying to run an executable. More info this topic at e.g. wikipedia.
FROM python:3.9.4-buster
RUN pip install pandas
COPY daterange.py /opt
ENV PATH=/opt:$PATH
Note
The ENV instruction can be used to set any variable.
Exercise: Start an interactive bash session inside the new container. Is the path variable updated? (i.e. can we execute daterange.py from anywhere?)
Answer
After re-building we start an interactive session:
docker run -it --rm own_script /bin/bash
The path is upated, /opt is appended to the beginning of the variable:
echo $PATH
returns:
/opt:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Now you can try to execute it from the root directory (or any other):
daterange.py --date 20220226
Instead of starting an interactive session with /bin/bash we can now more easily run the script non-interactively:
docker run --rm own_script daterange.py --date 20220226
Now it will directly print the output of daterange.py to stdout.
In the case you want to pack your script inside a container, you are building a container specifically for your script, meaning you almost want the container to behave as the program itself. In order to do that, you can use ENTRYPOINT. ENTRYPOINT is similar to CMD, but has two important differences:
ENTRYPOINTcan not be overwritten by the positional arguments (i.e.docker run image [CMD]), but has to be overwritten by--entrypoint.- The positional arguments (or
CMD) are pasted to theENTRYPOINTcommand. This means that you can useENTRYPOINTas the executable and the positional arguments (orCMD) as the options.
Let’s try it out:
FROM python:3.9.4-buster
RUN pip install pandas
COPY daterange.py /opt
ENV PATH=/opt:$PATH
# note that if you want to be able to combine the two
# both ENTRYPOINT and CMD need to written in the exec form
ENTRYPOINT ["daterange.py"]
# default option (if positional arguments are not specified)
CMD ["--date", "20220226"]
Exercise: Re-build, and run the container non-interactively without any positional arguments. After that, try to pass a different date to --date. How do the commands look?
Answer
Just running the container non-interactively would be:
docker run --rm own_script
Passing a different argument (i.e. overwriting CMD) would be:
docker run --rm own_script --date 20210330
Here, the container behaves as the executable itself to which you can pass arguments.
Most containerized applications need multiple build steps. Often, you want to perform these steps and executions in a specific directory. Therefore, it can be in convenient to specify a working directory. You can do that with WORKDIR. This instruction will set the default directory for all other instructions (like RUN, COPY etc.). It will also change the directory in which you will land if you run the container interactively.
FROM python:3.9.4-buster
RUN pip install pandas
WORKDIR /opt
# we don't have to specify /opt as target dir but the current dir
COPY daterange.py .
ENV PATH=/opt:$PATH
# note that if you want to be able to combine the two
# both ENTRYPOINT and CMD need to written in the exec form
ENTRYPOINT ["daterange.py"]
# default option (if positional arguments are not specified)
CMD ["--date", "20220226"]
Exercise: build the image, and start the container interactively. Has the default directory changed?
Note
You can overwrite ENTRYPOINT with --entrypoint as an argument to docker run.
Answer
Running the container interactively would be:
docker run -it --rm --entrypoint /bin/bash own_script
Which should result in a termina looking something like this:
root@9a27da455fb1:/opt#
Meaning that indeed the default directory has changed to /opt
Get information on your image with docker inspect
We have used docker inspect already in the previous chapter to find the default Cmd of the ubuntu image. However we can get more info on the image: e.g. the entrypoint, environmental variables, cmd, workingdir etc., you can use the Config record from the output of docker inspect. For our image this looks like:
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/opt:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
"PYTHON_VERSION=3.9.4",
"PYTHON_PIP_VERSION=21.1.1",
"PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1954f15b3f102ace496a34a013ea76b061535bd2/public/get-pip.py",
"PYTHON_GET_PIP_SHA256=f499d76e0149a673fb8246d88e116db589afbd291739bd84f2cd9a7bca7b6993"
],
"Cmd": [
"--date",
"20220226"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/opt",
"Entrypoint": [
"daterange.py"
],
"OnBuild": null,
"Labels": null
},
Adding metadata to your image
You can annotate your Dockerfile and the image by using the instruction LABEL. You can give it any key and value with <key>=<value>. However, it is recommended to use the Open Container Initiative (OCI) keys.
Exercise: Annotate our Dockerfile with the OCI keys on the creation date, author and description. After that, check whether this has been passed to the actual image with docker inspect.
Note
You can type LABEL for each key-value pair, but you can also have it on one line by seperating the key-value pairs by a space, e.g.:
LABEL keyx="valuex" keyy="valuey"
Answer
The Dockerfile would look like:
FROM python:3.9.4-buster
LABEL org.opencontainers.image.created="2022-04-12" \
org.opencontainers.image.authors="Geert van Geest" \
org.opencontainers.image.description="Great container for getting all dates in a week! \
You will never use a calender again"
RUN pip install pandas
WORKDIR /opt
COPY daterange.py .
ENV PATH=/opt:$PATH
# note that if you want to be able to combine the two
# both ENTRYPOINT and CMD need to written in the exec form
ENTRYPOINT ["daterange.py"]
# default option (if positional arguments are not specified)
CMD ["--date", "20220226"]
The Config record in the output of docker inspect was updated with:
"Labels": {
"org.opencontainers.image.authors": "Geert van Geest",
"org.opencontainers.image.created": "2022-04-12",
"org.opencontainers.image.description": "Great container for getting all dates in a week! You will never use a calender again"
}
Building an image with a browser interface
In this exercise, we will use the same base image (python:3.9.4-buster), but instead of installing pandas, we will install jupyterlab. JupyterLab is a nice browser interface that you can use for a.o. programming in python. With the image we are creating we will be able to run jupyter lab inside a container. Check out the Dockerfile:
FROM python:3.9.4-buster
RUN pip install jupyterlab
CMD jupyter lab --ip=0.0.0.0 --port=8888 --allow-root
This will create an image from the existing python image. It will also install jupyterlab with pip. As a default command it starts a jupyter notebook at port 8888.
Ports
We have specified here that jupyter lab should use port 8888. However, this inside the container. We can not connect to it yet with our browser.
Exercise: Build an image based on this Dockerfile and give it a meaningful name.
Answer
docker build -t jupyter-lab .
docker build --platform amd64 -t jupyter-lab .
You can now run a container from the image. However, you will have to tell docker where to publish port 8888 from the docker container with -p [HOSTPORT:CONTAINERPORT]. We choose to publish it to the same port number:
docker run --rm -it -p 8888:8888 jupyter-lab
Networking
More info on docker container networking here
By running the above command, a container will be started exposing jupyterhub at port 8888 at localhost. You can approach the instance of jupyterhub by typing localhost:8888 in your browser. You will be asked for a token. You can find this token in the terminal from which you have started the container.
We can make this even more interesting by mounting a local directory to the container running the jupyter-lab image:
docker run \
-it \
--rm \
-p 8888:8888 \
--mount type=bind,source=/Users/myusername/working_dir,target=/working_dir/ \
jupyter-lab
By doing this you have a completely isolated and shareable python environment running jupyter lab, but with your local files available to it. Pretty neat right?
Note
Jupyter has a wide range of pre-built images available here. Example syntax with a pre-built jupyter image would look like:
docker run \
--rm \
-e JUPYTER_ENABLE_LAB=yes \
-p 8888:8888 \
jupyter/base-notebook
Using the above will also give you easier control over security, users and permissions.