Poetry isn't working in the Dockerfile

I came back from some days out of the office and found a coworker implementing some odd workarounds to reinstall python dependencies for an app in its container on every execution. I decided to dig in and see what’s going on and learned a little bit about s2i-python-container and Python Poetry.

The Symptoms

So it looked like an update was made to bump the Poetry installer used in a Dockerfile and this resulted in the container no longer working. The python app that was containerised in it would crash with missing dependencies despite the container build operation logs showing all the dependencies being installed into the container rootfs.

First Debugging Step: Where Did The Python Modules End Up

First thing I did, was try to figure out if the Python modules actually were installed (to determine if this is a buildtime issue or a runtime issue). I dropped into a shell in the container and ran a quick find over the entire rootfs to find a module I expected to be installed by Poetry.

(app-root) sh-4.4# find / -name "sh.py"
find: ‘/proc/tty/driver’: Permission denied
/usr/lib/python3.9/site-packages/sh.py

Okay! The module is there. Can Python see it?

(app-root) sh-4.4# python3
Python 3.9.7 (default, Sep 13 2021, 08:18:39)
[snip]...
>>> import sh
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sh'

Hmmmm. Weird.

Second Debugging Step: Where Should They Have Gone

I next re-ran Poetry inside the container to re-install the dependencies and see what happens.

(app-root) sh-4.4# poetry install
Installing dependencies from lock file
[snip]...

(app-root) sh-4.4# python3
Python 3.9.7 (default, Sep 13 2021, 08:18:39)
[snip]...
>>> import sh
>>>

Okay! Now the import works! So we can see that running Poetry in the container changes its behaviour compared to running it in the Dockerfile. Next is to try to understand why.

The Container: VirtualEnv and Poetry

The container base image was something similar to centos/python-38-centos7, which I learned is built using s2i-python-container [1]. This container always had some quirky behaviour I didn’t really understand which is that when you drop into the shell it would default to /opt/app-root and the python installation would be found there rather than in some default system paths.

In our Dockerfile I found the following line to disable the virtualenvs creation by Poetry when using it to install the dependencies. It’s fairly common to see this line in a Dockerfile as its normal to want to install the dependencies into the system-wide packages for a container because there’s no need to add a custom virtualenv wrapper within the container.

poetry config virtualenvs.create false

After digging into s2i-python-container, I found out that actually when you drop into the container it automatically activates a virtualenv in /opt/app-root and the Python installation can be found there. That combined with how Poetry has some defaults to create its own virtualenv gave me a starting point.

By disabling the default Poetry behaviour around virtualenvs, it seems it was defaulting to the system-wide packages and not picking up on the virtualenv used in the container (which it does obey if run during runtime).

This seems to be the root issue. The next step is to understand how to make Poetry obey the /opt/app-root virtualenv paths when run in the container build.

[1] https://github.com/sclorg/s2i-python-container

The Fix

The fix ended up being very simple. We need Poetry to understand it needs to act in accordance to the virtualenv that will be executed when dropping into the container at runtime. Poetry can be informed about this virtualenv by setting an environment variable: VIRTUAL_ENV [2].

[2] https://docs.python.org/3/library/venv.html#creating-virtual-environments

Typically, when a virtual environment is active, this environment variable value is set to the path of the virtual environment. When this is set, Poetry will obey it and install the modules accordingly. So a quick one-liner addition to the Dockerfile before calling poetry install fixes it all up:

ENV VIRTUAL_ENV=/opt/app-root

Now all the modules are installed to the right place for the container python. For me, it was great to dig into our container base images and find out about s2i-python-container and also learn a bit more about how Poetry will act in different situations. I didn’t see any easy-to-google posts out there about this so I hope it helps save someone the hour I had to spend!