Python virtual environments, Poetry

Photo by Cassie Boca on Unsplash

An essential aspect of developing applications is managing their dependencies. Often, you find yourself in a situation where a project requires a specific version of a library that is incompatible with other projects you have created. Obviously, you don’t want to decide which one will live or die. You want all your projects to be safe and sound while developing new things without worrying about this situation.

In Python, virtual environments are commonly used for dependency management and isolation between applications. Dependency management gives you a sense of control by allowing you to define your dependencies in a centralized way (a single file) giving you the possibility of destroying and recreating your project environment easily.

This time, we will talk about creating and managing virtual environments using Poetry. To do this, we will create a Python project from scratch and, step by step, explore the basics of how Poetry works. We will also cover the necessary files you need to be aware of and demonstrate how to add, inspect, or remove dependencies from the virtual environment.

Let’s start from the very beginning. What is Poetry?

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution.

Sounds great, isn’t it? Let’s create a new Python project from scratch!

First, obviously, in order to use Poetry, you need to install it. Depending on the operating system (OS) you are using, Poetry has different installation procedures or commands. This time, we will follow the recommended installation process for Linux (for more information visit: https://python-poetry.org/docs/#installation).

curl -sSL https://install.python-poetry.org | python3 -

Now, let’s create a project. Run the following command, and let’s see what Poetry does for you.

poetry new --src example_project

As you can see in the following image, Poetry has created some directories and files for you, even a directory for your tests (no more excuses for tests).

The --src flag is not required. You can execute the command without it and, as a result, get a new project as well, but slightly different.

The “pyproject.toml” file

Now, let’s check pyproject.tomlfile. It allows you to basically control all aspects of your project, let's have a look at it.

The initial version of this file has three sections: tool.poetry, “tool.poetry.dependencies”, and build-system. The first one, “tool.poetry”, contains information about your project/package and, as you can see, it’s the biggest one, but let's talk about it deeper later.

The second section, “tool.poetry.dependencies”, is related to your project dependencies. In a real project, your project dependencies list can get bigger easily and organizing them can be a headache. For this, Poetry has a great feature that lets you organize them using groups that you can name conveniently such as development (for dependencies required only in a development environment) or notebooks (for your Jupyter notebooks dependencies in case you need them). Let’s install two dependencies using different groups to see how its works.

poetry add pydantic

The command above installs the latest compatible version of Pydantic based on your Python version. Pay attention to the group in which Poetry lists this dependency — it is located under the default/main group. Dependencies declared in this group are part of the implicit main group.

Now, let's create a dev group to install our development dependencies.

poetry add pytest --group dev

Take note of the new section in your pyproject.toml file. Poetry has automatically created a ‘tool.poetry.group.dev.dependencies’ section.

This allows you to install project dependencies based on the environment you are working on. For example, if you are working with notebooks for data analysis or a small proof of concept, you can use the following command:

poetry install --with notebooks

Here, ‘notebooks’ is the group name for the dependencies you want to install alongside the main dependencies group. Some of the dependencies in this group will not be included in the main group unless you decide to integrate the new functionality into your “production” code.

Note: “Optional group dependencies will still be resolved alongside other dependencies, so special care should be taken to ensure they are compatible with each other.”

Now, let’s talk a little bit about the tool.poetry section that we mentioned earlier. In this section, you can configure various aspects of your project. Let's focus on the parameters that Poetry configures by default when creating a new project.

The first four parameters are self-explanatory. They are related to the current name, version, short description, and author(s) of the package/application.

The next parameter, readme, specifies the path or list of paths corresponding to the README file(s) of your package/application. In this case, Poetry has created a README.md file in the root of the project and automatically pointed to it for you.

The last parameter, packages, is the list of packages and modules to include in the final distribution. With its current value, you are telling Poetry to include your 'example_project' directory in the final distribution and to start looking for it from the 'src' directory. This perfectly matches your project structure.

The “poetry.lock” file

So far, we have been talking about the pyproject.toml file, but you may have realized that after installing your first project dependency, a new file was automatically created in your root folder: poetry.lock. Every time you install a new dependency, Poetry reads the pyproject.toml file, resolves the dependencies, and if everything is okay, it performs the following actions: registers the new dependency in the corresponding group inside the pyproject.toml file and records information about the exact package version installed in the poetry.lock file.

The information registered inside the poetry.lock helps Poetry ensure repeatable installs, regardless of where you need to recreate the virtual environment. This is a helpful feature in various scenarios. For example, suppose you have been assigned to work on an existing project. In this case, you only need to clone the project from the repository to your local machine, set up the correct Python version, and execute the following command to create the virtual environment and install all the dependencies:

poetry install

Note: “If there is a poetry.lock file in the current directory, it will use the exact versions from there instead of resolving them. This ensures that everyone using the library will get the same versions of the dependencies.

If there is no poetry.lock file, Poetry will create one after dependency resolution.”

Working with the virtual environment

Now that you have installed all the dependencies, you should be able to work with your virtual environment. To do this, run the following command:

poetry shell

This command activates the Python interpreter within your virtual environment, which is aware of the installed packages. You can easily check this by executing:

poetry env info

As you can see, the Python executable used by the virtual environment is different from the Python interpreter in my system. If you try to use the pydantic package with out activating the virtual environment, you will get a ModuleNotFoundError: No module named pydantic.

Another simple but useful feature that Poetry provides is the ability to inspect installed packages. The following command will print all installed packages, or you can specify a package to display information about it and its dependencies.

poetry show
poetry show pydantic  # for information about an specifc package.

Removing dependencies

So far, we have created a virtual environment and install some dependencies on it but, what if I want to remove them? In order to achieve this just run:

poetry remove pydantic

What’s next?

In this article, we have learned how to use Poetry to create a Python project from scratch, along with its virtual environment. However, Poetry can do so much more for you. For example, I use “pyenv” as a tool to manage different Python versions in conjunction with Poetry, enabling me to work with projects that require a Python version different from the one installed on the system. Additionally, the pyproject.toml is a very powerful configuration file. It takes advantage of the Project configuration via a TOML file, which is a reasonably new configuration file format that the Python community has embraced over the last couple of years. TOML plays an essential part in the Python ecosystem ( https://realpython.com/python-toml/). This file allows me to integrate and configure many tools that I love, such as ruff, black, isort, pylint, among others, helping me improve my code quality.

References

12. Virtual Environments and Packages - Stackless-Python 3.7.9 documentation

Virtualenv - virtualenv 20.17.2.dev2+g6845f6f documentation

Introduction

Python and TOML: New Best Friends - Real Python