/ python

Python Package Management, the old way

tl;dr

This is part 1 of a two parter about python package management.

In this article we will focus on the current workflow. And in the next article (coming soon), we will look at one new tool that might shake things up!

Setting the scene

All things considered, Python is a pretty old language. According to wikipedia python is 26 years old! Compare that to Java, 22 years old. Or Rust 7 years old. As such some of the tooling can feel a little clunky.

One example of this is packaging. Here we will look at packaging, not from the pov of a developer looking to package and publish an application. But rather, from the pov of a developer wanting to manage dependencies for one or several applications on a development machine.

Current Flow

pip

pip is the current preferred installer for Python. More often than not it will come default with your Python install. With it you can install, upgrade and remove packages. You can even keep track of your applications dependencies in a specially formatted file (usually named requirements.txt) and by passing -r to pip get all the correct versions of your dependencies installed.

However, pip has a couple limitations:

  • pip installs packages globally (contrary to other package managers such as npm).
  • pip does not manage recursive dependencies, it just installs whatever your packages need at the top level. This means you as the developer will have to manage what your top level dependencies are and how to handle pinning and the such.

One could argue that this is by design. That pip does one thing and one thing well. But that does mean that we need more than just pip.

virtualenv

Because you most probably will be working on multiple Python applications each with there own set of dependencies, you will probably want a way to deal with each set of dependencies in isolation. This is where virtualenv comes in.

virtualenv is a tool to create isolated Python environments. It essentially let's you have multiple environments each with it's own packages installed.

It does come at the cost of a little dance (and some interesting conversations with people new to Python).

$ python -m venv my_venv  # Create venv in my_venv directory
$ source my_venv/bin/activate  # Activate venv
(my_venv) $ pip install package  # Install package within venv
(my_venv) $ deactivate  # Leave venv

Tools such as virtualenvwrapper make this a little easier. It manages all your virtualenvs in a central location, you just need to remember to switch to the correct venv. As such:

$ mkvirtualenv example
$ workon example
(example) $ ...  # Work on project
(example) $ deactivate  # Leave venv

Interestingly, virtualenvwrapper has proven complex enough to install that tools exist just to simplify the installation process. Shout-out to @brainsik and the very useful virtualenv-burrito.

requirements.txt

Before we take a look at pipenv and how it simplifies this whole process. Let's talk quickly about requirement files.
At it's simplest, a requirements file is just a list of package names separated by a new line.
Here is an extremely simple requirements file, with one package and no version requirements.

$ cat requirements.txt 
requests

If I pointed pointed pip at this file, it would install the latest version of requests and it's dependencies.
You probably don't want to do that. if ever some breaking change was introduced in a later version of requests, anybody trying to install your app would not be able to get it to work.
What you'll want to do is specify a version, or range of versions for you dependencies. And this is where pip and requirement files fall flat. Say we have a project that depends on just requests and I pin the version to 2.18.4.

$ cat requirements.txt
requests==2.18.4

Now, let's install things.

$ pip install -r requirements.txt
[pip install output]
$ pip freeze  # Show installed packages
certifi==2017.11.5
chardet==3.0.4
idna==2.6
pkg-resources==0.0.0
requests==2.18.4
urllib3==1.22

Cool, so requests has couple of dependencies. Should I pin those too? Probably, if you don't you're just hoping all those package maintainers are properly pinning their dependencies. So how do you do that? Well, for small projects people will just pip freeze > requirements.txt. The problem now is, what did you install? How do you make sure you're keeping everything up to date? It's possible, but it's a pain. And more importantly a task much more suited for a computer.

pipenv

Join us soon for our presentation of pipenv

Mathieu Sabourin

Mathieu Sabourin

I codez. I do the pythonz at WayUp, previously of NerdWallet and Sauce Labs. Contrary to what banner would lead to think, not into bitcoin or crypto(currencies).

Read More