This is part 1 of a two parter about python package management.
In this article we will focus on the current workflow. And in the next article (coming soon), we will look at one new tool that might shake things up!
Setting the scene
All things considered, Python is a pretty old language. According to wikipedia python is 26 years old! Compare that to Java, 22 years old. Or Rust 7 years old. As such some of the tooling can feel a little clunky.
One example of this is packaging. Here we will look at packaging, not from the pov of a developer looking to package and publish an application. But rather, from the pov of a developer wanting to manage dependencies for one or several applications on a development machine.
pip is the current preferred installer for Python. More often than not it will come default with your Python install. With it you can install, upgrade and remove packages. You can even keep track of your applications dependencies in a specially formatted file (usually named
requirements.txt) and by passing
-r to pip get all the correct versions of your dependencies installed.
pip has a couple limitations:
pipinstalls packages globally (contrary to other package managers such as
pipdoes not manage recursive dependencies, it just installs whatever your packages need at the top level. This means you as the developer will have to manage what your top level dependencies are and how to handle pinning and the such.
One could argue that this is by design. That
pip does one thing and one thing well. But that does mean that we need more than just
Because you most probably will be working on multiple Python applications each with there own set of dependencies, you will probably want a way to deal with each set of dependencies in isolation. This is where
virtualenv comes in.
virtualenv is a tool to create isolated Python environments. It essentially let's you have multiple environments each with it's own packages installed.
It does come at the cost of a little dance (and some interesting conversations with people new to Python).
$ python -m venv my_venv # Create venv in my_venv directory $ source my_venv/bin/activate # Activate venv (my_venv) $ pip install package # Install package within venv (my_venv) $ deactivate # Leave venv
Tools such as
virtualenvwrapper make this a little easier. It manages all your virtualenvs in a central location, you just need to remember to switch to the correct
venv. As such:
$ mkvirtualenv example $ workon example (example) $ ... # Work on project (example) $ deactivate # Leave venv
Before we take a look at
pipenv and how it simplifies this whole process. Let's talk quickly about requirement files.
At it's simplest, a requirements file is just a list of package names separated by a new line.
Here is an extremely simple requirements file, with one package and no version requirements.
$ cat requirements.txt requests
If I pointed pointed
pip at this file, it would install the latest version of
requests and it's dependencies.
You probably don't want to do that. if ever some breaking change was introduced in a later version of
requests, anybody trying to install your app would not be able to get it to work.
What you'll want to do is specify a version, or range of versions for you dependencies. And this is where
pip and requirement files fall flat. Say we have a project that depends on just
requests and I pin the version to
$ cat requirements.txt requests==2.18.4
Now, let's install things.
$ pip install -r requirements.txt [pip install output] $ pip freeze # Show installed packages certifi==2017.11.5 chardet==3.0.4 idna==2.6 pkg-resources==0.0.0 requests==2.18.4 urllib3==1.22
requests has couple of dependencies. Should I pin those too? Probably, if you don't you're just hoping all those package maintainers are properly pinning their dependencies. So how do you do that? Well, for small projects people will just
pip freeze > requirements.txt. The problem now is, what did you install? How do you make sure you're keeping everything up to date? It's possible, but it's a pain. And more importantly a task much more suited for a computer.
Join us soon for our presentation of