The pip-accel program is a wrapper for pip, the Python package manager. It accelerates the usage of pip to initialize Python virtual environments given one or more requirements files. It does so by combining the following two approaches:
The pip-accel command supports all subcommands and options supported by pip, however it is of course only useful for the pip install subcommand. So for example:
pip-accel install -r requirements.txt
If you pass a -v option then pip and pip-accel will both use verbose output. To enable verbose output from pip-accel only, you can set the environment variable PIP_ACCEL_VERBOSE.
Based on the user running pip-accel the following file locations are used by default:
Root user | All other users | Purpose |
---|---|---|
/root/.pip/download-cache | ~/.pip/download-cache | Assumed to be pip’s download cache |
/var/cache/pip-accel | ~/.pip-accel | Used to store the source/binary indexes |
These defaults can be overridden by defining the environment variables PIP_DOWNLOAD_CACHE and/or PIP_ACCEL_CACHE.
To give you an idea of how effective pip-accel is, below are the results of a test to build a virtual environment for one of the internal codebases of Paylogic. This code base requires more than 40 dependencies including several packages that need compilation with SWIG and a C compiler:
Program | Description | Duration | Percentage |
---|---|---|---|
pip | Default configuration | 444 seconds | 100% (baseline) |
pip | With download cache (first run) | 416 seconds | 94% |
pip | With download cache (second run) | 318 seconds | 72% |
pip-accel | First run | 397 seconds | 89% |
pip-accel | Second run | 30 seconds | 7% |
The way pip-accel works is not very intuitive but it is very effective. Below is an overview of the control flow. Once you take a look at the code you’ll notice that the steps below are all embedded in a loop that retries several times. This is mostly because of step 2 (downloading the source distributions).
- If the command succeeds it means all dependencies are already available as downloaded source distributions. We’ll parse the verbose pip output of step 1 to find the direct and transitive dependencies (names and versions) defined in requirements.txt and use them as input for step 3. Go to step 3.
- If the command fails it probably means not all dependencies are available as local source distributions yet so we should download them. Go to step 2.
- If the command fails it means that pip encountered errors while scanning PyPI, scanning a distribution website, downloading a source distribution or unpacking a source distribution. Usually these kinds of errors are intermittent so retrying a few times is worth a shot. Go to step 2.
- If the command succeeds it means all dependencies are now available as local source distributions; we don’t need the network anymore! Go to step 1.
If you have questions, bug reports, suggestions, etc. please create an issue on the GitHub project page. The latest version of pip-accel will always be available on GitHub. The internal API documentation is hosted on Read The Docs.
This software is licensed under the MIT license just like pip (on which pip-accel is based).
© 2013 Peter Odding and Paylogic International.