Faster Python Programs - Measure, don't Guess

Python is a great language. But it can be slow compared to other languages for certain types of tasks. If applied appropriately, optimization may reduce program runtime or memory consumption considerably. But this often comes at a price. Optimization can be time consuming and the optimized program may be more complicated. This, in turn, means more maintenance effort. How do you find out if it is worthwhile to optimize your program? Where should you start? This tutorial will help you to answer these questions. You will learn how to find an optimization strategy based on quantitative and objective criteria. You will experience that one’s gut feeling what to optimize is often wrong.

The solution to this problem is: „Measure, Measure, and Measure!“. You will learn how to measure program run times as well as profile CPU and memory. There are great tools available. You will learn how to use some of them. Measuring is not easy because, by definition, as soon as you start to measure, you influence your system. Keeping this impact as small as possible is important. Therefore, we will cover different measuring techniques.

Furthermore, we will look at algorithmic improvements. You will see that the right data structure for the job can make a big difference. Finally, you will learn about different caching techniques.

Software Requirements

You will need Python 2.7 or 3.5 installed on your laptop. Python 2.6 or 3.3/3.4 should also work. Python 3.x is strongly preferred.

IPython Notebook

I will use an IPython Notebook for the tutorial because it makes a very good teaching tool. You are welcome to use the setup you prefer, i.e editor, IDE, REPL. If you also like to use an IPython Notebook, I recommend conda for easy installation. Similarly to virtualenv, conda allows creating isolated environments but allows binary installs for all platforms.

There are two ways to installIPython via conda:

1. Use Minconda. This is a small install and (after you installed it) you can use the command conda to create an environment: conda create -n pycon2016 python=3.5 Now you can change into this environment: source activate pycon2016. The prompt should change to (pycon2016). Now you can install IPython: conda install IPython.

2. Install Anaconda and you are ready to go if you don’t mind installing lots of packages from the scientific field.

The second option also gives you conda and you can create more environments just as with Miniconda (see 1.).

Working witch conda environments

After creating a new environment, the system might still work with some stale settings. Even when the command which tells you that you are using an executable from your environment, this might actually not be the case. If you see strange behavior using a command line tool in your environment, use hash -r and try again. Please install pip inside this environment:

  conda install pip

Tools

You can install these with pip (in the active conda environment):

• SnakeViz
• line_profiler
• Pympler
• memory_profiler
• pyprof2calltree

Linux

Using the package manager of your OS should be the best option.