Module - Optimizing Python Programs

Guidelines for Optimization

Python programs are not principally slow. On the contrary: the execution speed is sufficient for many application. The basic question is if the program serves its intended purpose. Furthermore, other factors such as database access or network connections may determine the speed of the application.

The course provides important guidelines for optimization.

Optimization Strategies

The concept of benchmarking with pystone is introduced with examples for different Python implementations. Profiling of CPU usage can be used to find the spots of a program that use up most of the program run time. The built-in module cProfile is used in the course. Examples of common cases will help to understand how to use them. Memory usage will be measured with the heapy module from the Guppy_PE framework. Different possibilities of its usage are explored with examples.

Participants are encouraged to bring their own programs for profiling to the course.

Algorithms and Anti-Patterns

Typically the larges gains of performance can be achieved through algorithmic changes.

Algorithms that are especially slow or fast in Python are shown. We look at string concatenation and we use generators to minimize memory usage.

The Right Data Structure

The data structure can have significant impact on execution speed. We look at the built-in data structures compare lists, sets, deque and defaultdict. The big-O notation will be exemplified.

Caching

Caching can be a useful strategy to reduce run times. We look at deterministic and non-deterministic caching and develop decorates for these purposes.

The Example

We use a computationally demanding example and implement it first in pure Python. We look at some algorithmic improvements to speed up the computation.

Testing Speed

Measuring how fast a program really run is no simple task. We look at solutions to this problem in Python.

Psyco

Psyco is a so called "just-in-time-complier" (JIT), which allows to translate parts of the byte code to machine code. This may yield considerable speed improvements.

Examples are used to show the different possibilities of using Psyco. Participants are encouraged to bring their own programs to the course to optimize them.

Numerical Calculations with NumPy

Numerical calculations using large list are rather slow in Python. The dynamical typing can use up lots of performance. The library NumPy offers efficient processing of numerical arrays. Functionality and usage are inspired by MATLAB.

The basic possibilities of NumPy are covered in the course.

Using Multiple CPUs with pyprocessing/multiprocessing

The module pyprocessing is, renamed as multiprocessing part of the standard library of Python 2.6. It allows to take advantage of multiprocessor CPUs. We will have a short tour through the module and apply it to our example.

Combination of Optimization Strategies

We combine several of the strategies to get the maximum speed up.

Overview of Extensions to Python with Other Languages

The speed of Python programs can be increased through extensions written in other languages. An overview of possibilities to extend Python are given if this module is not combined with the module Python-Extensions with Other Languages.