Intro

Python and Science: the great mystery

Python isn't known for its baffling performance

Scientific computing <3 performance (think Fortran)

Yet they love each other, which paves the way for accelerating solutions

The Pythran compiler

Serge « sans paille » Guelton

Compiler Engineer / Wood Craft Lover

Groupe Calcul — 24th September 2021

In a nutshell

A non-intrusive compiler for scientific kernels written in Python

import numpy

#pythran export log_likelihood(float64[], float64,
#                              float64)
def log_likelihood(data, mean, sigma):
  s = (data - mean) ** 2 / (2 * (sigma ** 2))
  pdfs = numpy.exp(- s)
  pdfs /= numpy.sqrt(2 * numpy.pi) * sigma
  return numpy.log(pdfs).sum()
# native parallel vectorized module
pythran log_likelihood.py -fopenmp \
        -march=native -DUSE_XSIMD

What about... Cython?

Upside:

What about... PyPy?

Upside:

What about... Numba?

Upside:

4 compilers, 4 stories

Enough speak about others,

Let's dive into Pythran specificities

Limitations

Trading for good

Installation step

$ pip install pythran
$ conda install -c conda-forge pythran
$ dnf install pythran
...
$ git clone https://github.com/serge-sans-paille/pythran.git

Pythran export

Only comments, a tiny language to describe signatures and overloads:

#pythran export kernel(int)
#pythran export kernel(float)
#pythran export kernel(complex128?)
#pythran export kernel(int[] or float[])
#pythran export kernel(int[], int[:,:])

Pythran export - 2

Some types are copied:

#pythran export kernel(int set or int list or int:str dict)

Shape fine-tuning:

#pythran export kernel(int8[:, 3])

Strided and transposed views

#pythran export kernel(int8[::,:])
#pythran export kernel(int8[:,:].T)

Static Polymorphism

def foo(x, y):
    assert type(y) is int
    if isinstance(x, int):
        return np.ones(10)
    else:
        return x * y

foo(5, 8)
foo(np.arange(10), 3)

Option Types

def foo(x, y):
    if x is None:
        x = 1
    return x

foo(None)
foo(3)
foo(4.)

Pythran and OpenMP

Mostly OpenMP3, bits of OpenMP4

# inspired by https://software.intel.com/en-us/node/695675
def min_abs(omp_in, omp_out):
  return min(abs(omp_in), omp_out)

import numpy as np
def find_min_abs(data):
  'return the smallest magnitude among all the integers in data[N]'
  result = abs(data[0])
  #omp declare reduction(minabs : int :  \
     omp_out = min_abs(omp_in, omp_out)) \
     initializer(omp_priv=omp_orig)

  #omp parallel for reduction(minabs: result)
  for d in data:
    if abs(d) < result:
      result = abs(d)
  return result

Supported Modules/Packages

Almost always partial support!

On the shoulder of a…

Pythran dependencies:

From a Notebook

Requires an extension, provides a new cell magic:

> %load_ext pythran.magic
> %%pythran
#pythran export foo(int)
def foo(n): return n
> print(foo(3))

From a shell

$ pythran kernel.py

Sensible to common C++ flags: -O2, -ffast-math etc

From distutils

from distutils.core import setup
from pythran.dist import PythranExtension, PythranBuildExt

module1 = PythranExtension('demo', sources = ['a.py'])

setup(name = 'demo',
      version = '1.0',
      description = 'This is a demo package',
      cmdclass={"build_ext": PythranBuildExt},
      ext_modules = [module1])

O_RDWR

https://pythran.readthedocs.io

#pythran on iirc.oftc.net

https://www.freelists.org/archive/pythran/

https://github.com/serge-sans-paille/pythran/issues

Some Pythran Successes

Random thoughts

1