PyFAI is a library to deal with diffraction images for data reduction. This chapter describes the project from the computer engineering point of view.
PyFAI is an open source project licensed under the GPL (switching to MIT) mainly written in Python (v2.7 or newer, 3.4 or newer) and heavily relying on the Python scientific ecosystem: numpy, scipy and matplotlib. It provides high performances image treatment thanks to Cython and OpenCL... but only a C-compiler is needed to build it.
PyFAI is a Python project but uses many programming languages:
The OpenCL code has been tested using:
The project is hosted on GitHub: https://github.com/silx-kit/pyFAI
Which provides the issue tracker in addition to Git hosting. Collaboration is done via Pull-Requests in github’s web interface:
Everybody is welcome to fork the project and adapt it to his own needs: CEA-Saclay, Synchrotrons Soleil, Desy and APS have already done so. Collaboration is encouraged and new developments can be submitted and merged into the main branch via pull-requests.
A mailing list: pyfai@esrf.fr is publicly available, it is the best place to ask your questions: the author and many advanced users are there and willing to help you. To subscribe to this mailing list, send an email to sympa@esrf.fr with "subscribe pyfai" as subject.
On this mailing list, you will have information about release of the software, new features available and meet experts to help you solve issues related to azimuthal integration and diffraction in general. It also provides a knowledge-base of most frequently asked question which needs to be integrated into this documentation.
The volume of email on the list remains low, so you can subscribe without being too much spammed. As the mailing list is archived, and can be consulted at: http://www.edna-site.org/lurker, you can also check the volume of the list.
If you think you are facing a bug, the best is to create a new issue on the GitHub page (you will need a GitHub account for that).
Direct contact with authors is discouraged: pyFAI is open source software that we develop to aid the research community in doing what they do best. While we do enjoy doing this, we would not be able to dream of spending nearly as much time with pyFAI as we do if it wasn’t for your support. Interest of the scientific community (via a lively mailing list) and citation in scientific publication for our software is one of the main criterion for ESRF management when deciding if they should continue funding development.
In addition to the run dependencies, pyFAI needs a C compiler.
There is an issue with MacOS (v10.8 onwards) where the default compiler (Xcode5 or newer) dropped the support for OpenMP. On this platform pyFAI will enforce the generation of C-files from Cython sources (making Cython a build-dependency on MacOS) without support of OpenMP (options: –no-openmp –force-cython). On OSX, an alternative is to install a recent version of GCC (>=4.2) and to use it for compiling pyFAI. The options to be used then are * –force-cython –openmp*.
Otherwise, C files are which are provided with pyFAI sources are directly useable and Cython is only needed for developing new binary modules. If you want to generate your own C files, make sure your local Cython version is recent enough (v0.21 and newer), unless your Cython files will not be translated to C, nor used.
python setup.py build bdist_wheel
pip install dist/pyFAI-0.13.0*.whl --upgrade
There are few specific options to setup.py:
To test the installed version of pyFAI:
python run_test.py
or from python:
import pyFAI
pyFAI.tests()
Some Warning messages are normal as the test procedure also tests corner cases.
Setting the environment variable http_proxy can be necessary (depending on your network):
export http_proxy=http://proxy.site.org:3128
Especially at ESRF, the configuration of the network proxy can be obtained by asking at the helpdesk: helpdesk@esrf.fr
To test the development version (built but not yet installed):
python setup.py build test
or
python setup.py build
python run_test.py -i
PyFAI comes with 40 test-suites (338 tests in total) representing a coverage of 60%. This ensures both non regression over time and ease the distribution under different platforms: pyFAI runs under Linux, MacOSX and Windows (in each case in 32 and 64 bits). Test may not pass on computer featuring less than 2GB of memory or 32 bit architectures.
Note:: The test coverage tool does not count lines of Cython, nor those of OpenCL. Anyway test coverage is a tool for developers about possible weaknesses of their code and not a management indicator.
This software engineering practice consists in merging all developer working copies to a shared mainline several times a day and build the whole project for multiple targets.
Continuous integration is made by a home-made scripts which checks out the latest release and builds and runs the test every night. Nightly builds are available for debian8-64 bits. To install them:
sudo apt-get update
sudo apt-get install pyfai
You have to accept non-signed packages because they are automatically built.
In addition some “cloud-based” tools are used to ensure a larger coverage of operating systems/environment. They rely on a “local wheelhouse”.
Those wheels are optimized for Travis-CI, AppVeyor and ReadTheDocs, using them is not recommended as your Python configuration may differ (and those libraries could even crash your system).
Travis provides continuous integration on Linux, 64 bits computer with Python 2.7, 3.4 and 3.5.
The builds cannot yet be retrieved with Travis-CI, but manylinux-wheels are on the radar.
AppVeyor provides continuous integration on Windows, 64 bits computer with Python 2.7 and 3.4. Successful builds provide installers for pyFAI as wheels and msi, they are anonymously available as artifacts. Due to the limitation of AppVeyor’s build system, those installers have openMP disabled.
$ git log --pretty='%aN##%s' | grep -v 'Merge pull' | grep -Po '^[^#]+' | sort | uniq -c | sort -rn