Writing the extension modules and Python wrappers for a package is one thing, but a step that is often overlooked is making a build system that complies with the rest of your program, ensures the correct installation based on your dependencies and also is portable enough to be distributable.
I learned these things the hard way in Week 3 and 4, where I went as low level as I could to try to solve all the weird build errors and glitches I had while trying to build a Python Package using GNU Autotools.
As discussed in my last GSoC blog, I was mainly using distutils
along with it’s distutils.setup
script to take care of all the building and linking required for building the .so
(shared object) file required by the Python Interpreter. However, one of my co-mentors brought up a good point that setuptools
is the packaging tool that is recommended by PyPA and also using wheels
to package the modules instead of the standard setup.py build
command.
Hence, Week 3 was spent mainly learning about setuptools
and wheels
. What Are Python Wheels and Why Should You Care? is a great article to start with Python Wheels. The setuptools documentation is a great place to know about setuptools, if you already know about distutils like me! Luckily, while Setuptools
is a “beefier” version of distutils, as it offers better and more packaging utilities, it keeps the same functions, so in terms of code it was just a change of one line for me.
Originally, with distutils
, the plan was to have the files related to the Python Package in a separate python/
directory at the root of the Gnuastro source like:
📦python
┣ 📂gnuastro.arithmetic
┃ ┣ 📜arithmetic.c
┃ ┗ 🔧setup.py
┣ 📂gnuastro.cosmology
┃ ┣ 📜cosmology.c
┃ ┗ 🔧setup.py
┣ 📂gnuastro.fits
┃ ┣ 📜fits.c
┃ ┗ 🔧setup.py
┗ 📑Makefile.am
The idea was to have the setup.py
script in each folder build that specific extension, and let the Makefile handle the linking. But I soon realized that this was too excessive. A better structure would be:
📦python
┣ 📂src
┃ ┣ 📜arithmetic.c
┃ ┣ 📜cosmology.c
┃ ┗ 📜fits.c
┣ 📑Makefile.am
┗ 🔧setup.py
As the name suggests GNUastro is a GNU project, and thus depends on Autotools(Automake and Autoconf and Libtool) for its building and compiling. These are the tools behind the
./configure
make
make check
make install
set of instructions.
Alongwith the setup script, I also added a new file(python.c
) to the lib/ directory of Gnuastro. This file basically provides any utility functions I might require while building the Python package. Currently, the file provides type conversion functions, which facilitate converting between Gnuastro and NumPy’s datatypes.
So, what is the difference between your traditional Makefile and using Autotools instead:-
Configure generates a config.h
file (from a template) which programs can include to work around portability issues. For example, if HAVE_NUMPY is not defined, don’t build the Python package.
My job was to use these tools to also call the setup
script for building my Python package.
My approach to building the package using Autotools involved 4 basic steps:
configure.ac
script.
Python.h
file.Makefile.am's
.lib/python.c
) only if the above checks are passed.Makefile.am
in the python/
directory which would handle the build, install, uninstall and clean targets for the Python package.VPATH builds
are basically a way to separate your source and build tree, so that all the built files (.o, .so, etc) are in a separate directory than your source files but are symlinked to the source tree.This process took a lot of trial and error, digging into the Autotools(mostly Automake) documentation and playing around with the Makefile.am
to get right. But it introduced me to these amazing tools and taught me how to make any scrawny personal project distributable!
After running,
python3 setup.py build_ext bdist_wheel
the distributable wheel file, with all of the package’s metadata, is created under the dist/
folder. In order to install this file we use pip as follows:
pip install Gnuastro.whl
YES! It is in fact as simple as that!
But there is an issue that I faced here, suppose that a user wants to install the Gnuastro library in their root directory, or to any directory where they dont have privileges. This means they’ll run sudo make install
from the root of the source. This cascades to calling the Makefile in the python/ directory with root access as well. However, running pip
with sudo access is a big NO, NO. And pip
would warn you of that with a warning like:
This is because, Python packages are generally installed at a local level, in the /usr/local
directory. However, if you call pip
with sudo
then it installs the packages in the root directory. To sove this, we use
sudo -u "$SUDO_USER pip install Gnuastro,whl
which basically runs the pip command as the user who called sudo. This will ensure that your package gets installed in the local directory instead of root!