4. Compile from Source#

4.1. When to Compile imate#

Generally, it is not required to compile imate as the installation through pip and conda contains most of its features, including support for GPU devices. You may compile imate if you want to:

  • modify imate.

  • use OpenBLAS instead of the built-in matrix library of imate.

  • build imate for a specific version of CUDA Toolkit.

  • disable the dynamic loading feature of imate for CUDA libraries.

  • enable debugging mode.

  • or, build this documentation.

Otherwise, install imate through the Python Wheels.

This section walks you through the compilation process.

4.2. Install C++ Compiler (Required)#

Compile imate with either of GCC, Clang/LLVM, or Intel C++ compiler on UNIX operating systems. For Windows, compile imate with Microsoft Visual Studio (MSVC) Compiler for C++.

Install GNU GCC Compiler

sudo apt install build-essential

sudo yum group install "Development Tools"

sudo dnf group install "Development Tools"

sudo brew install gcc libomp

Then, export C and CXX variables by

export CC=/usr/local/bin/gcc
export CXX=/usr/local/bin/g++

Install Clang/LLVN Compiler

sudo apt install clang

sudo yum install yum-utils
sudo yum-config-manager --enable extras
sudo yum makecache
sudo yum install clang

sudo dnf install yum-utils
sudo dnf config-manager --enable extras
sudo dnf makecache
sudo dnf install clang

sudo brew install llvm libomp-dev

Then, export C and CXX variables by

export CC=/usr/local/bin/clang
export CXX=/usr/local/bin/clang++

Install Intel oneAPI Compiler

To install Intel Compiler see Intel oneAPI Base Toolkit.

4.3. Install OpenMP (Required)#

OpenMP comes with the C++ compiler installed. However, you may alternatively install it directly on UNIX. Install OpenMP library on UNIX as follows:

sudo apt install libgomp1 -y

sudo yum install libgomp -y

sudo dnf install libgomp -y

sudo brew install libomp

Note

In macOS, for libomp versions 15 and above, Homebrew installs OpenMP as keg-only. To utilize the OpenMP installation, you should establish the following symbolic links:

libomp_dir=$(brew --prefix libomp)
ln -sf ${libomp_dir}/include/omp-tools.h  /usr/local/include/omp-tools.h
ln -sf ${libomp_dir}/include/omp.h        /usr/local/include/omp.h
ln -sf ${libomp_dir}/include/ompt.h       /usr/local/include/ompt.h
ln -sf ${libomp_dir}/lib/libomp.a         /usr/local/lib/libomp.a
ln -sf ${libomp_dir}/lib/libomp.dylib     /usr/local/lib/libomp.dylib

4.4. OpenBLAS (Optional)#

imate can be compiled with and without OpenBLAS. If you are compiling imate with OpenBLAS, install OpenBLAS library by

sudo apt install libopenblas-dev

sudo yum install openblas-devel

sudo dnf install openblas-devel

sudo brew install openblas

Alternatively, you can install OpenBLAS using conda:

conda install -c anaconda openblas

Note

To build imate with OpenBLAS, you should also set USE_CBLAS environment variable as described in Configure Compile-Time Environment Variables.

4.5. Install CUDA Compiler (Optional)#

To use imate on GPU devices, it should be compiled with CUDA compiler. Skip this part if you are not using GPU.

Note

The minimum version of CUDA to compile imate is CUDA 10.0.

Attention

NVIDIA does not support macOS. You can install the NVIDIA CUDA Toolkit on Linux and Windows only.

To download and install the CUDA Toolkit on both Linux and Windows, refer to the NVIDIA Developer website. It’s important to note that NVIDIA’s installation instructions on their website include the entire CUDA Toolkit, which is typically quite large (over 6 GB in size).

However, for compiling imate, you don’t need to install the entire CUDA Toolkit. Instead, only the CUDA compiler and a few specific development libraries, such as cuBLAS and cuSparse, are required. Below are simplified installation instructions for Linux, allowing you to perform a minimal CUDA installation with only the necessary libraries. Note that in the following, you may change CUDA_VERSION to the CUDA version that you wish to install.

# Set to the desired cuda version
CUDA_VERSION="12-3"

# Machine architecture
ARCH=$(uname -m | grep -q -e 'x86_64' && echo 'x86_64' || echo 'sbsa')

# OS Version
UBUNTU_VERSION=$(awk -F= '/^VERSION_ID/{gsub(/"/, "", $2); print $2}' /etc/os-release)
OS_VERSION=$(dpkg --compare-versions "$UBUNTU_VERSION" "ge" "22.04" && echo "2204" || echo "2004")

# Add CUDA Repository
sudo apt update
sudo apt install wget -y
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${OS_VERSION}/${ARCH}/cuda-keyring_1.1-1_all.deb -P /tmp
sudo dpkg -i /tmp/cuda-keyring_1.1-1_all.deb
rm /tmp/cuda-keyring_1.1-1_all.deb

# Install required CUDA libraries
sudo apt-get update
sudo apt install -y \
    cuda-nvcc-${CUDA_VERSION} \
    libcublas-${CUDA_VERSION} \
    libcublas-dev-${CUDA_VERSION} \
    libcusparse-${CUDA_VERSION} \
    libcusparse-dev-${CUDA_VERSION}

# Set to the desired cuda version
CUDA_VERSION="12-3"

# Machine architecture
ARCH=$(uname -m | grep -q -e 'x86_64' && echo 'x86_64' || echo 'sbsa')

# OS Version
OS_VERSION=$(awk -F= '/^VERSION_ID/{gsub(/"/, "", $2); print $2}' /etc/os-release)

# Add CUDA Repository
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel${OS_VERSION}/${ARCH}/cuda-rhel${OS_VERSION}.repo

# Install required CUDA libraries
sudo yum install --setopt=obsoletes=0 -y \
    cuda-nvcc-${CUDA_VERSION} \
    cuda-cudart-devel-${CUDA_VERSION} \
    libcublas-${CUDA_VERSION} \
    libcublas-devel-${CUDA_VERSION} \
    libcusparse-${CUDA_VERSION} \
    libcusparse-devel-${CUDA_VERSION}

# Set to the desired cuda version
CUDA_VERSION="12-3"

# Machine architecture
ARCH=$(uname -m | grep -q -e 'x86_64' && echo 'x86_64' || echo 'sbsa')

# OS Version
OS_VERSION=$(awk -F= '/^VERSION_ID/{gsub(/"/, "", $2); print $2}' /etc/os-release)

# Add CUDA Repository
sudo dnf install -y dnf-utils
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel${OS_VERSION}/${ARCH}/cuda-rhel${OS_VERSION}.repo

# Install required CUDA libraries
sudo dnf install --setopt=obsoletes=0 -y \
    cuda-nvcc-${CUDA_VERSION} \
    cuda-cudart-devel-${CUDA_VERSION} \
    libcublas-${CUDA_VERSION} \
    libcublas-devel-${CUDA_VERSION} \
    libcusparse-${CUDA_VERSION} \
    libcusparse-devel-${CUDA_VERSION}

Update PATH with the CUDA installation location by

echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' >> ~/.bashrc
source ~/.bashrc

Check if the CUDA compiler is available with which nvcc.

Note

To build imate with CUDA, you should also set CUDA_HOME, USE_CUDA, and optionally set CUDA_DYNAMIC_LOADING environment variables as described in Configure Compile-Time Environment Variables.

4.6. Load CUDA Compiler on GPU Cluster (Optional)#

This section is relevant if you are using GPU on a cluster and skip this section otherwise.

On a GPU cluster, chances are the CUDA Toolkit is already installed. If the cluster uses the module interface, load CUDA as follows.

First, check if a CUDA module is available by

module avail

Load both CUDA and GCC by

module load cuda gcc

You may specify CUDA version if multiple CUDA versions are available, such as by

module load cuda/11.7 gcc/6.3

You may check if CUDA Compiler is available with which nvcc.

4.7. Configure Compile-Time Environment Variables (Optional)#

Set the following environment variables as desired to configure the compilation process.

CUDA_HOME, CUDA_PATH, CUDA_ROOT#

These variables are relevant only if you are compiling with the CUDA compiler. Install CUDA Toolkit and specify the home directory of CUDA Toolkit by setting either of these variables. The home directory should be a path containing the executable /bin/nvcc (or \bin\nvcc.exe on Windows). For instance, if /usr/local/cuda/bin/nvcc exists, export the following:

export CUDA_HOME=/usr/local/cuda

$env:CUDA_HOME = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7"
USE_CUDA#

This variable is relevant only if you are compiling with the CUDA compiler. By default, this variable is set to 0. To compile imate with CUDA, install CUDA Toolkit and set this variable to 1 by

export USE_CUDA=1

$env:USE_CUDA = "1"
CUDA_DYNAMIC_LOADING#

This variable is relevant only if you are compiling with the CUDA compiler. By default, this variable is set to 0. When imate is complied with CUDA, the CUDA runtime libraries bundle with the final installation of imate package, making it over 700MB. While this is generally not an issue for most users, often a small package is preferable if the installed package has to be distributed to other machines. To this end, enable the custom-made dynamic loading feature of imate. In this case, the CUDA libraries will not bundle with the imate installation, rather, imate is instructed to load the existing CUDA libraries of the host machine at runtime. To enable dynamic loading, make sure CUDA Toolkit is installed, then set this variable to 1 by

export CUDA_DYNAMIC_LOADING=1

$env:CUDA_DYNAMIC_LOADING = "1"
CYTHON_BUILD_IN_SOURCE#

By default, this variable is set to 0, in which the compilation process generates source files outside of the source directory, in /build directry. When it is set to 1, the build files are generated in the source directory. To set this variable, run

export CYTHON_BUILD_IN_SOURCE=1

$env:CYTHON_BUILD_IN_SOURCE = "1"

Hint

If you generated the source files inside the source directory by setting this variable, and later you wanted to clean them, see Clean Compilation Files.

CYTHON_BUILD_FOR_DOC#

Set this variable if you are building this documentation. By default, this variable is set to 0. When it is set to 1, the package will be built suitable for generating the documentation. To set this variable, run

export CYTHON_BUILD_FOR_DOC=1

$env:CYTHON_BUILD_FOR_DOC = "1"

Warning

Do not use this option to build the package for production (release) as it has a slower performance. Building the package by enabling this variable is only suitable for generating the documentation.

Hint

By enabling this variable, the build will be in-source, similar to setting CYTHON_BUILD_IN_SOURCE=1. To clean the source directory from the generated files, see Clean Compilation Files.

USE_CBLAS#

By default, this variable is set to 0. Set this variable to 1 if you want to use OpenBLAS instead of the built-in library of imate. Install OpenBLAS and set

export USE_CBLAS=1

$env:USE_CBLAS = "1"
DEBUG_MODE#

By default, this variable is set to 0, meaning that imate is compiled without debugging mode enabled. By enabling debug mode, you can debug the code with tools such as gdb. Set this variable to 1 to enable debugging mode by

export DEBUG_MODE=1

$env:DEBUG_MODE = "1"

Attention

With the debugging mode enabled, the size of the package will be larger and its performance may be slower, which is not suitable for production.

4.8. Compile and Install#

repo-size

Get the source code of imate from the GitHub repository by

git clone https://github.com/ameli/imate.git
cd imate

To compile and install, run

python -m pip install .

The above command may need sudo privilege.

A Note on Using sudo

If you are using sudo for the above command, add -E option to sudo to make sure the environment variables (if you have set any) are accessible to the root user. For instance

export CUDA_HOME=/usr/local/cuda
export USE_CUDA=1
export CUDA_DYNAMIC_LOADING=1

sudo -E python -m pip install .

$env:CUDA_HOME = "/usr/local/cuda"
$env:USE_CUDA = "1"
$env:CUDA_DYNAMIC_LOADING = "1"

sudo -E python -m pip install .

Once the installation is completed, check the package can be loaded by

cd ..  # do not load imate in the same directory of the source code
python -c "import imate; imate.info()"

The output to the above command should be similar to the following:

imate version   : 0.15.0
processor       : Intel(R) Xeon(R) CPU E5-2623 v3 @ 3.00GHz
num threads     : 8
gpu device      : GeForce GTX 1080 Ti
num gpu devices : 4
cuda version    : 11.2.0
process memory  : 61.4 (Mb)

Attention

Do not load imate if your current working directory is the root directory of the source code of imate, since python cannot load the installed package properly. Always change the current directory to somewhere else (for example, cd .. as shown in the above).

Cleaning Compilation Files

If you set CYTHON_BUILD_IN_SOURCE or CYTHON_BUILD_FOR_DOC to 1, the output files of Cython’s compiler will be generated inside the source code directories. To clean the source code from these files (optional), run the following:

python setup.py clean