1. Install#

1.1. Install pyrand From Wheels#

Python wheels for pyrand are available for various operating systems and Python versions on both PyPI and Anaconda Cloud.

1.1.1. Install with pip#

pypi

Install pyrand and its Python dependencies through PyPI by

python -m pip install --upgrade pip
python -m pip install pyrand

If you are using PyPy instead of Python, install with

pypy -m pip install --upgrade pip
pypy -m pip install pyrand

1.1.2. Install with conda#

conda-version

Alternately, install pyrand and its Python dependencies from Anaconda Cloud by

conda install -c s-ameli pyrand -y

1.2. Install pyrand in Virtual Environments#

If you do not want the installation to occupy your main python’s site-packages (either you are testing or the dependencies may clutter your existing installed packages), install the package in an isolated virtual environment. Two common virtual environments are virtualenv and conda.

1.2.1. Install in virtualenv Environment#

  1. Install virtualenv:

    python -m pip install virtualenv

  2. Create a virtual environment and give it a name, such as pyrand_env

    python -m virtualenv pyrand_env

  3. Activate python in the new environment

    source pyrand_env/bin/activate

  4. Install pyrand package with any of the above methods. For instance:

    python -m pip install pyrand

    Then, use the package in this environment.

  5. To exit from the environment

    deactivate

1.2.2. Install in conda Environment#

In the followings, it is assumed anaconda (or miniconda) is installed.

  1. Initialize conda

    conda init

    You may need to close and reopen your terminal after the above command. Alternatively, instead of the above, you can do

    sudo sh $(conda info --root)/etc/profile.d/conda.sh

  2. Create a virtual environment and give it a name, such as pyrand_env

    conda create --name pyrand_env -y

    The command conda info --envs shows the list of all environments. The current environment is marked by an asterisk in the list, which should be the default environment at this stage. In the next step, we will change the current environment to the one we created.

  3. Activate the new environment

    source activate pyrand_env

  4. Install pyrand with any of the above methods. For instance:

    conda install -c s-ameli pyrand

    Then, use the package in this environment.

  5. To exit from the environment

    conda deactivate

1.3. Optional Runtime Dependencies#

Runtime libraries are not required to be present during the installation of pyrand. However, they may be required to be installed during running pyrand.

1.3.1. CUDA Toolkit and NVIDIA Graphic Driver (Optional)#

To use GPU devices, install NVIDIA Graphic Driver and CUDA Toolkit. See the instructions below.

1.3.2. Sparse Suite (Optional)#

Suite Sarse is a library for efficient calculations on sparse matrices. pyrand does not require this library as it has its own library for sparse matrices. However, if this library is available, pyrand uses it.

Note

The Sparse Suite library is only used for those functions in pyrand that uses the Cholesky decomposition method by passing method=cholesky argument to the functions. See API reference for Functions for details.

  1. Install Sparse Suite development library by

    sudo apt install libsuitesparse-dev

    sudo yum install libsuitesparse-devel

    sudo dnf install libsuitesparse-devel

    sudo brew install suite-sparse

    Alternatively, if you are using Anaconda python distribution (on either of the operating systems), install Suite Sparse by:

    sudo conda install -c conda-forge suitesparse

  2. Install scikit-sparse python package:

    python -m pip install scikit-sparse

1.3.3. OpenBLAS (Optional)#

OpenBLAS is a library for efficient dense matrix operations. pyrand does not require this library as it has its own library for dense matrices. However, if you compiled pyrand to use OpenBLAS (see Compile from Source), OpenBLAS library should be available at runtime.

Note

A default installation of pyrand through pip or conda does not use OpenBLAS, and you may skip this section.

Install OpenBLAS library by

sudo apt install libopenblas-dev

sudo yum install openblas-devel

sudo dnf install openblas-devel

sudo brew install openblas

Alternatively, you can install OpenBLAS using conda:

conda install -c anaconda openblas

1.4. Compile from Source#

1.4.1. When to Compile pyrand#

Generally, it is not required to compile pyrand as the installation through pip and conda contains most of its features, including support for GPU devices. You may compile pyrand if you want to:

  • modify pyrand.

  • use OpenBLAS instead of the built-in matrix library of pyrand.

  • build pyrand for a specific version of CUDA Toolkit.

  • disable the dynamic loading feature of pyrand for CUDA libraries.

  • enable debugging mode.

  • or, build this documentation.

Otherwise, install pyrand through the Python Wheels.

This section walks you through the compilation process.

1.4.2. Install C++ Compiler and OpenMP (Required)#

Compile pyrand with either of GCC, Clang/LLVM, or Intel C++ compiler on UNIX operating systems. For Windows, compile pyrand with Microsoft Visual Studio (MSVC) Compiler for C++.

Install GNU GCC Compiler

sudo apt install build-essential

sudo yum group install "Development Tools"

sudo dnf group install "Development Tools"

sudo brew install gcc libomp

Then, export C and CXX variables by

export CC=/usr/local/bin/gcc
export CXX=/usr/local/bin/g++

Install Clang/LLVN Compiler

sudo apt install clang

sudo yum install yum-utils
sudo yum-config-manager --enable extras
sudo yum makecache
sudo yum install clang

sudo dnf install yum-utils
sudo dnf config-manager --enable extras
sudo dnf makecache
sudo dnf install clang

sudo brew install llvm libomp-dev

Then, export C and CXX variables by

export CC=/usr/local/bin/clang
export CXX=/usr/local/bin/clang++

Install Intel oneAPI Compiler

To install Intel Compiler see Intel oneAPI Base Toolkit.

1.4.3. Install OpenMP (Required)#

OpenMP comes with the C++ compiler installed. However, you may alternatively install it directly on UNIX. Install OpenMP library on UNIX as follows:

sudo apt install libgomp1 -y

sudo yum install libgomp -y

sudo dnf install libgomp -y

sudo brew install libomp

1.4.4. OpenBLAS (Optional)#

pyrand can be compiled with and without OpenBLAS. If you are compiling pyrand with OpenBLAS, install OpenBLAS library by

sudo apt install libopenblas-dev

sudo yum install openblas-devel

sudo dnf install openblas-devel

sudo brew install openblas

Alternatively, you can install OpenBLAS using conda:

conda install -c anaconda openblas

Note

To build pyrand with OpenBLAS, you should also set USE_CBLAS environment variable as described in Configure Compile-Time Environment Variables.

1.4.5. Install CUDA Compiler (Optional)#

To use pyrand on GPU devices, it should be compiled with CUDA compiler. Skip this part if you are not using GPU.

Note

The minimum version of CUDA to compile pyrand is CUDA 10.0.

Attention

NVIDIA does not support macOS. You can install the NVIDIA CUDA Toolkit on Linux and Windows only.

It is not required to install the entire CUDA Toolkit. Install only the CUDA compiler and the development libraries of cuBLAS and cuSparse by

sudo apt install -y \
    cuda-nvcc-11-7 \
    libcublas-11-7 \
    libcublas-dev-11-7 \
    libcusparse-11-7 -y \
    libcusparse-dev-11-7

sudo yum install --setopt=obsoletes=0 -y \
    cuda-nvcc-11-7.x86_64 \
    cuda-cudart-devel-11-7.x86_64 \
    libcublas-11-7.x86_64 \
    libcublas-devel-11-7.x86_64 \
    libcusparse-11-7.x86_64 \
    libcusparse-devel-11-7.x86_64

sudo dnf install --setopt=obsoletes=0 -y \
    cuda-nvcc-11-7.x86_64 \
    cuda-cudart-devel-11-7.x86_64 \
    libcublas-11-7.x86_64 \
    libcublas-devel-11-7.x86_64 \
    libcusparse-11-7.x86_64 \
    libcusparse-devel-11-7.x86_64

Update PATH with the CUDA installation location by

echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' >> ~/.bashrc
source ~/.bashrc

Check if the CUDA compiler is available with which nvcc.

Note

To build pyrand with CUDA, you should also set CUDA_HOME, USE_CUDA, and optionally set CUDA_DYNAMIC_LOADING environment variables as described in Configure Compile-Time Environment Variables.

1.4.6. Load CUDA Compiler on GPU Cluster (Optional)#

This section is relevant if you are using GPU on a cluster and skip this section otherwise.

On a GPU cluster, chances are the CUDA Toolkit is already installed. If the cluster uses the module interface, load CUDA as follows.

First, check if a CUDA module is available by

module avail

Load both CUDA and GCC by

module load cuda gcc

You may specify CUDA version if multiple CUDA versions are available, such as by

module load cuda/11.7 gcc/6.3

You may check if CUDA Compiler is available with which nvcc.

1.4.7. Configure Compile-Time Environment Variables (Optional)#

Set the following environment variables as desired to configure the compilation process.

CUDA_HOME, CUDA_PATH, CUDA_ROOT#

These variables are relevant only if you are compiling with the CUDA compiler. Install CUDA Toolkit and specify the home directory of CUDA Toolkit by setting either of these variables. The home directory should be a path containing the executable /bin/nvcc (or \bin\nvcc.exe on Windows). For instance, if /usr/local/cuda/bin/nvcc exists, export the following:

export CUDA_HOME=/usr/local/cuda

$env:export CUDA_HOME = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7"
USE_CUDA#

This variable is relevant only if you are compiling with the CUDA compiler. By default, this variable is set to 0. To compile pyrand with CUDA, install CUDA Toolkit and set this variable to 1 by

export USE_CUDA=1

$env:export USE_CUDA = "1"
CUDA_DYNAMIC_LOADING#

This variable is relevant only if you are compiling with the CUDA compiler. By default, this variable is set to 0. When pyrand is complied with CUDA, the CUDA runtime libraries bundle with the final installation of pyrand package, making it over 700MB. While this is generally not an issue for most users, often a small package is preferable if the installed package has to be distributed to other machines. To this end, enable the custom-made dynamic loading feature of pyrand. In this case, the CUDA libraries will not bundle with the pyrand installation, rather, pyrand is instructed to load the existing CUDA libraries of the host machine at runtime. To enable dynamic loading, make sure CUDA Toolkit is installed, then set this variable to 1 by

export CUDA_DYNAMIC_LOADING=1

$env:export CUDA_DYNAMIC_LOADING = "1"
CYTHON_BUILD_IN_SOURCE#

By default, this variable is set to 0, in which the compilation process generates source files outside of the source directory, in /build directry. When it is set to 1, the build files are generated in the source directory. To set this variable, run

export CYTHON_BUILD_IN_SOURCE=1

$env:export CYTHON_BUILD_IN_SOURCE = "1"

Hint

If you generated the source files inside the source directory by setting this variable, and later you wanted to clean them, see Clean Compilation Files.

CYTHON_BUILD_FOR_DOC#

Set this variable if you are building this documentation. By default, this variable is set to 0. When it is set to 1, the package will be built suitable for generating the documentation. To set this variable, run

export CYTHON_BUILD_FOR_DOC=1

$env:export CYTHON_BUILD_FOR_DOC = "1"

Warning

Do not use this option to build the package for production (release) as it has a slower performance. Building the package by enabling this variable is only suitable for generating the documentation.

Hint

By enabling this variable, the build will be in-source, similar to setting CYTHON_BUILD_IN_SOURCE=1. To clean the source directory from the generated files, see Clean Compilation Files.

USE_CBLAS#

By default, this variable is set to 0. Set this variable to 1 if you want to use OpenBLAS instead of the built-in library of pyrand. Install OpenBLAS and set

export USE_CBLAS=1

$env:export USE_CBLAS = "1"
DEBUG_MODE#

By default, this variable is set to 0, meaning that pyrand is compiled without debugging mode enabled. By enabling debug mode, you can debug the code with tools such as gdb. Set this variable to 1 to enable debugging mode by

export DEBUG_MODE=1

$env:export DEBUG_MODE = "1"

Attention

With the debugging mode enabled, the size of the package will be larger and its performance may be slower, which is not suitable for production.

1.4.8. Compile and Install#

repo-size

Get the source code of pyrand from the GitHub repository by

git clone https://github.com/ameli/pyrand.git
cd pyrand

To compile and install, run

python setup.py install

The above command may need sudo privilege.

A Note on Using sudo

If you are using sudo for the above command, add -E option to sudo to make sure the environment variables (if you have set any) are accessible to the root user. For instance

export CUDA_HOME=/usr/local/cuda
export USE_CUDA=1
export CUDA_DYNAMIC_LOADING=1

sudo -E python setup.py install

$env:export CUDA_HOME = "/usr/local/cuda"
$env:export USE_CUDA = "1"
$env:export CUDA_DYNAMIC_LOADING = "1"

sudo -E python setup.py install

Once the installation is completed, check the package can be loaded by

cd ..  # do not load pyrand in the same directory of the source code
python -c "import pyrand; pyrand.info()"

The output to the above command should be similar to the following:

pyrand version   : 0.15.0
processor       : Intel(R) Xeon(R) CPU E5-2623 v3 @ 3.00GHz
num threads     : 8
gpu device      : GeForce GTX 1080 Ti
num gpu devices : 4
cuda version    : 11.2.0
process memory  : 61.4 (Mb)

Attention

Do not load pyrand if your current working directory is the root directory of the source code of pyrand, since python cannot load the installed package properly. Always change the current directory to somewhere else (for example, cd .. as shown in the above).

Cleaning Compilation Files

If you set CYTHON_BUILD_IN_SOURCE or CYTHON_BUILD_FOR_DOC to 1, the output files of Cython’s compiler will be generated inside the source code directories. To clean the source code from these files (optional), run the following:

python setup.py clean

1.5. Generate Documentation#

The documentation consists of a Doxygen documentation and a Sphinx documentation. You should generate the Doxygen documentation first. Before generating the Sphinx documentation, you should compile the package.

Get the source code from the GitHub repository.

git clone https://github.com/ameli/pyrand.git
cd pyrand

If you already had the source code, clean it from any previous build (especially if you built in-source):

python setup.py clean

1.5.1. Generate Doxygen Documentation#

Install doxygen and graphviz by

sudo apt install doxygen graphviz -y

sudo yum install doxygen graphviz -y

sudo dnf install doxygen graphviz -y

sudo brew install doxygen graphviz -y

scoop install doxygen graphviz

Attention

Make sure you ran python setup.py clean as mentioned previously. Otherwise, if the source directory is not cleaned from any previous built (if there is any), Doxygen unwantedly generates all auto-generated cython files.

Generating the Doxygen documentation by

cd docs/doxygen
doxygen doxyfile.in
cd ../..

1.5.2. Compile Package#

Set CYTHON_BUILD_FOR_DOC to 1 (see Configure Compile-Time Environment variables). Compile and install the package by

export CYTHON_BUILD_FOR_DOC=1
export USE_CUDA=0
sudo -E python setup.py install

$env:export CYTHON_BUILD_FOR_DOC = "1"
$env:export USE_CUDA = "0"
sudo -E python setup.py install

1.5.3. Generate Sphinx Documentation#

Install Pandoc by

sudo apt install pandoc -y

sudo yum install pandoc -y

sudo dnf install pandoc -y

sudo brew install pandoc -y

scoop install pandoc

Install the requirements for the Sphinx documentation by

python -m pip install -r docs/requirements.txt

The above command installs the required packages in Python’s path directory. Make sure python’s directory is on the PATH, for instance, by

PYTHON_PATH=`python -c "import os, sys; print(os.path.dirname(sys.executable))"`
export PATH=${PYTHON_PATH}:$PATH

$PYTHON_PATH = (python -c "import os, sys; print(os.path.dirname(sys.executable))")
$env:Path += ";$PYTHON_PATH"

Now, build the documentation:

make clean html --directory=docs

cd docs
make.bat clean html

The main page of the documentation can be found in /docs/build/html/index.html.

1.6. Test with pytest#

codecov-devel

The package can be tested by running several test scripts, which test all sub-packages and examples.

Clone the source code from the repository and install the required test packages by

git clone https://github.com/ameli/pyrand.git
cd pyrand
python -m pip install -r tests/requirements.txt
python setup.py install

To automatically run all tests, use pytest which is installed by the above commands.

mv pyrand pyrand-do-not-import
pytest

Attention

To properly run pytest, rename /pyrand/pyrand directory as shown in the above code. This makes pytest to properly import pyrand from the installed location, not from the source code directory.