glearn.GaussianProcess

class glearn.GaussianProcess(mean, cov)

Gaussian process prior for regression.

Parameters:

meanglearn.LinearModel

The mean function \(\mu\) of the regression.

covglearn.Covariance

The covariance function \(\Sigma\) of the regression.

See also

glearn.LinearModel

glearn.Covariance

Notes

An instance of this class creates an object representing the Gaussian process \(\mathcal{GP}(\mu, \Sigma)\), where \(\mu\) and \(\Sigma\) are respectively the mean and covariance functions of the Gaussian process model.

Training Algorithm:

The training method of this class is based on the algorithm described in [1].

References

[1]

Ameli, S., and Shadden. S. C. (2022). Noise Estimation in Gaussian Process Regression. arXiv: 2206.09976 [cs.LG].

Examples

To define a Gaussian process object \(\mathcal{GP}(\mu, \Sigma)\), first, an object for the linear model where \(\mu\) and an object for the covariance model \(\Sigma\) should be created as follows.

1. Generate Sample Training Data:

>>> import glearn
>>> from glearn import sample_data

>>> # Generate a set of training points
>>> x = sample_data.generate_points(
...     num_points=30, dimension=1, grid=False,a=0.4, b=0.6,
...     contrast=0.9, seed=42)

>>> # Generate noise sample data on the training points
>>> y_noisy = glearn.sample_data.generate_data(x, noise_magnitude=0.1)

2. Create Linear Model:

Create an object for \(\mu\) function using glearn.LinearModel class. On training points, the mean function is represented by the array

\[\boldsymbol{\mu} = \boldsymbol{\phi}^{\intercal}(\boldsymbol{x}) \boldsymbol{\beta}.\]

>>> # Create mean object using glearn.
>>> mean = glearn.LinearModel(x, polynomial_degree=2)

3. Create Covariance Object:

Create the covariance model using glearn.Covariance class. On the training points, the covariance function is represented by the matrix

\[\boldsymbol{\Sigma}(\sigma, \varsigma, \boldsymbol{\alpha}) = \sigma^2 \mathbf{K}(\boldsymbol{\alpha}) + \varsigma^2 \mathbf{I}.\]

>>> # Define a Cauchy prior for scale hyperparameter
>>> scale = glearn.priors.Cauchy()

>>> # Create a covariance object
>>> cov = glearn.Covariance(x, scale=scale)

4. Create Gaussian Process Object:

Putting all together, we can create an object for \(\mathcal{GP} (\mu, \Sigma)\) as follows:

>>> # Gaussian process object
>>> gp = glearn.GaussianProcess(mean, cov)

To train the model and predict on test data, see glearn.GaussianProcess.train() and glearn.GaussianProcess.predict().

Train the model to find the regression parameter \(\boldsymbol{\beta}\) and the hyperparameters \(\sigma\), :math:``

Attributes:

znumpy.array

An array of the size \(n\) representing the training data.

meanglearn.LinearModel

An object representing the mean function of the Gaussian process prior. Once the model is trained, the hyperparameters of this object will be updated with their optimal values.

covglearn.Covariance

An object representing the covariance function of the Gaussian process prior. Once the model is trained, the hyperparameters of this object will be updated with their optimal values.

timerglearn.Timer

A timer object that keeps track of process time and elapsed time of the training process.

memoryglearn.Memory

A memory object that keeps track of resident memory acquired during the computation.

training_resultdict

A dictionary containing the training hyperparameters after the model is trained. This attribute is initially an empty dictionary, however, after calling glearn.GaussianProcess.train(), this dictionary is updated and will have the following keys:

• 'config':

  • 'max_bracket_trials': int, maximum number of trials to search for a bracket with sign-change in Chandrupatla root finding method. This option is relevant if optimization_method is set to chandrupatla.

  • 'max_iter': int, maximum number of iterations during optimization process.

  • 'optimization_method' : {‘chandrupatla’, ‘brentq’, ‘Nelder-Mead’, ‘BFGS’, ‘CG’, ‘Newton-CG’, ‘dogleg’, ‘trust-exact’, ‘trust-ncg’}, the optimization method.

  • 'profile_hyperparam' : {‘none’, ‘var’, ‘var_noise’}, the profiling method of the likelihood function.

  • 'tol' : float, the tolerance of convergence of hyperparameters during optimization.

  • 'use_rel_error': bool, whether to use relative error or absolute error for the convergence criterion of the hyperparameters during the optimization.

• 'convergence':

  • 'converged': numpy.array, an boolean array of the size of the number of the unknown hyperparameters, indicating which of the hyperparameters were converged according to the convergence criterion.

  • 'errors': numpy.ndarray, a 2D array of the size of \(n_{\mathrm{itr}} \times, p\) where \(n_{\mathrm{itr}}\) is the number of the optimization iterations and \(p\) is the number of unknown hyperparameters. The \(i\)-th row of this array is the error of the convergence of the hyperparameters at the \(i\)-th iteration.

  • 'hyperparams': numpy.ndarray, a 2D array of the size of \(n_{\mathrm{itr}} \times, p\) where \(n_{\mathrm{itr}}\) is the number of the optimization iterations and \(p\) is the number of unknown hyperparameters. The \(i\)-th row of this array is the solved hyperparameters at the \(i\)-th iteration.

• 'data':

  • 'size': int, the size of the data.

  • 'sparse': bool, determines whether the covariance matrix is sparse.

  • 'nnz': int, the number of non-zero elements of the covariance matrix.

  • 'avg_row_nnz': float, the average number of non-zero elements of the rows of the covariance matrix.

  • 'density': float, the density of the covariance matrix, if sparse.

  • 'dimension': int, the dimension of the space of data points.

  • 'kernel_threshold': float, the threshold to tapper the kernel function to produce sparse covariance matrix.

• 'device':

  • 'memory_usage': [int, str], a list where the first entry is the residence memory acquired during the computation, and the second entry of the list is the unit of the memory, which is one of 'b', 'Kb', 'M', 'Gb', etc.

  • 'num_cpu_threads': int, number of CPU threads used during the computation.

  • 'num_gpu_devices': int, number of GPU devices used during the computation.

  • 'num_gpu_multiproc': int, number of GPU multi-processors per a GPU device.

  • 'num_gpu_threads_per_multiproc': int, number of GPU threads per a GPU multiprocessors in each GPU device.

• 'hyperparam':

  • 'eq_sigma': float, equivalent variance, which is \(\sqrt{\sigma^2 + \varsigma^2}\).

  • 'sigma': float, the hyperparameter \(\sigma\).

  • 'sigma0': float, the hyperparameter \(\varsigma\).

  • 'eta': float, the ratio \(\eta = \varsigma^2/\sigma^2\).

  • 'scale': numpy.array, the scale hyperparameters \(\boldsymbol{\alpha} = (\alpha_1, \dots, \alpha_d)\).

• 'optimization':

  • 'max_fun': float, maximum value of the optimal likelihood function.

  • 'message': str, message about optimization success or error.

  • 'num_cor_eval': int, number of evaluations of the correlation matrix.

  • 'num_fun_eval': int, number of evaluations of the likelihood function.

  • 'num_jac_eval': int, number of evaluations of the Jacobian of the likelihood function.

  • 'num_hes_eval': int, number of evaluations of the Hessian of the likelihood function.

  • 'num_opt_eval': int, number of optimization iterations.

• 'time':

  • 'cor_count': int, number of evaluations of correlation matrix.

  • 'cor_proc_time': float, process time of evaluating correlation matrix.

  • 'cor_wall_time': float, wall time of evaluating correlation matrix.

  • 'det_count': int, number of evaluations of log-determinants.

  • 'cor_proc_time': float, process time of evaluating log-determinants.

  • 'det_wall_time': float, wall time of evaluating log determinants.

  • 'lik_count': int, number of evaluations of the likelihood function.

  • 'lik_proc_time': float, process time of evaluating the likelihood function.

  • 'lik_wall_time': float, wall time of evaluating the

  • 'opt_count': int, number of optimization iterations.

  • 'opt_proc_time': float, process time of optimization.

  • 'opt_wall_time': float, wall time of optimization.

  • 'sol_count': int, number of solving linear systems.

  • 'sol_proc_time': float, process time of linear systems.

  • 'sol_wall_time': float, wall time of solving linear systems.

  • 'trc_count': int, number of evaluating the trace of matrices.

  • 'trc_proc_time': float, process time of evaluating the trace of matrices.

  • 'trc_wall_time': float, wall time of evaluating the trace of matrices.

• 'imate_config':

  See info output of imate.logdet(), imate.trace(), or imate.traceinv().

prediction_result: dict

A dictionary containing the prediction results. This attribute is initially empty, but once the function glearn.GaussianProcess.predict() is called, this attribute is updated with the following keys:

• 'config':

  • 'cov': bool, indicates whether the posterior covariance was computed.

  • 'num_test_points': int, number of test points, \(n^{\ast}\).

  • 'num_training_points': int, number of training points \(n\).

• 'process':

  • 'memory': [int, unit], a list that contains the used resident memory and its unit in the form of either of b, Kb, Mb, Gb, etc.

  • 'proc_time': float, the process time of computing prediction.

  • 'wall_time': float, the wall time of computing prediction.

Methods

train(z[, hyperparam_guess, ...])	Train the hyperparameters of the Gaussian process model.
predict(test_points[, cov, plot, true_data, ...])	Make prediction on test points.
plot_likelihood()	Plot the likelihood function and its derivatives with respect to the hyperparameters.