glearn.LinearModel#

class glearn.LinearModel(x, polynomial_degree=0, trigonometric_coeff=None, hyperbolic_coeff=None, func=None, orthonormalize=False, b=None, B=None)#

Linear model for the mean function of regression.

For the regression problem \(y=\mu(\boldsymbol{x})+\delta(\boldsymbol{x})\) where \(\mu\) and \(\delta\) are the mean and miss-fit of the regression model respectively, this class implements the linear model \(\mu(\boldsymbol{x})= \boldsymbol{\phi}(\boldsymbol{x})^{\intercal} \boldsymbol{\beta}\), where:

  • The basis functions \(\boldsymbol{\phi}\) can be specified by polynomial, trigonometric, hyperbolic, or user-defined arbitrary functions.

  • The uniform or multi-dimensional normal prior distribution can be assigned to the vector of regression coefficients \(\boldsymbol{\beta}\sim \mathcal{N}(\boldsymbol{b},\sigma^2 \mathbf{B})\).

Parameters:
xnumpy.ndarray

A 2D array of data points where each row of the array is the coordinate of a point \(\boldsymbol{x} = (x_1, \dots, x_d)\). The array size is \(n \times d\) where \(n\) is the number of the points and \(d\) is the dimension of the space of points.

polynomial_degreeint, default=0

Degree \(p\) of polynomial basis functions \(\boldsymbol{\phi}_p(\boldsymbol{x})\), which defines the monomial functions of order less than or equal to \(p\) on the variables \(\boldsymbol{x} = (x_1, \dots, x_d)\). The total number of monomials is denoted by \(m_p\).

trigonometric_coefffloat or array_like[float], default=None

The coefficients \(t_i, i=1,\dots, m_t\), of the trigonometric basis functions \(\boldsymbol{\phi}_t(\boldsymbol{x})\) (see Notes below). If None, no trigonometric basis function is generated. The total number of trigonometric basis functions is \(2 m_t\).

hyperbolic_coefffloat or array_like[float], default=None

The coefficients \(h_i, i=1, \dots, m_h\), of the hyperbolic basis functions \(\boldsymbol{\phi}_h(\boldsymbol{x})\) (see Notes below). If None, no hyperbolic basis function is generated. The total number of hyperbolic basis functions is \(2 m_h\).

funccallable, default=None

A callable function to define arbitrary basis functions \(\boldsymbol{\phi}(\boldsymbol{x})\) where it accepts an input point \(\boldsymbol{x}=(x_1, \dots, m_d)\) and returns the array of \(m_f\) basis functions \(\boldsymbol{\phi} = ( \phi_1, \dots, \phi_{m_f})\). If None, no arbitrary basis function is generated. The total number of user-defined basis functions is denoted by \(m_f\).

orthonormalizebool, default=False

If True, the design matrix \(\mathbf{X}\) of the basis functions will be orthonormalized. The orthonormalization is applied on all \(m\) basis functions, including polynomial, trigonometric, hyperbolic, and user-defined basis functions.

bnumpy.array, default=None,

The variable \(\boldsymbol{b}\) which is the mean of the prior distribution for the variable \(\boldsymbol{\beta} \sim \mathcal{N}(\boldsymbol{b}, \sigma^2 \mathbf{B})\). The variable b should be given as a 1D array of the size \(m = m_p+2m_t+ 2m_h+m_f\). If None, uniform prior is assumed for the variable \(\boldsymbol{\beta}\).

Bnumpy.array, default=None,

The matrix \(\mathbf{B}\) which makes the covariance of the prior distribution for the variable \(\boldsymbol{\beta} \sim \mathcal{N}(\boldsymbol{b}, \sigma^2 \mathbf{B})\). The matrix B should be symmetric and positive semi-definite with the size \(m \times m\) where \(m = m_p+2m_t+2m_h+m_f\).

  • If B is not None, the vector b also cannot be None.

  • If B is None, it is assumed that \(\mathbf{B}^{-1} \to \mathbf{0}\), which implies the prior on \(\boldsymbol{\beta}\) is the uniform distribution. In this case, the argument b (if given) is ignored.

See also

glearn.Covariance
glearn.GaussianProcess

Notes

Regression Model:

A regression model to fit the data \(y = f(\boldsymbol{x})\) for the points \(\boldsymbol{x} \in \mathcal{D} \in \mathbb{R}^d\) and data \(y \in \mathbb{R}\) is

\[f(\boldsymbol{x}) = \mu(\boldsymbol{x}) + \delta(\boldsymbol{x}),\]

where \(\mu\) is a deterministic function representing the features of the data and \(\delta\) is a stochastic function representing both the uncertainty of the data and miss-fit of the regression model.

Linear Regression Model:

This class represents a linear regression model, where it is assumed that

\[\mu(\boldsymbol{x}) = \boldsymbol{\phi}(\boldsymbol{x})^{\intercal} \boldsymbol{\beta},\]

where \(\boldsymbol{\phi} = (\phi_1, \dots, \phi_m): \mathcal{D} \to \mathbb{R}^m\) is a vector basis function and \(\boldsymbol{\beta} = (\beta_1, \dots, \beta_m)\) is a vector of the unknown coefficients of the linear model. The basis functions can be a combination of the following functions:

\[\boldsymbol{\phi}(\boldsymbol{x}) = \left( \boldsymbol{\phi}_p(\boldsymbol{x}), \boldsymbol{\phi}_t(\boldsymbol{x}), \boldsymbol{\phi}_h(\boldsymbol{x}), \boldsymbol{\phi}_f(\boldsymbol{x}) \right),\]

consisting of

  • Polynomial basis functions \(\boldsymbol{\phi}_p: \mathcal{D} \to \mathbb{R}^{m_p}\).

  • Trigonometric basis functions \(\boldsymbol{\phi}_t: \mathcal{D} \to \mathbb{R}^{2m_t}\).

  • Hyperbolic basis functions \(\boldsymbol{\phi}_h: \mathcal{D} \to \mathbb{R}^{2m_h}\).

  • User-defined basis functions \(\boldsymbol{\phi}_f: \mathcal{D} \to \mathbb{R}^{m_f}\).

The total number of the basis functions, \(m\), is

\[m = m_p + 2m_t + 2m_h + m_f.\]

Each of the above functions are described below.

  • Polynomial Basis Functions: A polynomial basis function of order \(p\) (set by polynomial_order) is a tuple of all monomials up to the order less than or equal to \(p\) from the combination of the components of the variable \(\boldsymbol{x} = (x_1, \dots, x_d)\). For instance, if \(d = 2\) where \(\boldsymbol{x} = (x_1, x_2)\), a polynomial basis of order 3 is

    \[\boldsymbol{\phi}_p(x_1, x_2) = ( \ 1, \ x_1, x_2, \ x_1^2, x_1 x_2, x_2^2, \ x_1^3, x_1^2 x_2, x_1 x_2^2, x_2^3).\]

    The size of the tuple \(\boldsymbol{\phi}_p\) is denoted by \(m_p\), for instance, in the above, \(m_p = 10\).

  • Trigonometric Basis Function: Given the coefficients \((t_1, \dots, t_{m_t})\) which can be set by trigonometric_coeff as a list or numpy array, the trigonometric basis functions \(\boldsymbol{\phi}_t\) is defined by

    \[\boldsymbol{\phi}_t(\boldsymbol{x}) = (\boldsymbol{\phi}_s(\boldsymbol{x}), \boldsymbol{\phi}_c(\boldsymbol{x})),\]

    where

    \[\begin{split}\begin{align} \boldsymbol{\phi}_s(\boldsymbol{x}) = ( & \sin(t_1 x_1), \dots, \sin(t_1 x_d), \\ & \dots, \\ & \sin(t_i x_1), \dots, \sin(t_i x_d), \\ & \dots, \\ & \sin(t_{m_t} x_1), \dots, \sin(t_{m_t} x_d)). \end{align}\end{split}\]

    and

    \[\begin{split}\begin{align} \boldsymbol{\phi}_c(\boldsymbol{x}) = ( & \cos(t_1 x_1), \dots, \cos(t_1 x_d), \\ & \dots, \\ & \cos(t_i x_1), \dots, \cos(t_i x_d), \\ & \dots, \\ & \cos(t_{m_t} x_1), \dots, \cos(t_{m_t} x_d)). \end{align}\end{split}\]

  • Hyperbolic Basis Function: Given the coefficients \((h_1, \dots, h_{m_h})\) which can be set by hyperbolic_coeff as a list or numpy array, the hyperbolic basis functions \(\boldsymbol{\phi}_h\) is defined by

    \[\boldsymbol{\phi}_h(\boldsymbol{x}) = (\boldsymbol{\phi}_{sh}(\boldsymbol{x}), \boldsymbol{\phi}_{ch}(\boldsymbol{x})),\]

    where

    \[\begin{split}\begin{align} \boldsymbol{\phi}_{sh}(\boldsymbol{x}) = ( & \sinh(h_1 x_1), \dots, \sinh(h_1 x_d), \\ & \dots, \\ & \sinh(h_i x_1), \dots, \sinh(h_i x_d), \\ & \dots, \\ & \sinh(h_{m_h} x_1), \dots, \sinh(h_{m_h} x_d)). \end{align}\end{split}\]

    and

    \[\begin{split}\begin{align} \boldsymbol{\phi}_{ch}(\boldsymbol{x}) = ( & \cosh(h_1 x_1), \dots, \cosh(h_1 x_d), \\ & \dots, \\ & \cosh(h_i x_1), \dots, \cosh(h_i x_d), \\ & \dots, \\ & \cosh(h_{m_h} x_1), \dots, \cosh(h_{m_h} x_d)). \end{align}\end{split}\]

  • User-Defined Basis Functions: Given the function \(\boldsymbol{\phi}_f(\boldsymbol{x})\) which can be set by the argument func, the custom basis function are generated by

    \[\boldsymbol{\phi}_f(\boldsymbol{x}) = \left( \boldsymbol{\phi}_{f, 1}(\boldsymbol{x}), \dots, \boldsymbol{\phi}_{f, m_f}(\boldsymbol{x}) \right).\]

Design Matrix:

The design matrix \(\mathbf{X}\) of the size \(n \times m\) is generated from \(n\) points \(\boldsymbol{x}\) and \(m\) basis functions with the components \(X_{ij}\) as

\[X_{ij} = \phi_j(\boldsymbol{x}_i), \quad i = 1, \dots, n, \quad j = 1, \dots, m.\]

If orthonormalize is True, the matrix \(\mathbf{X}\) is orthonormalized such that \(\mathbf{X}^{\intercal} \mathbf{X} = \mathbf{I}\) where \(\mathbf{I}\) is the identity matrix.

Note

The design matrix on data points is automatically generated during the internal training process and can be accessed by X attribute. To generate the design matrix on arbitrary test points, call glearn.LinearModel.generate_design_matrix() function.

Prior of Unknown Coefficients:

The coefficients \(\boldsymbol{\beta}\) of the size \(m\) is unknown at the time of instantiation of glearn.LinearModel class. However, its prior distribution can be specified as

\[\boldsymbol{\beta} \sim \mathcal{N}(\boldsymbol{b}, \sigma^2 \mathbf{B}),\]

where the hyperparameter \(\sigma\) is to be found during the training process, and the hyperparameters \(\boldsymbol{b}\) and \(\mathbf{B}\) can be specified by the arguments b and B respectively.

Posterior of Estimated Coefficients:

Once the model has been trained by glearn.GaussianProcess, the hyperparameters of the posterior distribution

\[\boldsymbol{\beta} \sim \mathcal{N}(\bar{\boldsymbol{\beta}}, \mathbf{C}),\]

can be accessed by these attributes:

  • \(\bar{\boldsymbol{\beta}}\) can be accessed by LinearModel.beta.

  • \(\mathbf{C}\) can be accessed by LinearModel.C.

Note

The hyperparameters of the posterior distribution of \(\boldsymbol{\beta}\) is automatically computed after the training process. However, to manually update the hyperparameters of the posterior distribution after the training process, call glearn.LinearModel.update_hyperparam() function.

Examples

Create Polynomial, Trigonometric, and Hyperbolic Basis Functions:

First, create a set of 50 random points in the interval \([0, 1]\).

>>> # Generate a set of points
>>> from glearn.sample_data import generate_points
>>> x = generate_points(num_points=30, dimension=1)

Define a prior for \(\boldsymbol{\beta} \vert \sigma^2 \sim \mathcal{N}(\boldsymbol{b}, \sigma^2 \mathbf{B})\) by the mean vector \(\boldsymbol{b}\) and covariance matrix \(\mathbf{B}\).

>>> import numpy
>>> m = 10
>>> b = numpy.zeros((m, ))

>>> # Generate a random matrix B for covariance of beta.
>>> numpy.random.seed(0)
>>> B = numpy.random.rand(m, m)

>>> # Making sure the covariance matrix B positive-semidefinite
>>> B = B.T @ B

Create User-Defined Basis Functions:

Define the function \(\boldsymbol{\phi}\) as follows:

>>> # Create a function returning the Legendre Polynomials
>>> import numpy
>>> def func(x):
...     phi_1 = 0.5 * (3.0*x**2 - 1)
...     phi_2 = 0.5 * (5.0*x**3 - 3.0*x)
...     phi = numpy.array([phi_1, phi_2])
...     return phi

Create Linear Model:

Create a linear model with first order polynomial basis functions, both trigonometric and hyperbolic functions, and the user-defined functions created in the above.

>>> # Create basis functions
>>> from glearn import LinearModel
>>> f = LinearModel(x, polynomial_degree=1,
...                 trigonometric_coeff=[1.0, 2.0],
...                 hyperbolic_coeff=[3.0], func=func, b=b, B=B)

Note that we set \(m = 10\) since we have

  • \(m_p=2\) polynomial basis functions \(\boldsymbol{\phi}(x) = (1, x)\) of order 0 and 1.

  • \(2m_t\) trigonometric basis functions with \(m_t=2\) as \(\boldsymbol{\phi} = (\sin(x), \sin(2x), \cos(x), \cos(2x))\).

  • \(2m_h\) hyperbolic basis functions with \(m_h=1\) as \(\boldsymbol{\phi}=(\sinh(3x),\cosh(3x))\).

  • \(m_f=2\) user-defined basis functions \(\boldsymbol{\phi}(x) = (\frac{1}{2}(3x^3-1), \frac{1}{2}(5x^3-3x))\).

Recall, the total number of basis functions is \(m = m_p + 2m_t + 2m_h + m_f\), so in total there are 10 basis functions. Because of this, the vector b and matrix B were defined with the size \(m=10\).

Generate Design Matrix:

The design matrix \(\mathbf{X}\) can be accessed by X attribute:

>>> # Get design matrix on data points
>>> X = f.X
>>> X.shape
(30, 10)

Alternatively, the design matrix \(\mathbf{X}^{\ast}\) can be generated on arbitrary test points \(\boldsymbol{x}^{\ast}\) as follows:

>>> # Generate 100 test points
>>> x_test = generate_points(num_points=100, dimension=1)

>>> # Generate design matrix on test points
>>> X_test = f.generate_design_matrix(x_test)
>>> X_test.shape
(100, 10)

Orthonormalize Basis Functions:

Repeat the above example but set orthonormalize to True:

>>> # Create basis functions
>>> f = LinearModel(x, polynomial_degree=1,
...                 trigonometric_coeff=[1.0, 2.0],
...                 hyperbolic_coeff=[3.0], func=func,
...                 orthonormalize=True)

>>> # Check the orthonormality of X
>>> I = numpy.eye(m)
>>> numpy.allclose(f.X.T @ f.X, I, atol=1e-6)
True

>>> # Generate design matrix on test points
>>> X_test = f.generate_design_matrix(x_test, orthonormalize=True)

>>> # Check the orthonormality of X
>>> numpy.allclose(X_test.T @ X_test, I, atol=1e-6)
True
Attributes:
pointsnumpy.ndarray

The same as input variable x.

Xnumpy.ndarray

The generated design matrix.

Binvnumpy.ndarray, default=None

Inverse of the matrix B (if not None).

betanumpy.array

The mean of the coefficient \(\boldsymbol{\beta}\). This coefficient is computed for a given data.

Cnumpy.ndarray

The posterior covariance of coefficient \(\boldsymbol{\beta}\). This matrix computed for a given data.

Methods

generate_design_matrix(x[, orthonormalize])

Generates design matrix on test points.

update_hyperparam(cov, y)

Manually update the posterior mean and covariance of linear model coefficient.