loss

Regression loss for Gaussian kernel regression model

Syntax

L = loss(Mdl,X,Y)

L = loss(Mdl,X,Y,Name,Value)

Description

L = loss(Mdl,X,Y) returns the mean squared error (MSE) for the Gaussian kernel regression model Mdl using the predictor data in X and the corresponding responses in Y.

L = loss(Mdl,X,Y,Name,Value) uses additional options specified by one or more name-value pair arguments. For example, you can specify a regression loss function and observation weights. Then, loss returns the weighted regression loss using the specified loss function.

Examples

Calculate Sample Loss for Gaussian Kernel Regression Model

Train a Gaussian kernel regression model for a tall array, then calculate the resubstitution mean squared error and epsilon-insensitive error.

When you perform calculations on tall arrays, MATLAB® uses either a parallel pool (default if you have Parallel Computing Toolbox™) or the local MATLAB session. If you want to run the example using the local MATLAB session when you have Parallel Computing Toolbox, you can change the global execution environment by using the mapreducer function.

Create a datastore that references the folder location with the data. The data can be contained in a single file, a collection of files, or an entire folder. Treat 'NA' values as missing data so that datastore replaces them with NaN values. Select a subset of the variables to use. Create a tall table on top of the datastore.

varnames = {'ArrTime','DepTime','ActualElapsedTime'};
ds = datastore('airlinesmall.csv','TreatAsMissing','NA',...
    'SelectedVariableNames',varnames);
t = tall(ds);

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 4).

Specify DepTime and ArrTime as the predictor variables (X) and ActualElapsedTime as the response variable (Y). Select the observations for which ArrTime is later than DepTime.

daytime = t.ArrTime>t.DepTime;
Y = t.ActualElapsedTime(daytime);     % Response data
X = t{daytime,{'DepTime' 'ArrTime'}}; % Predictor data

Standardize the predictor variables.

Z = zscore(X); % Standardize the data

Train a default Gaussian kernel regression model with the standardized predictors. Set 'Verbose',0 to suppress diagnostic messages.

[Mdl,FitInfo] = fitrkernel(Z,Y,'Verbose',0)

Mdl = 
  RegressionKernel
            PredictorNames: {'x1'  'x2'}
              ResponseName: 'Y'
                   Learner: 'svm'
    NumExpansionDimensions: 64
               KernelScale: 1
                    Lambda: 8.5385e-06
             BoxConstraint: 1
                   Epsilon: 5.9303


  Properties, Methods

FitInfo = struct with fields:
                  Solver: 'LBFGS-tall'
            LossFunction: 'epsiloninsensitive'
                  Lambda: 8.5385e-06
           BetaTolerance: 1.0000e-03
       GradientTolerance: 1.0000e-05
          ObjectiveValue: 30.7814
       GradientMagnitude: 0.0191
    RelativeChangeInBeta: 0.0228
                 FitTime: 108.0340
                 History: []

Mdl is a trained RegressionKernel model, and the structure array FitInfo contains optimization details.

Determine how well the trained model generalizes to new predictor values by estimating the resubstitution mean squared error and epsilon-insensitive error.

lossMSE = loss(Mdl,Z,Y) % Resubstitution mean squared error

lossMSE =

  MxNx... tall array

    ?    ?    ?    ...
    ?    ?    ?    ...
    ?    ?    ?    ...
    :    :    :
    :    :    :

lossEI = loss(Mdl,Z,Y,'LossFun','epsiloninsensitive') % Resubstitution epsilon-insensitive error

lossEI =

  MxNx... tall array

    ?    ?    ?    ...
    ?    ?    ?    ...
    ?    ?    ?    ...
    :    :    :
    :    :    :

Evaluate the tall arrays and bring the results into memory by using gather.

[lossMSE,lossEI] = gather(lossMSE,lossEI)

Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 1: Completed in 2.5 sec
Evaluation completed in 2.7 sec

lossMSE = 2.8851e+03

lossEI = 28.0050

Specify Custom Regression Loss

Specify a custom regression loss (Huber loss) for a Gaussian kernel regression model.

Load the carbig data set.

load carbig

Specify the predictor variables (X) and the response variable (Y).

X = [Weight,Cylinders,Horsepower,Model_Year];
Y = MPG;

Delete rows of X and Y where either array has NaN values. Removing rows with NaN values before passing data to fitrkernel can speed up training and reduce memory usage.

R = rmmissing([X Y]); 
X = R(:,1:4); 
Y = R(:,end); 

Reserve 10% of the observations as a holdout sample. Extract the training and test indices from the partition definition.

rng(10)  % For reproducibility
N = length(Y);
cvp = cvpartition(N,'Holdout',0.1);
idxTrn = training(cvp); % Training set indices
idxTest = test(cvp);    % Test set indices

Standardize the training data and train the regression kernel model.

Xtrain = X(idxTrn,:);
Ytrain = Y(idxTrn);
[Ztrain,tr_mu,tr_sigma] = zscore(Xtrain); % Standardize the training data
tr_sigma(tr_sigma==0) = 1;
Mdl = fitrkernel(Ztrain,Ytrain)

Mdl = 
  RegressionKernel
              ResponseName: 'Y'
                   Learner: 'svm'
    NumExpansionDimensions: 128
               KernelScale: 1
                    Lambda: 0.0028
             BoxConstraint: 1
                   Epsilon: 0.8617


  Properties, Methods

Mdl is a RegressionKernel model.

Create an anonymous function that measures Huber loss $(\delta =1)$, that is,

$$L=\frac{1}{\sum w_j}\sum _{j=1}^n{w}_j{\ell }_j,$$

where

$$\begin{array}{l}\\ {\ell }_j=\{\begin{array}{c}0.5{\underset{}{\overset{ˆ}{e_j}}}^2;\\ \left|\underset{}{\overset{ˆ}{e_j}}\right|-0.5;\end{array}\begin{array}{c}\left|\underset{}{\overset{ˆ}{e_j}}\right|\le 1\\ \left|\underset{}{\overset{ˆ}{e_j}}\right|>1\end{array}.\end{array}$$

$\underset{}{\overset{ˆ}{e_j}}$ is the residual for observation j. Custom loss functions must be written in a particular form. For rules on writing a custom loss function, see the 'LossFun' name-value pair argument.

huberloss = @(Y,Yhat,W)sum(W.*((0.5*(abs(Y-Yhat)<=1).*(Y-Yhat).^2) + ...
    ((abs(Y-Yhat)>1).*abs(Y-Yhat)-0.5)))/sum(W);

Estimate the training set regression loss using the Huber loss function.

eTrain = loss(Mdl,Ztrain,Ytrain,'LossFun',huberloss)

eTrain = 1.7210

Standardize the test data using the same mean and standard deviation of the training data columns. Estimate the test set regression loss using the Huber loss function.

Xtest = X(idxTest,:);
Ztest = (Xtest-tr_mu)./tr_sigma; % Standardize the test data
Ytest = Y(idxTest);

eTest = loss(Mdl,Ztest,Ytest,'LossFun',huberloss)

eTest = 1.3062

Input Arguments

Mdl — Kernel regression model

RegressionKernel model object

Kernel regression model, specified as a RegressionKernel model object. You can create a RegressionKernel model object using fitrkernel.

X — Predictor data

n-by-p numeric matrix

Predictor data, specified as an n-by-p numeric matrix, where n is the number of observations and p is the number of predictors. p must be equal to the number of predictors used to train Mdl.

Data Types: single | double

Y — Response data

numeric vector

Response data, specified as an n-dimensional numeric vector. The length of Y and the number of observations in X must be equal.

Data Types: single | double

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: L = loss(Mdl,X,Y,'LossFun','epsiloninsensitive','Weights',weights) returns the weighted regression loss using the epsilon-insensitive loss function.

'LossFun' — Loss function

'mse' (default) | 'epsiloninsensitive' | function handle

Loss function, specified as the comma-separated pair consisting of 'LossFun' and a built-in loss function name or a function handle.

• The following table lists the available loss functions. Specify one using its corresponding character vector or string scalar. Also, in the table, $$f\left(x\right)=T(x)\beta +b.$$

  • x is an observation (row vector) from p predictor variables.

  • $$T(·)$$ is a transformation of an observation (row vector) for feature expansion. T(x) maps x in $${ℝ}^p$$ to a high-dimensional space ($${ℝ}^m$$).

  • β is a vector of m coefficients.

  • b is the scalar bias.

  Value	Description
  'epsiloninsensitive'	Epsilon-insensitive loss: $$\ell \left[y,f\left(x\right)\right]=\max \left[0,\left|y-f\left(x\right)\right|-\epsilon \right]$$
  'mse'	MSE: $$\ell \left[y,f\left(x\right)\right]={\left[y-f\left(x\right)\right]}^2$$

  'epsiloninsensitive' is appropriate for SVM learners only.

• Specify your own function by using function handle notation.

  Let n be the number of observations in X. Your function must have this signature:

  lossvalue = lossfun(Y,Yhat,W)

  • The output argument lossvalue is a scalar.

  • You choose the function name (lossfun).

  • Y is an n-dimensional vector of observed responses. loss passes the input argument Y in for Y.

  • Yhat is an n-dimensional vector of predicted responses, which is similar to the output of predict.

  • W is an n-by-1 numeric vector of observation weights.

  Specify your function using 'LossFun',@lossfun.

Data Types: char | string | function_handle

'Weights' — Observation weights

ones(n,1)/n (default) | numeric vector of positive values

Observation weights, specified as the comma-separated pair consisting of 'Weights' and a numeric vector of positive values. loss weighs the observations in X with the corresponding values in Weights. The size of Weights must equal n, the number of observations (rows in X). If you supply the observation weights, loss computes the weighted regression loss, that is, the Weighted Mean Squared Error or Epsilon-Insensitive Loss Function.

loss normalizes Weights to sum to 1.

Data Types: double | single

Output Arguments

L — Regression loss

numeric scalar

Regression loss, returned as a numeric scalar. The interpretation of L depends on Weights and LossFun. For example, if you use the default observation weights and specify 'epsiloninsensitive' as the loss function, then L is the epsilon-insensitive loss.

More About

Weighted Mean Squared Error

The weighted mean squared error is calculated as follows:

$$\text{mse}=\frac{{\displaystyle \sum _{j=1}^n{w}_j{\left(f\left(x_j\right)-y_j\right)}^2}}{{\displaystyle \sum _{j=1}^n{w}_j}},$$

where:

• n is the number of observations.

• x_j is the jth observation (row of predictor data).

• y_j is the observed response to x_j.

• f(x_j) is the response prediction of the Gaussian kernel regression model Mdl to x_j.

• w is the vector of observation weights.

Each observation weight in w is equal to ones(n,1)/n by default. You can specify different values for the observation weights by using the 'Weights' name-value pair argument. loss normalizes Weights to sum to 1.

Epsilon-Insensitive Loss Function

The epsilon-insensitive loss function ignores errors that are within the distance epsilon (ε) of the function value. The function is formally described as:

$$Los{s}_{\epsilon }=\{\begin{array}{c}0,if\left|y-f\left(x\right)\right|\le \epsilon \\ \left|y-f\left(x\right)\right|-\epsilon ,otherwise.\end{array}$$

The mean epsilon-insensitive loss is calculated as follows:

$$Loss=\frac{{\displaystyle \sum _{j=1}^n{w}_j\max \left(0,\left|y_j-f\left(x_j\right)\right|-\epsilon \right)}}{{\displaystyle \sum _{j=1}^n{w}_j}},$$

where:

• n is the number of observations.

• x_j is the jth observation (row of predictor data).

• y_j is the observed response to x_j.

• f(x_j) is the response prediction of the Gaussian kernel regression model Mdl to x_j.

• w is the vector of observation weights.

Each observation weight in w is equal to ones(n,1)/n by default. You can specify different values for the observation weights by using the 'Weights' name-value pair argument. loss normalizes Weights to sum to 1.

Extended Capabilities

Tall Arrays

Calculate with arrays that have more rows than fit in memory.

This function fully supports tall arrays. For more information, see Tall Arrays (MATLAB).