Doubts about cross-validation

4 views (last 30 days)
I'm trying to test performance of a KNN and a svm (cecoc) classifier for a multiclass problem.
For the knn model I do it by building mdlknn, which is a cross validated model (leave one out) and then using kfoldLoss function
mdlknn = fitcknn(X,labels, 'NumNeighbors', ...
k, 'Distance',@distKNN, 'Leaveout','on',...
'HyperparameterOptimizationOptions','UseParallel');
perf = 1-kfoldLoss(mdlknn);
Is this correct? My doubts regards the cross-bvalidation:
Using option ('Leaveout','on') in fitcknn is the same as not using it and then calling crossval on the trained model?
mdlknnNoCrossVal = fitcknn(X,labels, 'NumNeighbors', ...
k, 'Distance',@distKNN, 'HyperparameterOptimizationOptions','UseParallel');
mdlknn = crossval(mdlknnNoCrossVal, 'leaveout','on')
perf = 1-kfoldLoss(mdlknn);
Next, since in knn classifier I'm using my distance, I would like to do the same with the cecoc. How can I do that?
At this stage, the code I use for building the cecoc doesn't use it. how can I set it?
t = templateSVM('Standardize',true,'KernelFunction','rbf');
mdlsvm = fitcecoc(X,labels, 'Leaveout','on',...
'HyperparameterOptimizationOptions','UseParallel',...
'coding', 'ternarycomplete','Learners',t);
perf = 1-kfoldLoss(mdlsvm);
And again: Using option ('Leaveout','on') in fitcecoc is the same as not using it and then calling crossval on the trained model, right?
THANKS A LOT

Accepted Answer

Jyothis Gireesh
Jyothis Gireesh on 22 Aug 2019
I am assuming that you want to have some clarification regarding the cross-validation syntax in KNN and SVM
  • KNN Classifier
mdlknn = fitcknn(X,labels, 'NumNeighbors', ...
k, 'Distance',@distKNN, 'Leaveout','on',...
'HyperparameterOptimizationOptions','UseParallel');
mdlknnNoCrossVal = fitcknn(X,labels, 'NumNeighbors', ...
k, 'Distance',@distKNN, 'HyperparameterOptimizationOptions','UseParallel');
mdlknn = crossval(mdlknnNoCrossVal, 'leaveout','on')
These two syntaxes may be equivalent for the creation of a cross-validated KNN model. “mdlknn" is a "ClassificationPartitionModel" classifier. "mdlknnNoCrossVal” creates a "ClassificationKNN" classifier which is cross validated using the “crossval” function.
  • ECOC classifier
In the case of ECOC classifier you may use the templateKNN as the learner. It provides the feature to customize the distance metric using the ‘Distance’ argument. The distance metric can be passed as a function handle like that of the KNN classifier syntax.
And the two syntaxes are valid for the process of cross validation in the case of ECOC also.
  1 Comment
Elena Casiraghi
Elena Casiraghi on 22 Aug 2019
Thanks a lot!
this clarified what I was asking!

Sign in to comment.

More Answers (0)

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!