How to manually construct or modify a cross-validation object in MATLAB?

Short question:

I want your help to manually construct or modify Matlab cross-validation (CV) object created using function cvpartition when it's known which observations belong to training and which to validation partitions.

Extended question:

I have a kind of specific dataset in which several observations (each represented by a separate row) belong to the same object of investigation (e.g. measurements of the same object were performed several times). Each object (and it's observation) belongs to a certain class (e.g. yellow object, green object, etc.) as well. The problem is this: it's needed a CV object in which all observations of the same object belong to the same (either training or validation) partition as a block of data (unfortunately I'm not sure how to implement this) and the data would be distributed to partitions in stratified way (luckily, this is implemented in function cvpartition). I tried to access indices of CV object (cvo.indices) but they are locked. Here cvo is:

cvType = 'kfold';
nPartitions = 5;
cvo = cvpartition(my_classes,cvType,nPartitions);

Your insights and ideas how to construct or modify cvo manually are welcomed.

[Jan]

[MOVED from flags] Flagged as Unclear by James Ratti on 19 Oct 2018.

This solution no longer works - was there a concurrent patch that provided a workaround?

@James: This is the question. Which solution do you mean?

[Rik]

Comment posted as flag by patrck rich:

the solution is no longer working in recent versions

That is always a risk when trying to modify internal files. However, I ran the lines below in the online interface (I cleared the output, as I don't expect Mathworks would be happy with me sharing their files..)

%type(which('cvpartition'))
OS={'Windows','macOS','Linux'};fprintf('This is version %s, running on %s.',version,OS{[ispc ismac isunix&&~ismac]})

This is version 9.10.0.1613233 (R2021a), running on Linux.

As far as I can judge from the code I can read, I don't see a reason why it wouldn't still work, but feel free to post a comment explaining which exact steps you took and which lines you modified.

Answer 1

[Vilmantas Gegzna]

I found, that I can modify the cross-validation object (CVO) which class is cvpartition by modifying several lines in cvpartition.m file from:

properties(GetAccess = 'private', SetAccess = 'private')

to:

properties(GetAccess = 'public', SetAccess = 'public')

This allowed me to modify indices of CVO manually.

I wonder if there are any other way to set indices of CVO without changing the lines in cvpartition.m? This is the case when I want to run Matlab on my campus computer.

[RZM]

I cannot change the properties, it pops up an error message that Access is denied!

[Sergio Gutierrez]

I have done the same modification as Albert Sama said in the below comment. You require administrative rights to make changes. I used Notepad++ in administrator mode.

Answer 2

[Jo]

Hi,

It seems that one year later your solution is not working anymore, maybe they changed the function.

Indeed, I tried to modify as you suggested "properties(GetAccess = 'public', SetAccess = 'public')", but it only allows me to access to the number of observations and not the indices. Indices are created by "methods" defined in cvpartition.m (test, training). And I do not understand how to change these to choose my own indices.

Maybe if you have progessed in all that since you first posted your question.... I would be really greatful to have some help in there.

Using random crossvalidation is of none interest for me, I need to choose the block myself to have a more robust solution. Thanks to anyone that can help me!!!

[Albert Sama]

I have done a similar modification on the source code that works on MATLAB R2017b.

Basically, the file cvpartitionImpl.m (at ...\toolbox\stats\stats\+internal\+stats) has to be modified as the first answer specifies, making indices public:

    properties(GetAccess = public, SetAccess = public)
        indices = [];
        Group = [];
        holdoutT = [];
    end

This way, indices are updateable, for instance:

cvp = cvpartition(500,'KFold',9);
cvp.Impl.indices=cvp.Impl.indices(end:-1:1);

[Evelyn Tang]

This is wonderful and just what I need, thank you so much! Works perfectly :)

[Alex Ryabov]

This solution works for me also, maltab 2023a.

In windows I had to run Notepad as admin to be able to edit the file

Answer 3

[Roberto Herrera-Lara]

try to do this https://chrisjmccormick.wordpress.com/2013/07/31/k-fold-cross-validation-with-matlab-code/

[Vilmantas Gegzna]

Thank you, Roberto, your link provided an extremely good description of k-fold cross-validation and updated my knowledge. Unfortunately, I was searching for a bit different thing.

Answer 4

[John Smith]

Two small additions:

1.You need to update the toolbox cache https://www.mathworks.com/help/matlab/matlab_env/toolbox-path-caching-in-the-matlab-program.html

2.Restart Matlab after that