How can I use repeated, k-fold cross-validation results with rocmetrics?

I have 10-repeat 5-fold cross-validation scores and labels for a model that I'm trying to efficiently plot ROC curves for usine rocmetrics. When I run the line

robj = rocmetrics(target, prediction, 1);

I get the error

Error using rocmetrics>validateScoresLabelsAndWeights 
The cell array of cross-validated scores must be a vector.

Each cell in target and prediction are double arrays of shapes 54x1 or 55x1. The shapes match cell to cell between both. I'm confused by this error because it's clear that the cell array of my scores(predictions) are vectors. I think the issue is the repeated cross-validation. How can I format my target and prediction in order to use rocmetrics with my results?

Answer 1

[the cyclist]

Note the following line from the rocmetrics documentation:

"For cross-validated data, you must specify Labels, Scores, and Weights as cell arrays with the same number of elements. rocmetrics treats an element in the cell arrays as data from one cross-validation fold and computes pointwise confidence intervals for the performance metrics. The length of Labels{i} and the number of rows in Scores{i} must be equal."

You need to supply the fold weights.

Alternatively, you could loop over the folds to see ROC metrics on each fold, or decide prior to calling rocmetrics how you want to combine the folds into a single prediction.

[Thomas Kirsh]

Adding weights like

robj = rocmetrics(target, prediction, 1, 'Weights', weights);

does not solve this issue and returns the same error. I made weights to be a 10x5 cell arrays where each cell is ones(length(target{i}), 1)

I could loop over the folds but I wanted to try this approach first because it seemed like this object does that more efficiently.

[the cyclist]

Can you upload the data? You can use the paper clip icon in the INSERT section of the toolbar.

[Thomas Kirsh]

Yes, I added it to my question.

[the cyclist]

I'm not going to be able to look at this again for a while, but I would take a close look at the Compute Confidence Intervals with Cross-Validate Input Data, and see if the layout of your data mimics that. (Looks like I was mistaken about requiring Weights?)

One thing that is puzzling me about your data is that your target is not the same in every cell element. That's the true label, and I wouldn't expect that to change between folds.

[the cyclist]

It looks like for the cross-validated case, the cell array needs to be a vector (not arbritrary MxN). Here, I reshape your cell arrays into vectors, and all seems to be well.

% Load the data
load("example_rocmetrics.mat","target","prediction")

% Reshape the cell arrays into vectors
target = reshape(target,50,1);
prediction = reshape(prediction,50,1);

% Define the ROC object
robj = rocmetrics(target,prediction,1);

% Plot
plot(robj,ShowConfidenceIntervals=true)

[Thomas Kirsh]

Thank you, that's a good workaround! My only concern is thinking about if this gives an accurate mean ROC for my experiment. Wouldn't it make more sense to concatenate the folds first and then reshape?

[the cyclist]

I have to admit that I don't really have experience specifically with repeated k-fold cross-validation, so I don't know what is conventional in terms of combining information from repeats and folds. My impression is that one treats it as M*k results, which is what my code is doing. I don't think concatenating folds is typically done, because that would look like you had a dataset that was k times larger.

Also ... I hope the data you posted isn't your real data. The model performance is no better than random.