Concatenation is missing values

I am using the command:

doubArray = cat(1,cellStruct{:,3});

to extract an array of numeric values from a cell structure of mixed variable types. All of the variables in column 3 of the cell structure are of data type “double“.

The resulting array is missing 33 of the 116397 expected entires. The missing 33 entries are from a variety of locations within the cell structure and there does not appear to be any rhyme or reason behind which entries are missing.

Has anyone encountered this problem before and know of a solution?

[Stephen23]

@Charles D'Onofrio: please upload the cell array in a .mat file.

[Guillaume]

The resulting array is not missing anything that was present in the cells of cellStruct(:, 3). If there was a bug there, you can be sure that it would have been found by now.

So now, we need to understand why you think it does. Can you attach a mat file with that cellStruct cell array?

Personally, I would have written

doubArray = vertcat(cellStruct{:, 3});  %instead of cat(1, ...), produces the same result

so I don't even have to think which way the cells are concatenated. Alternatively, since you're vertically concatenating the rows of a column cell array, you could just do:

doubArray  = cell2mat(cellStruct(:, 3));

Neither of these expressions is going to produce a different result from what you already have.

[Charles D'Onofrio]

Stephen, unfortunately, due to the nature of the work I am unable to share the cell array. Are there any particular details I could provide in its place?

[Stephen23]

@Charles D'Onofrio: create some fake data that demonstrates this problem, and upload that.

[Guillaume]

unfortunately, due to the nature of the work I am unable to share the cell array

Well, that's going to be difficult for us to understand the problem. Note that we only need column 3 of the cell array. Since you say that this column only contain numbers, can you just share that?

As a last resort, replace all the numbers by something else. Since the code you've posted does not depend on the actual values, it shouldn't affect the behaviour:

%create a new cell array the same size as cellStruct(:, 3) where each cell contains
%the same type and size of variable but as integers 1 to the number of elements in the variable
testcell = cellfun(@(c) cast(reshape(1:numel(c), size(c)), class(c)), cellStruct(:, 3), 'UniformOutput', false)

[Jan]

"The resulting array is missing 33 of the 116397 expected entires."

The cat command does not ignore values. Therefore I assume that the expectation of 116397 elements is the actual problem. Why do you assume this amount of data? How do you determine, which elements are missing? What does this reply:

S = sum(cellfun('prodofsize', cellStruct(:,3)))

This is the number of elements of all arrays contained in the 3rd column of cellStruct - by the way: isn't the part "Struct" in the name confusing here?

[Charles D'Onofrio]

I have attached a representative cell structure and double array.

Answer 1

[Stephen23]

33 of the cells in the third column contain empty arrays:

>> find(cellfun('isempty',cellStruct(:,3)))
ans =
   21968
   21969
   27848
   35808
   43424
   52648
   52649
   52650
   52651
   52652
   52653
   52654
   52655
   60284
   60285
   60286
   60287
   60288
   60289
   60290
   60291
   60292
   60293
   60294
   60295
   60296
   60297
   60298
   60299
   65762
   65763
   65839
   65840
  

While the rest of the cells in the third column apparently have one row:

>> find(cellfun('size',cellStruct(:,3),1)>1)
ans = []

In any case, the total number of rows in all of the cells in the third colum is:

>> sum(cellfun('size',cellStruct(:,3),1))
ans =  116364

And that is the exact number of rows that your numeric array contains:

>> size(doubArray,1)
ans =  116364

So far nothing suggests that any data has gone "missing" when concatenating those cell contents.

[Charles D'Onofrio]

Stephen and all,

Thank you for your help. I did not realize the concatenate function ignored empty cells; quite frankly I never even expected empty cells could exist in my cellStruct due to the way data is entered. Again, thanks.

[Stephen23]

"I did not realize the concatenate function ignored empty cells"

It doesn't ignore them, they are concatenated along with everything else. You can show this quite easily using an empty array with a different number of columns, which throws an error (thus showing that the empty arrays are included in the concatenation):

>> vertcat([1,2,3],nan(0,4),[4,5,6])
Error using vertcat
Dimensions of matrices being concatenated are not consistent.

And of course when the empty array has the same number of columns it is totally included in the output (all zero rows of it!):

>> vertcat([1,2,3],nan(0,3),[4,5,6])
ans =
   1   2   3
   4   5   6

Nothing "missing" and nothing "ignored".

"quite frankly I never even expected empty cells could exist in my cellStruct due to the way data is entered"

It is more reliable to check, than to just rely on what you believe is happening.

[Tjeerd]

I have a similar problem in that NaNs are excluded in the concatenated array. Is there a way to include them, such that the answer in your example would be:

1 2 3

nan nan nan

4 5 6

[Stephen23]

"I have a similar problem in that NaNs are excluded in the concatenated array."

None of the concatenation operators "exclude" NaN values. Lets try them all right now:

[1:3;nan(1,3);4:6]

ans = 3×3

1 2 3 NaN NaN NaN 4 5 6

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

vertcat(1:3,nan(1,3),4:6)

ans = 3×3

1 2 3 NaN NaN NaN 4 5 6

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

horzcat(1:3,nan(1,3),4:6)

ans = 1×9

1 2 3 NaN NaN NaN 4 5 6

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

I do not see any values being "excluded".

"Is there a way to include them.."

Yes, you include the NaNs in the arrays being concatenated.