Clear Filters
Clear Filters

Tall arrays are not allowed to contain data of type struct

5 views (last 30 days)
I am having trouble with tall arrays in the following workflow,
  • Process data in a loop. In each iteration, generate structure containing results and save structure in a *.mat file
  • Read all *.mat files into a datastore
  • Turn datastore into a tall array. Gives the error "Tall arrays are not allowed to contain data of type struct"
  • Turn tall array into a timetable
Here's an example, which I generated by modifying Edric Ellis's answer here. Can you help me eliminate the error?
%% Setup
nRowLimit = 100; % Number of records in each file
nCalculation = 20; % Number of calculation iterations and also mat files
tStart = datetime(2013, 11, 1, 8, 0, 0); % For later use in generating datetime array
%% Prepare to save matlab files
iFileToSave = 0;
dirSave = 'c:\temp';
%%
for iCalc = 1:nCalculation % Loop through iterations of the calculation
clear StrucResults; % Structure to hold results of calculations
% Generate structure containing pretend calculation results
for iRow = 1:nRowLimit
% Generate example datetime to fill a datetime array
StrucResults.date(iRow) = tStart + 1000*iFileToSave + iRow;
% Generate example character to fill a cell array
StrucResults.label{iRow} = char(round(rand(1)*100));
% Generate example value to fill a numerical matrix
StrucResults.white(iRow, :) = rand(1, 5);
end
%% Save structure containing results to small mat file
iFileToSave = iFileToSave + 1;
% Make a distinct file name
fileSave = fullfile(dirSave, sprintf('FileContainingStruct_%05d.mat', iFileToSave));
save(fileSave, 'StrucResults');
end
%% Read the data back in as a tall array
% First create a datastore from all the mat files
fds = fileDatastore(fullfile(dirSave, '*.mat'), ... % Files to load
'ReadFcn', @(x)struct2table(load(x)), ... % Try to convert structure to table to avoid error in next code line
'UniformRead', true);
% Read into a tall array
tallData = tall(fds); % Throws error "Tall arrays are not allowed to contain data of type struct"
% Change to a tall timetable
tallTimeTable = table2timetable(tallData);
It's fine to eliminate the structure in favor of storing the results in individual variables, but I couldn't get that to work either.

Accepted Answer

Edric Ellis
Edric Ellis on 23 Apr 2021
Your code was so nearly working. I made two tiny changes, marked with %###:
%% Setup
nRowLimit = 100; % Number of records in each file
nCalculation = 20; % Number of calculation iterations and also mat files
tStart = datetime(2013, 11, 1, 8, 0, 0); % For later use in generating datetime array
%% Prepare to save matlab files
iFileToSave = 0;
dirSave = fullfile(tempdir(), '809910');
if ~exist(dirSave, 'dir')
mkdir(dirSave);
end
%%
for iCalc = 1:nCalculation % Loop through iterations of the calculation
clear StrucResults; % Structure to hold results of calculations
% Generate structure containing pretend calculation results
%### Ensure vector fields are columns
for iRow = 1:nRowLimit
% Generate example datetime to fill a datetime array
StrucResults.date(iRow, 1) = tStart + 1000*iFileToSave + iRow;
% Generate example character to fill a cell array
StrucResults.label{iRow, 1} = char(round(rand(1)*100));
% Generate example value to fill a numerical matrix
StrucResults.white(iRow, :) = rand(1, 5);
end
%% Save structure containing results to small mat file
iFileToSave = iFileToSave + 1;
% Make a distinct file name
fileSave = fullfile(dirSave, sprintf('FileContainingStruct_%05d.mat', iFileToSave));
%### Use "save -struct" to ensure the fields of StrucResults are saved into the file directly.
save(fileSave, '-struct', 'StrucResults');
end
%% Read the data back in as a tall array
% First create a datastore from all the mat files
fds = fileDatastore(fullfile(dirSave, '*.mat'), ... % Files to load
'ReadFcn', @(x)struct2table(load(x)), ... % Try to convert structure to table to avoid error in next code line
'UniformRead', true);
% Read into a tall array
tallData = tall(fds); % Throws error "Tall arrays are not allowed to contain data of type struct"
% Change to a tall timetable
tallTimeTable = table2timetable(tallData);
% Check the results
head(tallTimeTable)
ans = 8×2 tall timetable date label white ____________________ _____ ________________________________________________________ 02-Nov-2013 08:00:00 {''} 0.24734 0.82558 0.41452 0.3178 0.41834 03-Nov-2013 08:00:00 {'B'} 0.49954 0.61133 0.76577 0.67823 0.50137 04-Nov-2013 08:00:00 {''} 0.099728 0.28918 0.42864 0.62252 0.61956 05-Nov-2013 08:00:00 {'H'} 0.67335 0.32173 0.25268 0.057868 0.17846 06-Nov-2013 08:00:00 {''} 0.82981 0.059187 0.34086 0.31046 0.25538 07-Nov-2013 08:00:00 {'@'} 0.53214 0.78883 0.30166 0.77258 0.68763 08-Nov-2013 08:00:00 {'Y'} 0.37458 0.19037 0.75301 0.62862 0.85332 09-Nov-2013 08:00:00 {'8'} 0.27491 0.22561 0.12215 0.46316 0.66067
The first change was simply to ensure that the vectors in the struct were columns.
The second more important change was to use save -struct to get rid of the extra layer of "struct" in the MAT file.

More Answers (0)

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!