How to recursively read in .TAB files that are located in a hierarchy, then save as one matlab table?

Hello, I want to create one GIANT matlab table (if possible) that contains 19 columns of chronological data:

I have the data sorted like this:

where each folder 001_031_JAN, 032_059_FEB, etc contains daily files:

The .LBL files are of no importance to me. All the data I want is located in the .TAB files.

I want to read in ALL of the .TAB files from each month and each year. There is ~25 GB worth of data.

Each of the .TAB files are organized into 19 tab delimited columns which I have specified as varNames below.

I want to create one GIANT matlab table (if possible) that contains 19 columns of data from the time period 2011 to 2015.

This is my attempt to go through the MAG_tab_and_lbl_files folder and subfolders to load all the data into one table:

Loc = dir('/Users/chle4971/Google Drive/LASP/MERCURY/MESSENGER_instruments/MAG_tab_and_lbl_files/')'
varNames = {'year','DOY', 'hour', 'min','sec','time_tag','navg', ...
     'Bx_sensor','By_sensor','Bz_sensor','DBx_sensor','DBy_sensor',...
     'DBz_sensor','Bx_spacecraft','By_spacecraft','Bz_spacecraft',...
     'DBx_spacecraft','DBy_spacecraft','DBz_spacecraft'};
 varTypes = {'int64','int64','double','double','double','double',...
     'double','double','double','double','double','double','double',...
     'double','double','double','double','double','double'};
 delimiter='\t';
 datastartline=1;
 opts = delimitedTextImportOptions('VariableNames',varNames,'VariableTypes',...
     varTypes,'Delimiter',delimiter,'DataLines',datastartline);
ttds = tabularTextDatastore(Loc,opts,'IncludeSubfolders','0','FileExtensions','.TAB')

That's all I got. I don't know how to proceed. Also any suggestions on a better way to store such a large dataset is much appreciated.

Thanks,

-Christy

[Siddharth Bhutiya]

For such large datasets, datastore would be the way to go. You could use Tall arrays with it as well.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

When you tried using it, were you getting any errors ?