How to best store and work with timeseries datasets in Matlab?

I am currently using a trial version of Matlab and I have matlab data files that were collected for 1.5 years in separate files. Within these files the data is collected almost every second, hence there would be sometimes 700 to 900 thousand rows in each of these files. I would like to know how best I can combine these files and if I will really need to create a database to store the raw data? I would like to then visualise this data over the 1.5 year period so I do need to store them all in one file. I am not familar with setting a database so will welcome suggestions on what I can do and which tutorial I can use that can help me with setting up the db or combining the files.

Thank you.

Answer 1

[Peter Perkins]

MSani, you probably want to take a look at datastores, and tall (or not tall) timetables.

[MSani]

Thanks for the suggestion. Ideally I would want to work with this data using Python but all the files are stored as .mat files now so I am really hoping to find a good workaround to combine them so that I can load them on Python

Answer 2

[Aylin]

Hi MSani, as Peter suggested you could use datastore to make it easier to work with large files and datasets that may not fit in memory.

In particular, FileDatastore can be used to load a collection of MAT files using the load function. Here's an example that loads and reads a MAT file from the MATLAB demos directory:

files = fullfile(matlabroot, "toolbox", "matlab", "demos", "trimesh" + ["2d", "3d"] + ".mat");

% A function that reads MAT file data and converts it to a table
readFcn = @(filename) struct2table(load(filename, 'x', 'y'));

% Construct a FileDatastore to read all the MAT files with the custom reading function
fds = fileDatastore(files, "ReadFcn", readFcn, "UniformRead", true);

If your data fits in memory, you could read all the MAT files at once and plot it using the stackedplot function:

data = readall(fds); % Reads all the data into memory

% Convert to the 'timetable' datatype and plot the time series data
ttData = table2timetable(data, 'SampleRate', 0.1); % Sample rate is 0.1 Hz

stackedplot(ttData)

This generates a plot that looks like this:

If you have data that doesn't fit into memory, you could construct a tall timetable instead. Here's an example of this workflow on the MATLAB doc.