Database - Large Structure of arrays :: BigData

Hello Everyone,

I would like to ask a suggestion to find an efficient way to manage a big amount of data (>2GB) organised in a structure of array and saved on .mat file.

I am working on a big structure of arrays that reppresent a collection of components data (e.g. Database.ElectricMotor.ComponentName.EfficiencyMap.400V.25degC.[large matrix]).

Which is the best way to save and load a big collection of data, keeping them orgaised and easy accessible on matlab?

I have found just couple 'rough' way to speed-up the code and lower memory allocation effort:

• split the large mat file into smaller ones arranged inside a 'database' repository
• use 'matfile' function and associate the mat file to an object

(not very efficient either way).

Thanks in advance!

[Ive J]

Have you tried tall arrays? They're one of the most memory friendly way of working with big data in MATLAB!

[Smoone]

To be honest I'm not very familiar with tall arrays, but I'm not sure I can re-arrange my data, that are not just a list of infos, but matrices, arrays, strings, ... on tall arrays. However, thanks for your help, I will try to gather more informations.

Answer 1

[Ayush Modi]

Hi Smoone,

As per my understanding, you have a large collection of structure of array, and you would like to save and load them efficiently. You can achieve this using “matfile” function. Here is an example to demonstrate how you can accomplish this:

matObj = matfile(filename)

You can use “matfile” object to access and change variables in a MAT-file without loading the file into memory. You can even load or save parts of variables. Partial loading and saving of variables using a MAT-file object requires less memory than the load and save commands.

Please refer to the following MathWorks documentation for more information on “matfile” function and various distributions:

https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html

I hope this resolves the issue you were facing.

[Smoone]

It looks good at first sight, I will restructure the code in order to test the solution proposed.

Thank you Ayush Modi.