Processing Tall Arrays Taking Too Long
7 views (last 30 days)
Show older comments
I have a tall array with about 1.7 billion rows of data and 14 columns. I want to be able to process this data in the same way that several examples (with airline data) do it. I am just trying to extract one column and find the mean. My code is something like:
ds = datastore('some-file.csv');
tt = tall(ds); %Mx14 tall table (M should be about 1.7 billion)
a = tt.V; %Mx1 tall double %(M should be the same as above)
m = mean(a); %One integer
gather_m = gather(m);
The gather step is taking way too much time. I haven't seen it complete at all. In the examples I have seen, this step is shown to be completed in a few seconds. Eventually, I want to be able to make calculations and plots, but I want to start by making this simple step work first. Can anyone recognize the problem and recommend a solution? I have parallel pool turned on and there are two workers.
Thank you very much.
2 Comments
Walter Roberson
on 1 Nov 2017
I had not realized that tall arrays used parallel if available, but I see that they do; https://www.mathworks.com/help/distcomp/run-tall-arrays-on-a-parallel-pool.html
Answers (1)
Kojiro Saito
on 2 Nov 2017
It may speed up by configuring read size of datastore. You can know the default read size by
ds.ReadSize
This is the data size which MATLAB reads from the file at one time. You can set higher size than your default and this will reduce file I/O. Please add ds.ReadSize setting, for example,
ds = datastore('some-file.csv');
ds.ReadSize = 100000; % Or higher
tt = tall(ds); %Mx14 tall table (M should be about 1.7 billion)
a = tt.V; Mx1 tall double %(M should be the same as above)
m = mean(a); %One integer
gather_m = gather(m);
Hope this help.
4 Comments
Dominique Ingala
on 7 Apr 2021
I came from Python and R... Same struggle. So I'm now trying Matlab... If you managed to fix this, please share some secrets. Thanks.
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!