- You can try to use tall arrays. These help to work with arrays that are too large to fit in the memory. You can find more information on them through the following MATLAB documentation: https://in.mathworks.com/help/releases/R2021b/matlab/import_export/tall-arrays.html
- If the issue persists, you can untick the checkbox ‘Limit the maximum array size to a percentage of RAM’. To do this, you can go to: Preferences -> MATLAB -> Workspace. This allows you to use virtual memory in addition to the machine’s RAM. You can refer to the following documentation for more information about this: https://www.mathworks.com/help/releases/R2021b/matlab/matlab_env/set-workspace-and-variable-preferences.html
- You can also refer to the following documentation for more information on resolving "Out of Memory" errors, if these occur at a later point: https://www.mathworks.com/help/releases/R2021b/matlab/matlab_prog/resolving-out-of-memory-errors.html
Array too large for wdenoise?
2 views (last 30 days)
Show older comments
Hello,
I am struggling with denoising a large dataset. I have a parquet file that has 8712 columns and each column has 800000 rows (conceptually it is signal data with each column representing a signal). Also, each column has a variable name, "0", "1", "2", etc. Since it is so large, I thought it would be best to use a datastore. Here it what I have so far:
ds=parquetDatastore("file.parquet"); %read data
isPartitionable(ds); %checking if I can split the data
ds.SelectedVariableNames=["0","1","2"]; %for practice/proof of concept I just chose the first three columns
ds2=read(ds); %get the first three columns and name ds2
%I want to denoise next, which requires an array and type double
ds2=table2array(ds2); %array
ds2T=transpose(ds2); %transpose
ds2Tdouble=double(ds2T); %double; note data was originally int8
ds2Tdouble_den=wdenoise(ds2Tdouble); %attempt to denoise
%when running the last line, I get the error:
%%%
Error using repmat
Requested 31x640000000000 (147819.5GB) array exceeds maximum array size preference (15.9GB). This might cause MATLAB to become
unresponsive.
Error in wavelet.internal.weightfromdata (line 163)
deltaweight = repmat(temp_deltaweight,1,n);
Error in wavelet.internal.ebayesthresh (line 80)
weight = wavelet.internal.weightfromdata(x,30,transformtype);
Error in wavelet.internal.ebayesdenoise (line 32)
xdecdenoised.cd{lev} = wavelet.internal.ebayesthresh(...
Error in wdenoise (line 273)
wavelet.internal.ebayesdenoise(...
I am guessing this a problem with the size of my array. Any advice on how I can work around this?
Thank you
0 Comments
Answers (1)
arushi
on 4 Jul 2024
Hi Reid,
I understand that you are trying to denoise a signal and are getting an error because the size of the array that you are trying to use is exceeding your MATLAB’s memory preference.
Here are a few ways this issue can be resolved:
Hope this helps.
0 Comments
See Also
Categories
Find more on Large Files and Big Data in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!