Parallel Toolbox + net.cdf: net.cdf function works with for, but not with parfor?

Hello again. I have a question on how to use Parallel Toolbox to read net.cdf file using netcdf functions. Structure-wise, here is the code.

% assume that file_ID and loc_str already exist.
% file_ID = output from netcdf.open
% loc_str = char for excel filename
vars_taken = 47; % so the data have 47 variables. Dimensions not count.
parfor var_count = 3:vars_taken+2 % variables are listed in 3rd to 49th.
    % determine what variable is it?
    [varname,~,~,~] = netcdf.inqVar(file_id,var_count);
    % get variable ID
    vars_id = netcdf.inqVarID(file_id,varname);

    % % get attribute name and value of that variable
    vars_attname = netcdf.inqAttName(file_id,vars_id,1); % variable name
    vars_attval = netcdf.getAtt(file_id,vars_id,vars_attname);
    unit_attname = netcdf.inqAttName(file_id,vars_id,2); % units
    unit_attval = netcdf.getAtt(file_id,vars_id,unit_attname);
    
    % write on excel. This has resolved on my previous question
    sheet_name = 'Values';

    writematrix(time_data,[loc_str varname '.xlsx'],"Sheet",sheet_name,"Range",'B1:Y1');
    writematrix(vars_attval,[loc_str varname '.xlsx'],"Sheet",sheet_name,"Range",'AA2');
    writematrix(unit_attval,[loc_str varname '.xlsx'],"Sheet",sheet_name,"Range",'AA3');

    disp([varname ' done']); % show if it's done
end
netcdf.close(file_id);

This produce the following error

---

Error using netcdf.inqVar (line 26)

The NetCDF library encountered an error during execution of 'inqVar' function - 'Not a valid ID (NC_EBADID)'.

Error in Untitled_parfor (line 95)

parfor var_count = 3:vars_taken+2

---

But if I replace parfor with for, no error, disp are showing properly, and I get Excel files as intended.

I need this parfor so when I read the actual data it can finish faster (approx. 6 hours for a month data without parallel). How can I do it? Thank you.

Answer 1

[Walter Roberson]

matlab.internal.imagesci.netcdflib() creates internal resources that are not automatically copied to workers.

See https://www.mathworks.com/help/parallel-computing/parallel.pool.constant.html#buzshy6-1 for an example of creating a parpool constant by executing a function on each worker -- in this case you would do the netcdf.open on each worker.

[Muhammad Robith]

Thank you for your answer. I will try to modify the code on that example for my need.

Maybe I should consider different approach for my problem, just in case.

[Daniele Massaro]

Hi, have you managed? I'm facing the same issue.

[Walter Roberson]

You need to do the netcdf open on each worker, either using the parpool constant example or using parfevalOnAll()

You will not be able to just do the open once on the client

[Daniele Massaro]

Thanks for your reply. How does the parpool constant work with a function like netcdf.open?

I'm struggling to replicate that example for ntcdf. I attach the frame of my code.

dirData = dir('*.cdf.dat');
nfile=size(dirData,1);

% First open all files (SERIAL)
ncid_v = single(zeros(nfile,1));
for i=1:nfile
    filename=dirData(i).name;
    ncid_v(i) = netcdf.open(filename,'NC_NOWRITE');
end

% 
parfor (s=1:100,10)
    
    for indsnp=1:nfile
        varid = netcdf.inqVarID(ncid_v(indsnp),'r');
        r = netcdf.getVar(ncid_v(indsnp),varid);
        
        ...
    end
end

[Walter Roberson]

Try this:

dirData = dir('*.cdf.dat');
nfile=size(dirData,1);

% First open all files (SERIAL)
ncid_v = single(zeros(nfile,1));
for i=1:nfile
    filename=dirData(i).name;
    ncid_v{i} = parallel.pool.constant(@() netcdf.open(filename,'NC_NOWRITE'), @netcdf.close);
end

% 
parfor (s=1:100,10)
    
    for indsnp=1:nfile
        ncid = ncid_v{indsnp}.Value;
        varid = netcdf.inqVarID(ncid, 'r');
        r = netcdf.getVar(ncid, varid);
        
        ...
    end
end

[Daniele Massaro]

Dear Walter,

Sorry for my late reply. Eventually your suggestion worked out, but with this code I do not manage to change the number of workers (either increase or reduce it). For example, with the above code it doesn't use 10, but 12. DO you have any ideas?

Thanks,

Daniele

[Walter Roberson]

The ,10 in the parfor statement is the maximum number of workers that the statements will be dispatched to.

The documentation does not indicate that it controls the size of the pool in any way. If a pool already exists with 12 workers then it would send to 10 of the 12 workers, leaving the other two workers existing but idle.