Help with parfor progress bar using data queue

Hello. I am trying to implement a progress par for a parfor loop, following the example shown at

https://www.mathworks.com/help/parallel-computing/parallel.pool.dataqueue.html#bvo5uvj-1

I am using a script instead of a function as shown in the example, but I am unable to adapt the code in a way that works.

My code:

clear
D = parallel.pool.DataQueue;
h = waitbar(0, 'Please wait ...');
afterEach(D, @nUpdateWaitbar);

num_files = 1000;


parfor i = 1:num_files
    % do stuff
    send(D, [i num_files]);
end


function p = nUpdateWaitbar(input)
    p = input(1)/input(2)
    waitbar(p, h);
end

Executing this gives

Warning: Unrecognized function or variable 'h'. 

which I assume is because the function nUpdateWaitbar doesnt know what 'h' is

but if I try to pass the waitbar handle 'h' into the dataqueue, I get another error:

Cannot convert double value 1000 to a handle

Can you please point me towards what I am doing wrong?

the parfor loop runs fine without the waitbar code so I dont think that is the issue

thanks!

[Mario Malic]

The example you took the code from works, because functions are nested and nUpdateWaitbbar has access to the variables. Best is to copy the code from example and understand how does it work.

[HpW]

Thanks. I dont understand how to pass in the required variables without nested functions. Can you help?

[HpW]

To clarify- the way i have it now passes in the values for i and num_files.

[Mario Malic]

Try this

function p = nUpdateWaitbar(input)
    persistent h
    if isempty(h)
        h = findall(0, 'type', 'figure', 'Tag', 'TMWWaitbar'); 
    end
    p = input(1)/input(2)
    waitbar(p, h);
end

[HpW]

thanks!

I was able to make it work this way:

function nUpdateWaitbar(input)

        h = findall(0, 'type', 'figure', 'Tag', 'TMWWaitbar'); 
        p = input(1)/input(2)
       
        waitbar(p, h);
        
    end

Answer 1

[Edric Ellis]

The problem here is that the documentation example is using a nested function, which is able to acces variables in the containing workspace. Your function definition inside a script is a local function, which cannot automatically access the variables in the containing workspace. An alternative fix is to use an anonymous function handle to "bind" in the value of h.

clear
D = parallel.pool.DataQueue;
h = waitbar(0, 'Please wait ...');
% Note anonymous function captures the value of "h" to pass
% in to "nUpdateWaitbar"
afterEach(D, @(data) nUpdateWaitbar(data, h));
num_files = 1000;
parfor i = 1:num_files
    % do stuff
    send(D, [i num_files]);
end
function p = nUpdateWaitbar(input, h)
    p = input(1)/input(2)
    waitbar(p, h);
end

This "works", but it isn't right. If you run it you'll see the value flickering about because the parfor loop iterations are not processed in order. So, you need a different approach for nUpdateWaitbar. I would essentially make nUpdateWaitbar contain persistent data and have an "initialisation" mode. Like this:

clear
D = parallel.pool.DataQueue;
h = waitbar(0, 'Please wait ...');
num_files = 1000;
% Dummy call to nUpdateWaitbar to initialise
nUpdateWaitbar(num_files, h);

% Go back to simply calling nUpdateWaitbar with the data
afterEach(D, @nUpdateWaitbar);

parfor i = 1:num_files
    % do stuff
    % Note we send only an "increment" for the waitbar.
    send(D, 1);
end
function p = nUpdateWaitbar(data, h)
persistent TOTAL COUNT H
if nargin == 2
    % initialisation mode
    H = h;
    TOTAL = data;
    COUNT = 0;
else
    % afterEach call, increment COUNT
    COUNT = 1 + COUNT;
    p = COUNT / TOTAL;
    waitbar(p, H);
end
end

[HpW]

thanks! this worked great. Prior to your response I had put something together using global variables for the count variable, so that the function would just increment this global variable. Is there a benefit to using a persistent variable vs a global variable in this case?

thanks!!

[Edric Ellis]

persistent is more tightly-scoped than global - so you don't have to worry about whether other bits of code might interfere. (In general, it's good practice to try and keep your data as tightly-scoped as possible). In my 2nd example above, nUpdateWaitbar is a "local" function inside the script - this means it is only accessible to code within that script file.

[HpW]

Thanks so much!!