Is there a way to automatically calculate the batch mean standard error of the training perplexity of fitlda?

It is standard practice to calculate the standard error of the training perplexity from an algorithm like fitlda using the method of batch means. See Jones, et al (2006) for detail.

I can do this calculation by hand using fitlda's verbose output. However, the verbose output only gives the training perplexity out to 3 digits. It would be much easier (for me) and more accurate if there were a way to do this automatically.

Is there a way to automatically calculate the batch mean standard error of the training perplexity of fitlda?

Reference:

Jones, Galin L, Murali Haran, Brian S Caffo, and Ronald Neath, "Fixed-width output analysis for Markov chain Monte Carlo," Journal of the American Statistical Association, 2006, 101 (476), 1537-1547.

Answer 1

[Stephen Bruestle]

I answered this question myself.

While I have found no precise way to calculate batch mean standard error, I have found that you can get precise measurements of perplexity in "mdl.FitInfo.History.Perplexity"

Therefore to calculate batch mean standard error:

% Input Initial Burn
burn = 100;

% Calculate Batch Size (b), number of batches (a), and true burn (whatever is left from a*b)
Matrix = mdl.FitInfo.History.Perplexity;
[m,n] = size(Matrix);
b = floor(sqrt(m-burn));
a = floor((m-burn)/b);
burn = m - a*b;

% Burn initial values
Matrix(1:burn,:)=[];

% Create Groups (G)
G = [];
for t=1:a
  temp=t*ones(b,1);
  G=[G;temp];
end

% Calculate Batch Means
Y = splitapply(@mean,Matrix,G);

% Calculate Batch Mean Standard Error (stdev)
Ybar = mean(Y) * ones(a,1)
Err = Y-Ybar;
VAR = b/(a-1) * sum(Err.^2);
stdev = sqrt(VAR);

[Stephen Bruestle]

Note: This fit statistic should be standard.

[Wei Wong]

Hey there! May I know how do we determine the burn value? As setting different burn values will yield different results. Thank you!