How to compute friedman and holm post test in matlab?
2 views (last 30 days)
Show older comments
Hello professional colleagues,
I'm trying to reproduce the results of friedman and holm test from a paper for the first time usage of these techniques.
The contains the accuracies of seven classifiers where the last column is the results of the algorithm proposed in the papre. The table is as follows
x = [81.16, 85.36, 84.50, 81.00, 83.10, 83.74, 86.96;
95.28, 97.00, 96.30, 96.30, 93.60, 96.28, 97.42;
53.73, 63.18, 67.60, 60.60, 66.10, 62.86, 64.73;
81.01, 85.35, 85.10, 81.30, 82.20, 84.93, 87.10;
48.84, 57.66, 58.20, 56.60, 52.50, 65.98, 60.37;
70.52, 70.56, 68.70, 71.10, 71.80, 70.36, 71.11;
75.56, 83.34, 82.60, 79.60, 75.60, 84.52, 81.85;
77.50, 81.88, 68.50, 79.60, 57.50, 86.28, 84.84;
60.82, 73.80, 66.90, 64.80, 68.60, 85.60, 83.43;
86.33, 87.76, 92.00, 86.30, 85.50, 93.11, 86.88;
57.20, 74.88, 69.00, 68.50, 68.50, 48.80, 71.80;
63.47, 62.93, 69.00, 62.30, 64.60, 64.93, 67.59;
70.31, 75.79, 75.80, 70.40, 72.20, 74.51, 75.38;
82.09, 88.64, 87.90, 82.10, 78.20, 90.56, 79.24;
86.60, 84.12, 76.40, 86.50, 73.10, 84.87, 83.11;
94.44, 92.22, 95.60, 96.70, 97.80, 95.53, 97.00]
After typing
p = friedman(x,1)
The result is 9.8727e-05 which is identical to what is written in the paper. The hurdle now is that the authors wrote "we perform the post hoc test to the hypotheses of comparison between the kNNimp classifier and the others, we obtain their p values: 0.13670, 0.05622, 0.00085, 0.00290,0.01994, 0.05622, 0.07420. Holm’s procedure rejects the null hypotheses". I'm unable to understand these 6 values are the results of what. I tried to compute the friedman for pairs of classifier (1,7), (2,7), (3,7), (4,7), (5,7), (6,7)
The used code is as follows.
p = [];
for i = 1:size(x,2)-1
p = [p friedman([x(:,i) x(:,size(x,2))],1, 'off')]
end
[cor_p, h]=bonf_holm(p,.05)
where I downloaded the bonf_holm code as follows.
function [corrected_p, h]=bonf_holm(pvalues,alpha)
if nargin<1,
error('You need to provide a vector or matrix of p-values.');
else
if ~isempty(find(pvalues<0,1)),
error('Some p-values are less than 0.');
elseif ~isempty(find(pvalues>1,1)),
fprintf('WARNING: Some uncorrected p-values are greater than 1.\n');
end
end
if nargin<2,
alpha=.05;
elseif alpha<=0,
error('Alpha must be greater than 0.');
elseif alpha>=1,
error('Alpha must be less than 1.');
end
s=size(pvalues);
if isvector(pvalues),
if size(pvalues,1)>1,
pvalues=pvalues';
end
[sorted_p sort_ids]=sort(pvalues);
else
[sorted_p sort_ids]=sort(reshape(pvalues,1,prod(s)));
end
[dummy, unsort_ids]=sort(sort_ids); %indices to return sorted_p to pvalues order
m=length(sorted_p); %number of tests
mult_fac=m:-1:1;
cor_p_sorted=sorted_p.*mult_fac;
cor_p_sorted(2:m)=max([cor_p_sorted(1:m-1); cor_p_sorted(2:m)]); %Bonferroni-Holm adjusted p-value
corrected_p=cor_p_sorted(unsort_ids);
corrected_p=reshape(corrected_p,s);
h=corrected_p<alpha;
Unfortunately, the results did not match. The results are
cor_p = 0.016 0.95 0.95 0.016 0.049 0.634
Any suggestions about what I am doing wrong?!
0 Comments
Answers (0)
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!