Which pairwise hypothesis test is used on multcompare

HI All,

I have a feeling this question should be easily answered, so I suspect I might just be missing something.

On the function page of multcompare it aptly states that it performs the appropriate pairwise testing based on the stats input. But I'm trying to find what actual test is used. On the function page it keeps talking about a "hypothesis test", but I can't seem to find which actually test is being used.

In the "More about" section, it talks about t-tests, but I'm not sure if this isn't used as an example or if it means that is simply performs t-tests on the input (which would be performs on the the appropriate type of data, for instance on the ranks when using Kruskal-Wallis?).

Any help is greatly appreciated.

[Ive J]

You can set the desired method using CType, default is Turkey's.

res = multcompare(ES, 'CType', 'dunn-sidak'); 

[Reinder Dorman]

Hey Ive,

Thanks, but if I'm not mistaken, that is the multiple-comparison correction method used, which would affect the alpha for significance. But I don't think this is the actual test being used, right?

[Ive J]

Yes it is correct. But the actual test being used depends on the background test; for instance a test like ANOVA would require a t-tes while a non-parametric ones like Kriskal-Wallis would requie a non-parametric one like Wilcoxon. Regardless, the function uses the means or meanranks from stat structure.

[~, ~, stats] = anova1(MPG, Origin, 'off')
stats = 

  struct with fields:

    gnames: {6×1 cell}
         n: [64 15 9 3 2 1]
    source: 'anova1'
     means: [21.1328 31.8000 28.4444 23.6667 22.5000 28]
        df: 88
         s: 7.0513
         
[~, ~, stats] = kruskalwallis(MPG, Origin, 'off')
stats = 
  struct with fields:

       gnames: {6×1 cell}
            n: [64 15 9 3 2 1]
       source: 'kruskalwallis'
    meanranks: [38.6719 75.2000 62.1111 49.3333 43.7500 67.5000]
         sumt: 1266

You can check this further by

edit multcompare

So, you can realize that the function calculates critical values using tinv (t inverse CDF) for parametric tests (anova1 or anova2), while it uses norminv (normal inverse CDF) for non-parametric (KW or Friedman) ones.

Answer 1

[Scott MacKenzie]

The pairwise comparisons test used by default with multcompare is Tukey-Kramer. Using the ctype option, you can change the test to Bonferroni, Scheffe, Dunn-Sidak, or Fisher's LSD. Generally, the different tests yield the same or similar outcomes.

The discussion on t-tests in "More About" has nothing to do with how the tests are done in multcompare. This discussion, which I just read, is a nicely written explanation on why a customized test, such as Tukey-Kramer, is needed in the first place. The key point is this: When there are many group means, there are also many pairs to compare. If you applied an ordinary t-test in this situation, the alpha value would apply to each comparison, so the chance of incorrectly finding a significant difference would increase with the number of comparisons. All the pairwise comparisons tests accommodate the latter issue (incorrectly finding a significant difference) by applying a more conservative criterion than would exist if a simple t-test were used for each pairwise comparison.

[Hamed Nassar]

Tukey-Kramer, Bonferroni, Scheffe, Dunn-Sidak, etc. are not statistical tests. They are post-hoc correction schemes to adjust the p-value resulting from the original test, being Anov1, ttest, Freidman, etc.

Answer 2

[Frederik Bergmann]

So basically the question is still unanswered isn't it? What type of test is done to compare groups?

For example, if I do a Friedman test and submit the stats struct to mutlcompare, will I get the results of a Wilcoxon signed rank test - which should be done in that case?

unfortunately the output of multcompare does not provide any insight. There is just one column in the c matrix containing the pvalue of each comparison (row). But what if I want to know the t stats or r stat?

[Ive J]

Tukey-Kramer is by default performed for Friedman non-parametric test. The function is a general purpose comment for different post-hoc tests on different omnibus test, meaning user must decide which post-hoc is suited for the corresponding omnibus test.

For your specific case, see the following links:

https://stats.stackexchange.com/questions/371562/tukey-kramer-or-nemenyi-following-friedman

https://stats.stackexchange.com/questions/355439/post-hoc-test-for-friedman-test

Tukey-Kramer in MATLAB is same as Nemenyi test, so you can safely use it for Friedman test. You can also compare the stats from MATLAB multcompare and PMCMRplus::frdAllPairsNemenyiTest function from R package PMCMRplus.