Multiple Comparison procedures for One-Way layouts: Kruskal-Wallis and multiple comparison tests. Starting random-number seed: 123456 (Enter a number on the command line to specify a different seed.) Data: Numbers of glucocorticoid receptor sites per Leukocyte cell Hollander & Wolfe (Table 6.4, p201) Number of treatments: 5 Total number of observations: 37 Group #1 (n=14): 3500 3500 3500 4000 4000 4000 4300 4500 4500 4900 5200 6000 6750 8000 Group #2 (n=5): 5710 6110 8060 8080 11400 Group #3 (n=6): 2930 3330 3580 3880 4280 5120 Group #4 (n=4): 6320 6860 11400 14000 Group #5 (n=8): 3230 3880 7640 7890 8280 16200 18250 29900 Data with midranks (RAv=RankSum/n_i): #1 (RAv=14.43, n=14, Normal): 3500( 5.0) 3500( 5.0) 3500( 5.0) 4000(11.0) 4000(11.0) 4000(11.0) 4300(14.0) 4500(15.5) 4500(15.5) 4900(17.0) 5200(19.0) 6000(21.0) 6750(24.0) 8000(28.0) #2 (RAv=26.70, n=5, HCell Anemia): 5710(20.0) 6110(22.0) 8060(29.0) 8080(30.0) 11400(32.5) #3 (RAv= 8.42, n=6, CLL): 2930( 1.0) 3330( 3.0) 3580( 7.0) 3880( 8.5) 4280(13.0) 5120(18.0) #4 (RAv=28.63, n=4, CML): 6320(23.0) 6860(25.0) 11400(32.5) 14000(34.0) #5 (RAv=25.31, n=8, Acute): 3230( 2.0) 3880( 8.5) 7640(26.0) 7890(27.0) 8280(31.0) 16200(35.0) 18250(36.0) 29900(37.0) The large-sample Kruskal-Wallis test: HP=16.6682 Tiegroups=5 Tiesum=66 Tiecorr=0.00130394 P=0.0022 (Chi-square approx, df=4) Simulating the true KW P-value with n=1,000,000 permutations: Hobs = Sum RankSum_i^2/n_i = 15307.4 P = 431/1,000,000 = 0.00043 95% CI: (0.00039, 0.00043, 0.00047) Which of the 10 (= k*(k-1)/2) pairwise comparisons between the 5 (=k) treatment groups are significant, allowing for multiple comparisons? Using the Steel-Dwass-Crichlow-Fligner multiple-comparison procedure described in Section 6.5 of Hollander & Wolfe. That is, let ObsZ(i,j) be the pairwise Wilcoxon rank-sum score between all treatment-group pairs, normalized to have mean zero and variance one. For a permutation of the data, let mstar = max_{a,b}Z(a,b) where Z(i,j) is the corresponding normalized Wilcoxon score between pairs of permuted data. Finally, let P(i,j) be the proportion of permutations for which mstar >= |ObsZ(i,j)|. Then P(i,j) are multiple-comparison-corrected P-values for all treatment pairs (i,j). For comparison, the following are multiple-comparison-UNCORRECTED P-values. Pairwise Wilcoxon W and tie-corrected Z values: #1 vs. #2: W= 80.0 E(W)= 50.0 Z=2.788 P=0.0053 (2-sided, UNCORR) #1 vs. #3: W= 43.0 E(W)= 63.0 Z=1.655 P=0.0979 (2-sided, UNCORR) #1 vs. #4: W= 63.0 E(W)= 38.0 Z=2.667 P=0.0076 (2-sided, UNCORR) #1 vs. #5: W=121.0 E(W)= 92.0 Z=1.984 P=0.0472 (2-sided, UNCORR) #2 vs. #3: W= 21.0 E(W)= 36.0 Z=2.739 P=0.0062 (2-sided, UNCORR) #2 vs. #4: W= 23.5 E(W)= 20.0 Z=0.861 P=0.3893 (2-sided, UNCORR) #2 vs. #5: W= 59.0 E(W)= 56.0 Z=0.439 P=0.6605 (2-sided, UNCORR) #3 vs. #4: W= 34.0 E(W)= 22.0 Z=2.558 P=0.0105 (2-sided, UNCORR) #3 vs. #5: W= 76.5 E(W)= 60.0 Z=2.132 P=0.0330 (2-sided, UNCORR) #4 vs. #5: W= 54.0 E(W)= 52.0 Z=0.340 P=0.7341 (2-sided, UNCORR) Maximum observed absolute Z-value: 2.7885 Simulating P-values for MULTIPLE-COMPARISON-CORRECTED comparisons for all pairs (using 10,000 permutations of data): Pairwise MULTIPLE-COMPARISON-CORRECTED P-values: #1 vs. #2: ObsZ=2.79 P = 157/10,000 = 0.0157 (*) #1 vs. #3: ObsZ=1.66 P = 4800/10,000 = 0.4800 #1 vs. #4: ObsZ=2.67 P = 305/10,000 = 0.0305 (*) #1 vs. #5: ObsZ=1.98 P = 2634/10,000 = 0.2634 #2 vs. #3: ObsZ=2.74 P = 217/10,000 = 0.0217 (*) #2 vs. #4: ObsZ=0.86 P = 9313/10,000 = 0.9313 #2 vs. #5: ObsZ=0.44 P = 9974/10,000 = 0.9974 #3 vs. #4: ObsZ=2.56 P = 541/10,000 = 0.0541 #3 vs. #5: ObsZ=2.13 P = 1927/10,000 = 0.1927 #4 vs. #5: ObsZ=0.34 P = 9995/10,000 = 0.9995 (**) P<0.01 (*) 0.01<=P<0.05 NOTE: The text (p240-242) uses Wstar=Sqrt(2)*Z, not Z. Then Max Wstar(i,j) has a normal-range distribution for large ni Thus large-sample multiple-comparison-corrected Wilcoxon P-values can be found by the probability that a normal-range statistic is larger than Wstar(i,j). By definition, the normal-range statistic is the maximum minus the minimum of k independent N(0,1) variables, which can be simulated: Note that these P-values are similar to the exact MC-corrected P-values above, but are more conservative. Pairwise MULTIPLE-COMPARISON-CORRECTED P-values via the normal-range statistic: #1 vs. #2: W= 80.0 Wstar=3.944 P= 401/10,000 = 0.0401 (*) #1 vs. #3: W= 43.0 Wstar=2.341 P= 4586/10,000 = 0.4586 #1 vs. #4: W= 63.0 Wstar=3.772 P= 594/10,000 = 0.0594 #1 vs. #5: W=121.0 Wstar=2.806 P= 2708/10,000 = 0.2708 #2 vs. #3: W= 21.0 Wstar=3.873 P= 473/10,000 = 0.0473 (*) #2 vs. #4: W= 23.5 Wstar=1.218 P= 9082/10,000 = 0.9082 #2 vs. #5: W= 59.0 Wstar=0.621 P= 9915/10,000 = 0.9915 #3 vs. #4: W= 34.0 Wstar=3.618 P= 793/10,000 = 0.0793 #3 vs. #5: W= 76.5 Wstar=3.016 P= 2017/10,000 = 0.2017 #4 vs. #5: W= 54.0 Wstar=0.480 P= 9967/10,000 = 0.9967 You can also do multiple comparisons WITH A CONTROL, by assuming one of the treatment groups is a `null group' and comparing all other treatment groups to it. This can be done either TWO-SIDED (looking for absolute differences) or (more commonly) ONE-SIDED (looking for larger or smaller), in both cases in comparison with the control (or null) group. Here we consider treatment group #1 AS THE CONTROL and use a ONE-SIDED procedure to look for treatment groups that are SIGNIFICANTLY LARGER than the control. (See Section 6.7 in the text.) As before, we use normalized Wilcoxon rank-sum differences, and base a multiple-comparison-corrected P-value for treatment #i on the number of times that the maximum of treatment #i vs. control group normalized comparisons is larger than the observe treatment #1 vs control comparison. Since this is a one-sided procedure, absolute values are not used. NOTE THAT this is the method of Steel (1959) (text p259) and NOT the Nemenyi-Damico-Wolfe method discussed in most of Section 6.7. For comparison, the following are multiple-comparison-UNCORRECTED P-values. Pairwise Wilcoxon W and Z values: #1 vs. #2: W= 80.0 E(W)= 50.0 Z= 2.788 P=0.0026 (1-sided, UNCORR) #1 vs. #3: W= 43.0 E(W)= 63.0 Z=-1.655 P=0.9511 (1-sided, UNCORR) #1 vs. #4: W= 63.0 E(W)= 38.0 Z= 2.667 P=0.0038 (1-sided, UNCORR) #1 vs. #5: W=121.0 E(W)= 92.0 Z= 1.984 P=0.0236 (1-sided, UNCORR) Maximum observed Z-value: 2.7885 Simulating P-values for multiple-comparison-corrected comparisons for control vs groups #2-#5, using 10,000 permutations: Pairwise MULTIPLE-COMPARISON-CORRECTED P-values: #1 vs. #2: ObsZ= 2.79 P = 52/10,000 = 0.0052 (1-sided) (**) #1 vs. #3: ObsZ=-1.66 P = 9997/10,000 = 0.9997 (1-sided) #1 vs. #4: ObsZ= 2.67 P = 82/10,000 = 0.0082 (1-sided) (**) #1 vs. #5: ObsZ= 1.98 P = 765/10,000 = 0.0765 (1-sided) (**) P<0.01 (*) 0.01<=P<0.05 Random numbers used: 36,783,788