$Multiple comparisons problem$

Question

$Multiple comparisons problem$

Hi,

I have a question about correcting for multiple comparisons. My question is: for exactly how many tests should I correct in the scenario below? I can't seem to wrap my head around it.

Background: I’m testing for differences between groups A (patient group) and B (control group) in variable X. Variable X is a biological variable and is present in the left and right side of the body. Variable Y is a score on a test.

Step 1. Is there a significant difference within groups in (the weight of) left and right variable X?
For this purpose will conduct two paired sample t-tests

If I find a significant difference in step 1, then I will continue with both step 2A and 2B. If I don’t find a significant difference in step 1, then I will only continu with step 2C.

Step 2A. Is there a significant difference between groups in (the weight of) left variable X?
For this purpose I will conduct a independent t-test

Step 2B. Is there a significant difference between groups in (the weight of) right variable X?
For this purpose I will conduct a independent t-test

Step 2C. Is there a significant difference between groups in (the weight of) total variable X (= weight left + right variable X)?
For this purpose I will conduct a independent t-test

After step 2, and depending on the outcome in step 1 (significant, or not significant differences within groups), I will continue with both step 3A and 3B, or only step 3C.

Step 3A. Is there a significant correlation between (the weight of) left variable X in group A and variable Y?
For this purpose I will conduct a Pearson correlation

Step 3B. Is there a significant correlation between (the weight of) right variable X in group A and variable Y?
For this purpose I will conduct a Pearson correlation

Step 3C. Is there a significant correlation between (the weight of) total variable X in group A and variable Y?
For this purpose I will conduct a Pearson correlation

Thank you!

Statistics

Deruiteraleta

Report

Answer

The answer is accepted.

Join Matchmaticians Affiliate Marketing Program to earn up to 50% commission on every question your affiliated users ask or answer.

Answer 1

Answers can be viewed only if

The questioner was satisfied and accepted the answer, or
The answer was disputed, but the judge evaluated it as 100% correct.

View the answer

Hi At,

There isn't a direct, one-size-fits all answer to this question, but there are some general ways to approach it.

1. At one end of the spectrum, if all of these tests were independent of each other, you could apply a Bonferroni correction and use an alpha level divided by your number of tests as the new alpha level you use for all of your tests. The idea behind this is that every time you're testing, you're flipping a biased coin that tells you if you'll get a false positive or true positive as your result. The more times you flip (or test) then the more likely it comes up "heads" (or, the more likely it comes up with a false positive). By diving your alpha level by the total number of tests you're going to apply, you're making your alpha level more stringent: smaller alpha levels are harder to reject. But, if you have many, many tests, then alpha divided by number of tests will be too stringent and your false negative rate will increase.

TL;DR: you can divide your alpha level, say if it were .05, by your number of tests here, 7, and now you only reject your null hypothesis when your p value is less than or equal to .05/7. It's not the best, though.

2. reformat your problem to use some form of ANOVA to check if there are ANY significant differences between the groups you'd like to compare, and then use Tukey's honest significant difference (HSD) test. Tukey's test makes all possible pairwise comparisons and controls the family-wise error rate, which is better. Tukey's is the most powerful comparison if ANOVA assumptions are valid and you're just making pair-wise comparisons. You will have to work with an adjusted version of ANOVA / Tukey's test for paired or dependent data.

3. Benjamini-Hochberg Step Up Method. This procudure attempts to balance your tests' false discovery rate and family-wise error rate, and is also more nuanced /balanced than the Bonferroni correction. For instructions, see a basic description here as a starting point: https://www.statology.org/benjamini-hochberg-procedure/.

Cmartman

$Multiple comparisons problem$

Answer

Related Questions

Search