If I run second or third correlation matirx after my main hypothesis, can I on say the second non hypothesis matrix only include the amount of pairwise comparisions in the second matrix to calculate my multiple comparison alpha value, even if some of my varibles were incldued in the first?
1 Answer
$\begingroup$
$\endgroup$
A multiple comparison correction provides protection for a family of tests. You should take into account all tests in this family when determining the multiple comparison alpha value. As a result, you can say something about this family of tests only – i.e. I have controlled for a single false positive over this family of tests at the alpha level.
Let’s say you do a second and third suite of post-hoc tests. If you include only the number of comparisons in each suite of tests, then you are offering familywise protection at the alpha level for each suite only.
In other words, each suite of tests is protected at its given alpha level. This says nothing about overall protection for all the tests you have carried out on the data. If you want to protect all tests carried out on the data , i.e. the entire family of tests, then then you need to use the number of tests in this entire family.
