Misconception Mutt extract from chapter 14
The Misconception Mutt was very tired. His owner had been walking him early before lectures and he needed some sleep. This morning’s game of fetch was killing him, and yet his little legs seemed incapable of not speeding his body off towards the stick that his owner seemed to lose so carelessly, persistently and forcefully. As he sprinted, his mind was wandering to F-statistics in factorial designs. He thought they were a neat way to control Type I errors. As he picked up the stick, it winked at him. Was it the sleep deprivation? He wasn’t sure but then its bark turned a ginger hue, and he noticed fur shooting out from it. Soon legs began to poke out, and a tail. Before he knew it he was carrying a very heavy ginger cat in his mouth. This was all he needed.
‘Did you notice how the F-statistics in factorial designs are computed?’ the cat said.
Determined not to be outwitted by a cat, the mutt replied, ‘Yes, the mean squares for the effect divided by the residual mean squares.’
‘Exactly,’ the cat grinned. ‘They all use the residual mean squares. This means that they are not independent.’ The cat waited as though expecting applause. The mutt was confused, so the cat continued. ‘The F-statistics are not independent, which means that they do not control the Type I error rate. In the example from your lecture there are three Fs. The probability that at least one of them is a Type I error is not 0.05, but is 0.143 because the Fs are not independent. If you had even more predictors and interactions the probability of at least one Type 1 error would be even greater.’
‘What can be done?’ the mutt asked.
‘You could use a more stringent criterion for accepting an F as significant. With three effects you might use 0.05/3 = 0.017, which is effectively a Bonferroni correction. Or simply ignore any effects that don’t test your substantive hypothesis. In this case interpret the interaction and ignore the main effects (even if they are significant).’
With that the legs, fur and grin retracted and the mutt dropped the stick at his owner’s feet.