Of Statistics and Significance

Most psychology students are steeped in the culture of null hypothesis testing, with convention dictating that p-values below 0.05 be treated rather differently from p-values above that threshold.  If one had little exposure to other branches of science, one might even come to believe that this form of inferential practice was at the heart of the scientific method.  Nothing could be more (significantly) wrong.  In fact, null hypothesis testing, the concept of statistical significance, and the holy p-value are all rather local phenomena, found primarily in the soft sciences, where the entities being discussed are in desperate need of shoring up to ensure their very reality: a job that null hypothesis significance testing does very poorly.

Interesting then that a reputable journal smack in the middle of the soft sciences, Basic and Applied Social Psychology has now reached a point where it is banning null hypothesis testing and the associated argument from "significance".

In its most recent editorial, the editors simply state that "null hypothesis significance testing procedure (NHSTP) is invalid, and thus authors would be not required to perform it".  They go on to express the hope that "by liberating authors from the stultified structure of NHSTP thinking [we will] thereby eliminat[e] an important obstacle to creative thinking. The NHSTP has dominated psychology for decades; we hope that by instituting the first NHSTP ban, we demonstrate that psychology does not need the crutch of the NHSTP, and that other journals follow suit."

I confess I used to teach NHSTP to cognitive science students, albeit without much enthusiasm.  My strongest advice was always "Don't do statistics.  Understand your data instead.  Only then, if it contributes to making a clearer or better argument, cautiously do statistics, but don't allow your statistics to spring any surprises on you".  I am glad that there has recently been considerable blowback against a very flawed methodology that has had the pernicious consequence of infecting the literature.

For those wishing to explore the strengths and considerable limitations of NHSTP, I recommend this blog post by Kristoffer Magnusson.