Professor Cumming: dispelling the illusion of certainty
The world’s largest scientific psychology organisation from next year will encourage authors in its highly influential global scientific journal Psychological Science to use what for them will be a radically different statistical method to report their research findings.
The move by the Washington-based Association for Psychological Science (APS) to embrace the ‘New Statistics’ follows the work of La Trobe University Emeritus Professor Geoff Cumming.
The APS said it was undertaking this important step to help boost the ‘replicability of scientific studies’ and promote ‘robust research practices’ across all areas of the discipline.
The Association is making available freely on line an extended tutorial The New Statistics: Why and How developed by Professor Cumming for its more than 20,000 members and anyone else who wishes to submit their research to the top APS journal.
When significance is not significant
Head of La Trobe’s Statistical Cognition Laboratory, Professor Cumming is author of Understanding The New Statistics, published by Routledge in the US in 2012.
He said most psychologists – as well as many researchers in biomedical science and other disciplines – have traditionally relied on a statistical technique called ‘null-hypothesis significance testing’. This requires their results to reach an arbitrary ‘.05 p value’ before the research outcome can be described as ‘significant’.
He explained that significance testing gives an illusion of certainty, but is actually extremely unreliable. ‘It uses weird backward logic and bamboozles countless students every year in their introduction to statistics.’
Instead the ‘New Statistics’, for which Professor Cumming has been a long-time campaigner, involves the use of effect sizes, estimation, and meta-analysis.
On its website the APS said the changes will help ‘strengthen the overall integrity of scientific research, conveying benefits not only (for) the scientific community but also for the general public.’
Matter of life and death
So why is this important? ‘It’s important because statistics and the way we understand and act upon them can be a matter of life and death,’ Professor Cumming said.
‘For example, in the 1970s parents were advised to put new babies to sleep face-down on a sheepskin, even while evidence was gradually accumulating that back sleeping is much safer, and greatly reduces the risk of SIDS (cot death).
‘Meta-analysis was, however, not available then to integrate the scattered evidence, so the dangerous advice for face-down sleeping persisted,’ he said.
‘It’s estimated that if meta-analysis had been available and used, and the resulting recommendation for back sleeping had been made, as many as 50,000 infant deaths in the developed world could have been prevented.
Distortion of published research
Professor Cumming said while new to most researchers in psychology and biomedical science, estimation has been widely used by physical scientists and engineers. ‘It’s a much more informative technique, and avoids the worst problems of significance testing.
‘Meta-analysis, a vital component of the New Statistics, allows researchers to integrate findings over a number of related studies. But meta-analysis can only give reliable results if all studies on a topic are available.
‘However, significance testing distorts the published research record,’ Professor Cumming explained. ‘Scientific journals are more likely to publish a significant result. So studies that fail to obtain “significance” tend not to see the light of day and therefore escape the attention of anyone conducting meta-analysis.’
Clearer conclusions
‘Meta-analysis is based on estimation and makes statistical significance virtually irrelevant. And it can allow clear conclusions to be drawn from messy research literature’, Professor Cumming said.
Why has it taken so long to achieve change? ‘I suspect one reason is that declaring a result “significant” strongly suggests certainty and that the result is large and important – even though statistical significance does not imply that.’
‘Now it is up to researchers to change their deeply-entrenched habit of using statistical significance, and move forward to the much more informative New Statistics,’ he concluded.
Why we need ‘The New Statistics’
How significant are p values, really?
Tutorial on ‘The New Statistics: Why and How’
Listen the podcast on Ockham’s Razor, ABC Radio National
