Is there publication bias in criminology? There could be a bias towards only publishing significant results, which is sometimes referred to as a “significance filter”. If so, researchers have incentives to do a bit of data snooping and p-hacking to make sure they get significant results without necessarily reporting all the massaging they have done.
Although not a study of the “significance filter”, a recent study of published experimental studies in criminology nevertheless includes some relevant information. It is reported that 68% of the reported effect estimates were not significant. This might be interpret as it is no not likely to be much of a significance filter in criminology. However, the finding is based on 402 effect estimates from 66 publications, which implies an average of six effect estimates per study. The probability of getting one out of six with p<0.05 is surely much larger than 0.05.
Perhaps it only takes at least one significant result to get published? If so, the expectation given the existence of a significance filter would be that that at least 66 out of the 402 estimates were statistically significant (16%). It would have been interesting to know how many of the studies did not find any significant findings, and how many reported significant findings only for subgroup analysis or alternative specifications – as well as how many subgroups and alternative specifications were (actually) tried out.
Maybe the cartoon XKCD got it uncomfortably right?
And perhaps also: “Hey, look at this interesting alternative outcome measure”.
P.S. I have no reason to think the significance filter is more/less prevalent in criminology than in other fields. It is probably similar to other social sciences.
The post Is there a significance filter in criminology? appeared on The Grumpy Criminologist 2016-07-25 12:38:33 by Torbjørn.