Click on the following links. Please note these will open in a new window.

Journal Article Link 2.1: Fritz, A., Scherndl, T., & Kuhberger, A. (2012). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? Theory & Psychology, 23(1), 98–122.

Abstract: Overreliance on significance testing has been heavily criticized in psychology. Therefore the American Psychological Association recommended supplementing the p value with additional elements such as effect sizes, confidence intervals, and considering statistical power seriously. This article elaborates the conclusions that can be drawn when these measures accompany the p value. An analysis of over 30 summary papers (including over 6,000 articles) reveals that, if at all, only effect sizes are reported in addition to ps (38%). Only every 10th article provides a confidence interval and statistical power is reported in only 3% of articles. An increase in reporting frequency of the supplements to ps over time owing to stricter guidelines was found for effect sizes only. Given these practices, research faces a serious problem in the context of dichotomous statistical decision making: Since significant results have a higher probability of being published (publication bias), effect sizes reported in articles may be seriously overestimated.

1. In your own words, what is publication bias?
2. What are examples of publication bias that the article discusses?
3. What are the dangers of publication bias for both the current study and the field as a whole?

Journal Article Link 2.2: Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15(4), 624–662.

Abstract: Publication bias poses multiple threats to the accuracy of meta-analytically derived effect sizes and related statistics. Unfortunately, a review of the literature indicates that unlike meta-analytic reviews in medicine, research in the organizational sciences tends to pay little attention to this issue. In this article, the authors introduce advances in meta-analytic techniques from the medical and related sciences for a comprehensive assessment and evaluation of publication bias. The authors illustrate their use on a data set on employment interview validities. Using multiple methods, including contour-enhanced funnel plots, trim and fill, Egger’s test of the intercept, Begg and Mazumdar’s rank correlation, meta-regression, cumulative meta-analysis, and selection models, the authors find limited evidence of publication bias in the studied data.

1. Why do you think research in organizational sciences pay little attention to these issues raised in the study?

Journal Article Link 2.3: Chapman, S., Ragg, M., & McGeechan, K. (2009). Citation bias in reported smoking prevalence in people with schizophrenia. Australian & New Zealand Journal of Psychiatry Australian and New Zealand Journal of Psychiatry, 43(3), 277–282.

Abstract: A meta-analysis of 42 studies on tobacco smoking among schizophrenia subjects found an average smoking prevalence of 62% (range1488%). Statements are common, however, in the research literature and the media that between 80% and 90% of people with schizophrenia smoke. The purpose of the present paper was therefore to determine if citation bias exists in the over-citation and reportage of studies finding high rates of smoking prevalence in schizophrenia subjects. Methods: Two hypotheses were tested: (1) that studies on the prevalence of smoking in people with schizophrenia reporting high smoking rates would be cited more often than studies reporting lower rates; and (2) that statements about smoking rates among schizophrenic people on the Internet would report very high rates more often than more realistic, less dramatic rates. Results: A 10% increase in reported prevalence of smoking was associated with a 61% (95% confidence interval [CI] = 30–98%) increase in citation rate. Journal impact factor (IF) was significantly associated with citation rate (p = 0.001) but the country in which a study was carried out did not have an effect (p = 0.90). After adjusting for IF, a 10% increase in prevalence of smoking was associated with a 28% increase (95%CI = 1–62%) in citation rate. This bias is mirrored on the Internet, where statements abound about uncommonly highly rates of smoking by people with schizophrenia. Conclusions: Studies reporting very high prevalence of smoking among people with schizophrenia are cited more often than those studies reporting a low prevalence, a result consistent with citation bias. This citation bias probably contributes to the misinformation available on the Internet, and may have adverse policy and clinical implications.

1. In your own words, what is citation bias?
2. Why do authors engage in citation bias?
3. What are the problems with citation bias related to this study?