Research Methods for the Behavioral Sciences
SAGE Journal Articles
Click on the following links. Please note these will open in a new window.
Journal Article Link 2.1: Fritz, A., Scherndl, T., & Kuhberger, A. (2012). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? Theory & Psychology, 23(1), 98-122.
Over-reliance on significance testing has been heavily criticized in psychology. Therefore the American Psychological Association recommended supplementing the p value with additional elements such as effect sizes, confidence intervals, and considering statistical power seriously. This article elaborates the conclusions that can be drawn when these measures accompany the p value. An analysis of over 30 summary papers (including over 6,000 articles) reveals that, if at all, only effect sizes are reported in addition to p’s (38%). Only every 10th article provides a confidence interval and statistical power is reported in only 3% of articles. An increase in reporting frequency of the supplements to p’s over time owing to stricter guidelines was found for effect sizes only. Given these practices, research faces a serious problem in the context of dichotomous statistical decision making: since significant results have a higher probability of being published (publication bias), effect sizes reported in articles may be seriously overestimated.
- In your own words, what is publication bias?
- What are examples of publication bias that the article discusses?
- What are the dangers of publication bias, for both the current study and the field as a whole?
Journal Article Link 2.2: Kepes, S., Banks, G. C., Mcdaniel, M., & Whetzel, D. L. (2012). Publication Bias in the Organizational Sciences. Organizational Research Methods, 15(4), 624-662.
Publication bias poses multiple threats to the accuracy of meta-analytically derived effect sizes and related statistics. Unfortunately, a review of the literature indicates that unlike meta-analytic reviews in medicine, research in the organizational sciences tends to pay little attention to this issue. In this article, the authors introduce advances in meta-analytic techniques from the medical and related sciences for a comprehensive assessment and evaluation of publication bias. The authors illustrate their use on a data set on employment interview validities. Using multiple methods, including contour-enhanced funnel plots, trim and fill, Egger’s test of the intercept, Begg and Mazumdar’s rank correlation, meta-regression, cumulative meta-analysis, and selection models, the authors find limited evidence of publication bias in the studied data.
- Why do you think research in organizational sciences pay little attention to these issues raised in the study?
Journal Article Link 2.3: Chapman, S., Ragg, M., & Mcgeechan, K. (2009). Citation bias in reported smoking prevalence in people with schizophrenia. Aust NZ J Psychiatry Australian and New Zealand Journal of Psychiatry, 43(3), 277-282.
Objective: A meta-analysis of 42 studies on tobacco smoking among schizophrenia subjects found an average smoking prevalence of 62% (range1488%). Statements are common, however, in the research literature and the media that between 80% and 90% of people with schizophrenia smoke. The purpose of the present paper was therefore to determine if citation bias exists in the over-citation and reportage of studies finding high rates of smoking prevalence in schizophrenia subjects. Methods: Two hypotheses were tested: (i) that studies on the prevalence of smoking in people with schizophrenia reporting high smoking rates would be cited more often than studies reporting lower rates; and (ii) that statements about smoking rates among schizophrenic people on the Internet would report very high rates more often than more realistic, less dramatic rates. Results: A 10% increase in reported prevalence of smoking was associated with a 61% (95% confidence interval (CI)3098%) increase in citation rate. Journal impact factor (IF) was significantly associated with citation rate (p0.001) but the country in which a study was carried out did not have an effect (p0.90). After adjusting for IF, a 10% increase in prevalence of smoking was associated with a 28% increase (95%CI162%) in citation rate. This bias is mirrored on the Internet, where statements abound about uncommonly highly rates of smoking by people with schizophrenia. Conclusions: Studies reporting very high prevalence of smoking among people with schizophrenia are cited more often than those studies reporting a low prevalence, a result consistent with citation bias. This citation bias probably contributes to the misinformation available on the Internet, and may have adverse policy and clinical implications.
- In your own words, what is citation bias?
- Why do authors engage in citation bias?
- What are the problems with citation bias related to this study?