This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Falk, R. Articles by Greenbaum, C. W. Search for Related Content Social Bookmarking                   What's this?

Significance Tests Die Hard

The Amazing Persistence of a Probabilistic Misconception

The Hebrew University Of Jerusalem

Charles W. Greenbaum

The Hebrew University Of Jerusalem

We present a critique showing the flawed logical structure of statistical significance tests. We then attempt to analyze why, in spite of this faulty reasoning, the use of significance tests persists. We identify the illusion of probabilistic proof by contradiction as a central stumbling block, because it is based on a misleading generalization of reasoning from logic to inference under uncertainty. We present new data from a student sample and examples from the psychological literature showing the strength and prevalence of this illusion. We identify some intrinsic cognitive mechanisms (similarity to modus tollens reasoning; verbal ambiguity in describing the meaning of significance tests; and the need to rule out chance findings) and extrinsic social pressures which help to maintain the illusion. We conclude by mentioning some alternative methods for presenting and analyzing psychological data, none of which can be considered the ultimate method.

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

				R. Hubbard and R. M. Lindsay Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing Theory Psychology, February 1, 2008; 18(1): 69 - 88. [Abstract] [PDF]

				R. Hubbard Alphabet Soup: Blurring the Distinctions Betweenp's anda's in Psychological Research Theory Psychology, June 1, 2004; 14(3): 295 - 327. [Abstract] [PDF]

				S. Finch, N. Thomason, and G. Cumming Past and Future American Psychological Association Guidelines for Statistical Practice Theory Psychology, December 1, 2002; 12(6): 825 - 853. [Abstract] [PDF]

				B. Hess, S. Olejnik, and C. J Huberty The Efficacy of two Improvement-Over-Chance Effect Sizes for Two-Group Univariate Comparisons under Variance Heterogeneity and Nonnormality Educational and Psychological Measurement, December 1, 2001; 61(6): 909 - 936. [Abstract] [PDF]

				J. T. Chibnall, J. M. Jeral, and M. A. Cerullo Experiments on Distant Intercessory Prayer: God, Science, and the Lesson of Massah Arch Intern Med, November 26, 2001; 161(21): 2529 - 2536. [Full Text] [PDF]

				S. Finch, G. Cumming, and N. Thomason Colloquium on Effect Sizes: the Roles of Editors, Textbook Authors, and the Publication Manual: Reporting of Statistical Inference in the Journal of Applied Psychology: Little Evidence of Reform Educational and Psychological Measurement, April 1, 2001; 61(2): 181 - 210. [Abstract] [PDF]

				T. Vacha-Haase Statistical Significance should not be Considered one of Life's Guarantees: Effect Sizes are Needed Educational and Psychological Measurement, April 1, 2001; 61(2): 219 - 224. [PDF]

				R. Hubbard and P. A. Ryan Statistical Significance with Comments by Editors of Marketing Journals: The Historical Growth of Statistical Significance Testing in Psychology--and its Future Prospects Educational and Psychological Measurement, October 1, 2000; 60(5): 661 - 681. [Abstract] [PDF]

				C. J Huberty and L. L. Lowman Group Overlap as a Basis for Effect Size Educational and Psychological Measurement, August 1, 2000; 60(4): 543 - 563. [Abstract] [PDF]

				T. Vacha-Haase, J. E. Nilsson, D. R. Reetz, T. S. Lance, and B. Thompson Reporting Practices and APA Editorial Policies Regarding Statistical Significance and Effect Size Theory Psychology, June 1, 2000; 10(3): 413 - 425. [Abstract] [PDF]

				H. J. Stam Ten Years after, Decade to Come: The Contributions of Theory to Psychology Theory Psychology, February 1, 2000; 10(1): 5 - 21. [Abstract] [PDF]

				B. Thompson If Statistical Significance Tests are Broken/Misused, What Practices Should Supplement or Replace Them? Theory Psychology, April 1, 1999; 9(2): 165 - 181. [Abstract] [PDF]

				D. Sohn Statistical Significance and Replicability: Why the Former Does not Presage the Latter Theory Psychology, June 1, 1998; 8(3): 291 - 311. [Abstract]

				R. Falk Replication-A Step in the Right Direction: Commentary on Sohn Theory Psychology, June 1, 1998; 8(3): 313 - 321. [Abstract]

				R. Hubbard, R. A. Parsa, and M. R. Luthy The Spread of Statistical Significance Testing in Psychology: The Case of the Journal of Applied Psychology, 1917-1994 Theory Psychology, August 1, 1997; 7(4): 545 - 554. [Abstract]

				R. E. Kirk Practical Significance: A Concept Whose Time Has Come Educational and Psychological Measurement, October 1, 1996; 56(5): 746 - 759. [Abstract]