Abstract: "The field of psychology, including cognitive science, is vexed in a crisis of confidence. Although the causes and solutions are assuredly varied, we focus here on a common logical problem in inference. The default mode of inference is significance testing, and significance testing as well as inference by confidence intervals has a free lunch property where researchers need not make detailed assumptions about the alternative to test the null hypothesis. We present the argument here that there is no free lunch, that is, to perform valid testing researchers must test the null against a well specified alternative. We show how this requirement follows from the very basic tenets of conventional and Bayesian probability. Moreover, we show in both the conventional and Bayesian framework that not paying for lunch is problematic because it leads to rejections of the null hypothesis on scant evidence, perhaps even in cases when reasonable alternatives are less likely. Paying for lunch is not hard, and we review both frequentist and Bayes factor approaches. Paying for lunch has consequential effects: the threshold for declaring effects in data is higher and many currently claimed effects do not meet this higher threshold. The field of cognitive science will undoubtably benefit from paying for lunch -- it is not only principled, but consideration of reasonable alternatives will undoubtably sharpen the intellectual backdrop for inference."

Abstract: "Recent advances in Bayesian hypothesis testing have led to the development of uniformly most powerful Bayesian tests, which represent an objective, default class of Bayesian hypothesis tests that have the same rejection regions as classical significance tests. Based on the correspondence between these two classes of tests, it is possible to equate the size of classical hypothesis tests with evidence thresholds in Bayesian tests, and to equate P values with Bayes factors. An examination of these connections suggest that

recent concerns over the lack of reproducibility of scientific studies can be attributed largely to the conduct of significance tests at unjustifiably high levels of significance. To correct this problem, evidence thresholds required for the declaration of a significant finding should be increased to 25–50:1, and to 100–200:1 for the

declaration of a highly significant finding. In terms of classical hypothesis tests, these evidence standards mandate the conduct of tests at the 0.005 or 0.001 level of significance"