P-values based on approximate conditioning and p  Chris J. Lloyd University of Melbourne, Carlton 3053, Australia article info Article history: Received 13 November 2008 Received in revised form 22 May 2009 Accepted 21 October 2009 Available online 29 October 2009 Keywords: Nuisance parameters Exact test Tests of independence r  abstract P-values based on higher order asymptotic formulas such as p  are now readily available for practitioners. However, it is not always clear what these P-values mean for discrete models. For a canonical parameter, p  should approximate a tail probability of the conditional distribution. Yet when this conditional distribution becomes degenerate, p  still gives a non-degenerate answer. So there is the need for a more general interpretation of p  . Pierce and Peters (1999) have argued that p  approximates an approximately conditional P-value and, implicitly, that this is an inferentially sensible quantity worth approximating. We investigate these twin claims for the simple case of 2  2 tables. We find that approximately conditional P-values have rather erratic properties and that they are not especially well approximated by p  , particularly when the observed data are near the boundary of the sample space. We also argue that approximately conditional P-values suffer from two, previously unrecognised, logical flaws. The consequences of these conclusions are discussed. & 2009 Elsevier B.V. All rights reserved. 1. Introduction Recent advances in likelihood theory have seen so-called higher order approximations to P-values for scalar parameters. An overview of the theory may be found in Barndorff-Neilsen and Cox (1994) and Reid (2003), and many details of practical implementation are in Brazzale et al. (2007). For canonical parameters in continuous exponential families, the methods reduce to a double saddlepoint approximation to the tail probability of the estimator conditional on the sufficient statistics for the nuisance parameters. For non-canonical parameters, the tail probability is conditional on an approximate sufficient statistic. The approximations are extremely accurate and conditioning for continuous models is well accepted. While there are various versions of these P-values, we refer to them generically as p  . For discrete models, the situation is less clear. Firstly, there is controversy over whether a conditional P-value is appropriate. Secondly, conditional P-values can become degenerate in which case it is not clear what a non-degenerate p  is approximating. Thirdly, it is unclear if a continuity correction should be applied. Pierce and Peters (1992) show that a continuity corrected version of p  can give accurate approximations to conditional P-values, for both logistic regression and inference on a common log-odds ratio from several 2  2 tables. Explicit error rates are not given. Davison et al. (2006) argue that p  approximates the mid-P-value from the conditional distribution with error Oðn 1 Þ and verify this numerically for several examples including a single 2  2 table. The conditional distributions they investigate are not close to degenerate, nor is the data near the boundary of the sample space. What does p  mean when the conditional distribution approaches degeneracy? Pierce and Peters (1999) argue that it approximates what they call an ‘approximately conditional’ P-value. After reading this literature, one is left with the impression that issues of conditionality are automatically resolved by using p  . Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/jspi Journal of Statistical Planning and Inference ARTICLE IN PRESS 0378-3758/$ - see front matter & 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.jspi.2009.10.007 E-mail address: c.lloyd@mbs.edu Journal of Statistical Planning and Inference 140 (2010) 1073–1081