Journal of Applied Psychology 1989, Vol. 74, No. l, 168-170 Cq,Jyright 1989 by the American Psycholoti.cal ASM>Ciation, Inc. 0021-9010/89/$00. 75 Meta-Analysis and Moderator Variables: A Cautionary Note on Transportability Edward R. Kemery Department of Management University of Baltimore Kevin W. Mossholder Department of Management Auburn University William P. Dunlap Tulane University Despite the recent popularity or meta-analysis as a tool for summarizing empirical results across a number or studies, surprisingly little research has been conducted on the accuracy or these proce- dures under a variety or population conditions. or concern in this study was the 90% credibility value (Pearlman. Schmidt, & Hunter, 1980) advocated as a rule or thumb regarding the transportability or employment test validities. We investigated the ability or this meta-analytic rule to detect the pres- ence or discretely defined moderator variables, that is, the ability or the rule to detect instances where transportability is inappropriate. An infinite sample size analysis and a mathematical proor demonstrated that the transportability rule may produce erroneous inferences at rates higher than expected. Meta-analysis has become an increasingly pqiular tool in the social sciences for summarizing and making inferences from a number of studies. Using the empirical study rather than the individual as the unit of analysis, meta-analysis has been used in two ways. First, it has been used descriptively to summarize (in terms of a mean and standard deviation) the results from many studies. Second, meta-analysis has been used inferentially to make statements regarding sources of between-study vari- ance in effect sizes. The end product ofinferential meta-analysis is a summary statement about the relationship between two constructs that is based on a consideration of a number of stud- ies. Such statements tend to be cast in one of three forms. One form has been that the "true" relationship between constructs is a particular value, and that most, if not all, of the discrepan- cies between studies can be attributable to statistical artifacts. In other words, it is inferred that the focal relationship does not differ across contexts (i.e., persons or situations). A second type of statement has been that, although discrepancies between studies cannot be entirely explained by artifacts, the population correlation is probably not zero. The logic of this kind of state- ment is based on the 90% confidence interval concept (e.g., Pearlman, Schmidt, & Hunter, 1980). That is, if the Iower- bound credibility value excludes zero, then it is reasonable to conclude that the population effect size is not zero. This argu- ment has been used largely when meta-analysis has been used to investigate the transportability of test validity (Pearlman et al., 1980). A third conclusion drawn from meta-analysis has We wish to thank the three anonymous reviewers for their helpful comments on earlier versions of this article. Correspondence concerning this article should be addressed to Ed- ward R. Kemery, Department of Management, Robert G. Merrick School of Business, University of Baltimore, 1420 Charles Street, Balti- more, Maryland 2120 l. 168 been that transportability should not be inferred. That is, strong evidence of contextual moderation h!!s been found. Comparatively few investigations have focused on meta-anal- ysis procedures themselves; however, because meta-analysis is becoming increasingly popular, it should be scrutinized to pre- vent misapplication. Whereas some studies have examined data-gathering problems in meta-analysis (e.g., Bullock & Svy- antek, 1985), others have investigated inferential issues. Such studies have raised some concerns about meta-analysis as a tool for discovering moderator variables. Contrary to the assumption that validities are unimodal, Al- gera, Jansen, Roe, and Vijn (1984) suggested that there might be several subpopulations of employment test validities. James, Demaree, and Mulaik (1986) noted that several basic psycho- logical theories suggest that behavior is determined by person variables, environmental variables, and their interaction, mak- ing it reasonable to hypothesize that employment test validities could be moderated by person and environmental variables. From the perspective of these challenges, it is important to ex- amine the conditions under which meta-analysis will yield ac- curate inferences regarding the presence of moderator vari- ables. The research that has examined the acceptability of meta- analysis inferences has seldom explicitly included moderator variables. Much of the Monte Carlo research conducted on meta-analysis can be characterized either as model develop- ment (Raju & Burke, 1983) or as confirmation that particular procedures yield accurate mean rho and variance estimates un- der the implicit assumption of a continuous moderator (i.e., when there is variance about some "population" value). For ex- ample, Callender, Osburn, Greener, and Ashworth ( 1982) tested algorithms for estimating rho and sigma given a continuous un- derlying distribution of effects and found that, under the as- sumed condition of one pqiulation rho (with or without vari- ance), meta-analysis was reasonably accurate. If a variable