Wavelet-Based Nonparametric Modeling of Hierarchical Functions in Colon Carcinogenesis Jeffrey S. MORRIS, Marina VANNUCCI, Philip J. B ROWN, and Raymond J. CARROLL In this article we develop new methods for analyzing the data from an experiment using rodent models to investigate the effect of type of dietary fat on O 6 -methylguanine-DNA-methyltransferase (MGMT), an important biomarker in early colon carcinogenesis. The data consist of observed proles over a spatial variable contained within a two-stage hierarchy, a structure that we dub hierarchical functional data. We present a new method providing a unied framework for modeling these data, simultaneously yielding estimates and posterior samples for mean, individual, and subsample-level proles, as well as covariance parameters at the various hierarchical levels. Our method is nonparametric in that it does not require the prespecication of parametric forms for the functions and involves modeling in the wavelet space, which is especially effective for spatially heterogeneous functions as encountered in the MGMT data. Our approach is Bayesian; the only informative hyperparameters in our model are effectively smoothing parameters. Analysis of this dataset yields interesting new insights into how MGMT operates in early colon carcinogenesis, and how this may depend on diet. Our method is general, so it can be applied to other settings where hierarchical functional data are encountered. KEY WORDS: Bayesian method; Carcinogenesis; Functional data analysis; Hierarchical model; Model averaging; Nonparametric regres- sion; Wavelet. 1. INTRODUCTION 1.1 Colon Carcinogenesis Studies Colorectal cancer is a major international health problem. It is the third most common cancer worldwide and the second- leading cause of cancer deaths in the United States. Because colon cancer is often asymptomatic until it is advanced, and current treatment of advanced disease has limited effectiveness, the development of preventive approaches is crucial in ghting this disease. A large part of this effort entails gaining a better understandingof the mechanismsunderlyingcolon carcinogen- esis, identifying important risk factors, and understanding how they operate. There are indications that environmental factors (most no- tably, diet) play a primary role in the development of colon cancer (see, e.g., Giovannucci and Willet 1994). Carcinogen- induced colon cancer in rodent models are extensively used to delineatemechanisms in colon carcinogenesis.In these models, rodents are fed particular diets of interest for a specic period, exposed to a carcinogen known to induce colon cancer, and then later euthanized, with their colons removed and examined for carcinogenic responses. In modeling biological mechanisms in the colon, it is im- portant to consider the special architecture of cells within the colon. Colon cells replicate and spend their entire life cycles Jeffrey S. Morris is Assistant Professor, Department of Biostatistics, Uni- versity of Texas M.D. Anderson Cancer Center, Houston TX 77030 (E-mail: jeffmo@mdanderson.org ). Marina Vannucci is Associate Professor, Depart- ment of Statistics, Texas A&M University, College Station TX 77843 (E-mail: mvannucci@stat.tamu.edu ). Philip J. Brown is the Pzer Professor of Medical Statistics, Institute of Mathematics and Statistics, University of Kent, Canter- bury, Kent CT2-7NZ, U.K. (E-mail: Philip.J.Brown@ukc.ac.uk ). Raymond J. Carroll is Distinguished Professor, Department of Statistics and Department of Epidemiology and Biostatistics, Texas A&M University, College Station TX 77843 (E-mail: carroll@stat.tamu.edu ). Vannucci’s research was supported by National Science Foundation CAREER award DMS-0093208. Carroll’s re- search was supported by a grant from the National Cancer Institute (CA-57030), and by the Texas A&M Center for Environmental and Rural Health (CERH) via a grant from the National Institute of Health (P30-ES09106). The rst author thanks Phil Brown for his hospitality while visiting the University of Kent at Canterbury under EPSRC Visiting Fellowship grant GR/R52770/01. The au- thors extend sincere thanks to Joanne Lupton, Robert Chapkin, Nancy Turner, and Mee Young Hong, who have helped invaluably with substantive issues in- volved with this work. They also thank the editor and referees, whose insightful comments and questions have dramatically improved this article. within crypts, ngerlike structures that grow into the wall of the colon. An individual cell is “born” in a region known as the stem cell region, toward the bottom of the crypt, and moves up the crypt wall as it matures and differentiates, until it is nally exfoliated out into the lumen at the end of its natural life cycle. This special cell-life sequence in the crypts suggests two impor- tant facts. First, cells at the same relative depths within differ- ent crypts will tend to share common biological characteristics. Second, cells at different depths of a given crypt are at different stages of maturity and could in principle react differently to car- cinogens and other stimuli. As a result, it is important to study biological measurements in the colon as a function of relative cell position, because averaging over all crypt cells obscures any potential depth-specic effects. The relative cell position t is dened such that the bottom of each crypt has t D 0 and the top has t D 1, with positions in between coded proportionally. 1.2 Application Epidemiologic and animal studies have suggested that diets high in sh oil fats, or n-3 polyunsaturatedfatty acids, have a protective effect against colon cancer when compared with di- ets high in corn oil fats (e.g., Boyle, Zaridze, and Smans 1985). However, the biological mechanisms behind this observed ef- fect remain unknown and are of considerableinterest. Nutrition researchers at Texas A&M University (Hong et al. 2001) inves- tigated how dietary fat type affects the initiation stage of colon carcinogenesis (the rst few hours after exposure to a carcino- gen). During this stage, the carcinogen exposure leads to dam- age to the cells’ DNA, which, if not either repaired or removed, may eventually lead to cancer. The biological responses of in- terest in this study include DNA adduct levels, which quantify the amount of carcinogen-induced damage;O 6 -methylguanine- DNA-methyltransferase (MGMT), measuring the amount of an enzyme that can repair this damage; apoptosis, the elimination of damaged cells; and BCL2, a protein related to apoptosis. In this study, 30 rats were randomized to a diet high in ei- ther sh oil or corn oil. After being fed these diets for 2 weeks, each rat was exposed to the carcinogen azoxymethane (AOM) © 2003 American Statistical Association Journal of the American Statistical Association September 2003, Vol. 98, No. 463, Applications and Case Studies DOI 10.1198/016214503000000422 573