Cross-classified multilevel models (CCMM) in health research: A systematic review of published empirical studies and recommendations for best practices

Abstract

Recognizing that health outcomes are influenced by and occur within multiple social and physical contexts, researchers have used multilevel modeling techniques for decades to analyze hierarchical or nested data. Cross-Classified Multilevel Models (CCMM) are a statistical technique proposed in the 1990s that extend standard multilevel modeling and enable the simultaneous analysis of non-nested multilevel data. Though use of CCMM in empirical health studies has become increasingly popular, there has not yet been a review summarizing how CCMM are used in the health literature. To address this gap, we performed a scoping review of empirical health studies using CCMM to: (a) evaluate the extent to which this statistical approach has been adopted; (b) assess the rationale and procedures for using CCMM; and (c) provide concrete recommendations for the future use of CCMM. We identified 118 CCMM papers published in English-language literature between 1994 and 2018. Our results reveal a steady growth in empirical health studies using CCMM to address a wide variety of health outcomes in clustered non-hierarchical data. Health researchers use CCMM primarily for five reasons: (1) to statistically account for non-independence in clustered data structures; out of substantive interest in the variance explained by (2) concurrent contexts, (3) contexts over time, and (4) age-period-cohort effects; and (5) to apply CCMM alongside other techniques within a joint model. We conclude by proposing a set of recommendations for use of CCMM with the aim of improved clarity and standardization of reporting in future research using this statistical approach.

1. Introduction

There has been substantial interest among researchers in understanding multilevel phenomena, including how features of the physical and psychosocial environment in which individuals live, learn, work, and play are associated with individual health, disease, and behavior (Mair, Diez Roux, & Galea, 2008; Pickett & Pearl, 2001). This interest can be traced partially to theoretical works describing the nesting of individuals in multiple social ecologies (see for example Bronfenbrenner & Morris, 2006; Dunn, Masyn, Yudron, Jones, & Subramanian, 2014; Krieger, 2001; Stokols, 1996) as well as advances in statistical modeling, namely multilevel modeling (MLM) or hierarchical linear modeling (HLM) (Aitkin, Anderson, & Hinde, 1981; Diez Roux, 1998, 2002; Raudenbush & Bryk, 2002; Subramanian, Jones, & Duncan, 2003). The major achievement of these multilevel methods is that they address weaknesses of standard multiple regression by taking into account the hierarchical or nested structures that arise naturally in most kinds of social data (Fielding & Goldstein, 2006). In doing so, multilevel models enable researchers to expand focus beyond individual-level characteristics to investigate the social and geographic contexts or groups to which individuals belong.

Despite these benefits, a major limitation of standard multilevel models is that they require researchers to study contexts configured according to strict hierarchies, for example a clean hierarchical nesting of students in classrooms and classrooms in schools (see Fig. 1). To overcome this limitation, educational researchers Goldstein and Rasbash (Goldstein, 1994; Rasbash & Goldstein, 1994) developed statistical software and approaches known as cross-classified multilevel models (CCMM). CCMM are an extension of standard hierarchical multilevel models and allow researchers to examine more complex data structures to determine, among other things, what proportion of total variance is attributable to multiple contextual (or area level) units of analysis that do not follow a strict hierarchical structure. These types of data structures are common in social, behavioral, and health data and are referred to as cross-classified data. For example, children may be nested simultaneously in schools and neighborhoods of residence where no clear hierarchies exist between schools and neighborhoods (see Fig. 2). The analytical extensions represented by CCMM are important because multilevel studies are vulnerable to ‘omitted context bias’ (Evans, Onnela, Williams, & Subramanian, 2016; Goldstein, 1994; Meyers & Beretvas, 2006) whereby the variance associated with relevant omitted contexts is misattributed, in part, to the context(s) included in the regression model.

Fig. 1.

Hierarchical multilevel data structure.

XFig. 2.

Cross-classified multilevel data structure.

Excellent resources describing the CCMM approach are available to researchers wishing to capitalize upon its capacity to more accurately model the social reality represented in complex data structures (see, e.g., Goldstein, 2011; Leckie & Bell, 2013). Yet the extent to which this versatile approach has been utilized in empirical health research has not been adequately characterized and remains unknown. An improved understanding of this is necessary to identify emerging issues in the literature that could be addressed in future research and to identify areas of the empirical health literature in which the approach has thus far been underutilized. To address this gap in the literature, we conducted a scoping review to characterize when, why, and how CCMM have been applied to multilevel data in empirical health research. Informed by these findings, we additionally conclude with recommendations for best research practices in the use of CCMM for health researchers wishing to employ this innovative statistical approach.

2. Methods

The aim of a scoping review is to map the body of literature pertaining to a broad topic (Levac, Colquhoun, & O'Brien, 2010). As recommended by Levac et al. (2010), the review began with the establishment of a research team consisting of individuals with expertise in applied CCMM and social epidemiology who provided both methodological and substantive expertise. In coordination with a reference librarian trained in library science, the research team then developed the overall study protocol, including identification of search terms and databases.

2.1. Data sources and search strategy

Steps for the review follow those outlined by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and reflect the synergistic nature of the review process (Moher et al., 2015). Fig. 3 provides a flowchart summarizing the article review and exclusion process. We started with a systematic search of articles published as of February 1, 2018 in the following eight databases: ABI/Inform, Academic Search Premier, Cab Abstract, EconLit, ERIC, PsycINFO, PubMed, and Web of Science. In these databases, we searched for “cross-classified multilevel model.” In addition, searches included instances in which “cross classified” or “cross-classified” was listed eight or fewer words apart from either “multilevel” or “multi-level” in order to identify publications where “cross classified”, with or without the hyphen, was not listed immediately with “multilevel” or “multi-level.” Occasionally authors use CCMM but do not mention it in either the title or abstract; in order to attempt to identify these articles, which would not be captured by the search criteria, we also searched for articles by examining the reference pages of published empirical articles focusing primarily on health. In addition, given that disciplines outside of the health sciences refer to CCMM approaches with different naming protocols, we also searched using the terms “cross-classification” and “cross-classified random effects models,” to be inclusive of studies using this statistical technique under different naming strategies. We acknowledge that despite these attempts, the search terms used here are imperfect and may not have fully captured studies using other disciplinary-specific naming strategies for CCMM. Therefore, we may not have been able to identify all papers using this statistical technique. In addition, in a project of this scope it is always possible (and perhaps even likely) for researchers to misclassify or misunderstand a published piece. We take full responsibility for any misrepresentations of the original study in our synthesis and categorizations.

Fig. 3.

Record Identification and Exclusion.

2.2. Citation management

All citations were imported into the bibliographic manager EndNote, with this software automatically removing duplicate citations. Further duplicates were removed manually when found later in the process.

2.3. Inclusion/exclusion criteria

To be included, studies had to be written in English and use CCMM for at least two levels of analysis. Studies using multiple membership multilevel models were excluded from analysis. Multiple membership multilevel models are closely-related to cross-classified multilevel models in that both can handle non-hierarchical data, however multiple membership models differ methodologically and substantively from CCMM so are not included in this review. From this pool of English-language CCMM studies, we reviewed titles and abstracts to determine whether a study was either a non-empirical or empirical CCMM study. Non-empirical studies (such as theory papers, simulation studies, review articles, and meta-analyses) were excluded. Empirical CCMM studies were further sorted by domain of outcome examined: health, education, and other. For the health studies, we allowed for a range of diverse health and health behavior outcomes, including: mental health, subjective well-being, self-rated health, anthropometric measures, cancer and cardiovascular health, physical activity, and mortality. Included records came from peer-reviewed journal articles, book chapters and dissertations. Abstracts, conference posters, and commentaries were excluded.

2.4. Study selection

Following this initial review process using the abstract and title screening process, we conducted a full-text review of all empirical health studies. Reviewers discussed records throughout this screening process to reach consensus on any uncertainties related to study selection (Levac et al., 2010). After initial full-text review of these publications, it became apparent that some authors examining age, period, and cohort (APC) effects referred to them as hierarchical rather than cross-classified models. To ensure we were not missing literature that referred to CCMM as hierarchical, we conducted a secondary search. The basis of this search was a paper by Yang and Land (2006) that proposed the use of CCMM to examine APC effects. This additional search comprised all citations listed in Google Scholar that cited the original Yang and Land 2006 paper. This resulted in 18 additional empirical health CCMM studies.

2.5. Data extraction

For each study from the empirical health literature, the following information was extracted: author(s), year, publication journal, health outcome, data source, study participants, sample size, reason for use of CCMM, and the multilevel data structure (e.g., student, school, neighborhood). In addition to indexing the studies, the date of publication for each record allowed us to assess growth in the literature over time and the number of studies published by health outcome. For APC studies, the use of CCMM has been highly controversial. We therefore also noted timing of publication for APC studies relative to the publication of the original major critique of this approach (Bell & Jones, 2014) and the extent to which authors addressed this controversy.

2.6. Data summary and synthesis

Data were extracted into a single spreadsheet in Microsoft Excel (2010) for validation and coding for health topic and methodological rationale for use of CCMM. The reporting of this review conforms to recommendations from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher et al., 2015).

3. Results

3.1. Records identified for review

The search methods yielded 1404 articles (1385 from database searches and 19 from other sources) with 1277 remaining after duplicates were removed. Following title and abstract screening, an additional 847 were subsequently excluded as they were either non-English (n = 5) or did not use CCMM (n = 842). Of 430 publications that did address CCMM, a further 312 were excluded as they focused on topics unrelated to empirical health research: non-empirical CCMM on any domain (n = 79); empirical CCMM studies from education (n = 95) or other non-health outcomes (n = 138). Research areas covered in this latter category included studies from a wide range of disciplines such as economics, political science, sociology, psychology, demography, transportation studies, criminal justice, ecology, forestry, and agriculture. Following the exclusion of these studies, a total of 118 empirical CCMM studies focusing on health outcomes were included in this review. Table 1 provides information on the author, year of publication, journal, health outcome examined, study sample and participants, sample size, data source and structure, and the reason for use of CCMM for non-APC studies. Table 2 provides similar information for studies applying CCMM for APC analysis.

Table 1.

Summary of empirical health studies using cross-classified multilevel models (CCMM) by reason for use of CCMM.

Authors	Journal	Outcome	Country	Sample Sizea	Data Source	Data Structure
Reason for Use of CCMM: Concurrent Contexts
Aminzadeh et al. (2013)	Social Science & Medicine	Subjective well-being	New Zealand	N = 5567	Youth 2007 Health Survey	L1: Individuals L2a: Schools L2b: Neighborhoods
Bozorgmehr et al. (2015)	Journal of Epidemiology and Community Health	Disease management program enrollment	Germany	N = 1280	Epidemiological Study for the Prevention, Early Diagnosis and Optimal Treatment of Chronic Diseases in an Elderly Population (ESTHER)	L1: Individuals L2a: Neighborhoods L2b: General Practitioners
Cafri and Fan (2018)	Statistical Methods in Medical Research	Hip implant survival	USA	N = 13,920	Kaiser Permanente Total Joint Replacement Registry	L1: Individuals L2a: Hospitals L2b: Surgeons
Carroll-Scott et al. (2015)	American Journal of Public Health	Body mass index (BMI)	USA	N = 811	Community Interventions for Health Study; School Administrative Records; US Census; School Learning Environment Survey	L1: Individuals L2a: Schools L2b: Neighborhoods
Chaix et al. (2012)	PLoS ONE	Body mass index (BMI); Waist circumference	France	N = 7131	Residential Environment and Coronary Heart Disease	L1: Individuals L2a: Neighborhoods L2b: Supermarket
Cheung, Goodman, Leckie, and Jenkins (2011)	Children and Youth Services Review	Externalizing behaviors	Canada	N = 1063	Ontario Looking after Children (OnLAC) project	L1: Individuals L2a: Foster Families L2b: Foster Care Workers
Chum and O'Campo (2013)	Health & Place	Cardiovascular disease risk	Canada	N = 1626	Neighborhood Effects on Health and Well-being Study	L1: Individuals L2a: Residential Neighborhoods L2b: Workplace Neighborhoods
De Clerq et al. (2014)	Social Science & Medicine	Tobacco use	Belgium	N = 8453	2005-6 Health Behavior in School-Aged Children Study	L1: Individuals L2a: Communities L2b: Schools
Di Martino et al. (2016)	BMJ Open	Adherence to chronic polytherapy after Myocardial Infarction	Italy	N = 9606	Hospital Information System (patient records)	L1: Individuals L2a: Hospitals of Discharge L2b: Primary Care Providers
Di Martino et al. (2017)	Journal of Chronic Obstructive Pulmonary Disease	Adherence to long acting bronchodilators	Italy	N = 13,178	Hospital Information System (patient records)	L1: Individuals L2a: Hospitals of Discharge L2b: Local Health District
Dundas, Leyland, and Macintyre (2014)	American Journal of Epidemiology	Self-rated health and mental health	Scotland	N = 6285	Aberdeen Children of the 1950s Study	L1: Individuals L2a: Schools L2b: Families L3: Neighborhoods
Dunn, Milliren, Evans, Subramanian, and Richmond (2015)	American Journal of Public Health	Depressive symptoms	USA	N = 16,172	National Longitudinal Study of Adolescent to Adult Health (Add Health)	L1: Individuals L2a: Schools L2b: Neighborhoods
Dunn, Milliren, et al. (2015)	Health & Place	Tobacco use	USA	N = 16,070	National Longitudinal Study of Adolescent to Adult Health (Add Health)	L1: Individuals L2a: Schools L2b: Neighborhoods
Ecob et al. (2004)	International Journal of Methods in Psychiatric Research	Ratings of clinical and psychosocial needs of patients (Health of the Nation Outcome Scales (HoNOS))	UK	N = 384	Referrals to Secondary Care Psychiatric Services	L1: Individuals L2a: Primary Care Practice L2b: Health Care Professionals in Secondary Site
Ecochard and Clayton (1998)	Statistics in Medicine	Conception; pregnancy	France	N = 1901	Center for the Study and Preservation of Eggs and Sperm	L1: Ovulation Cycles L2a: Women L2b: Sperm Donors
Gifford and Foster (2008)	Medical Care	Inpatient length of stay for a given admission	USA	N = 8400	Tennessee Impact Study	L1: Hospital Admissions L2a: Children L2b: Facility
Groenewegen et al. (2018)	Health & Place	All-cause morbidity	Netherlands	N =  1,159,929	NIVEL Primary Care Database; Dutch Integral Safety Monitor 2011; Statistics Netherlands	L1: Individuals L2a: General Practitioner Practices L2b: Neighborhoods L3: Municipalities
Gross, Herrin, Wong, and Krumholz (2005)	Journal of Clinical Oncology	Likelihood of enrollment in a cancer trial	Australia	N = 36,167	National Cancer Institute Clinical Trial Evaluation Program Database	L1: Individuals L2a: Protocols L2b: Centers L2c: Counties
Hofer et al. (2004)	BMC Health Services Research	Reliability of physician-assessed patient quality of care	USA	N = 496 physician reviews	Veterans' Medical Records	L1: Quality of Care scores L2a: Patient Records L2b: Physician Reviewers
Kendler, Ohlsson, Sundquist, and Sundquist (2015)	Social Psychiatry and Psychiatric Epidemiology	Drug abuse	Sweden	N =  1,089,940	Primary Health Care Register, the Total Population Register, the Swedish Hospital Discharge Register, and the Multi-Generation Register	L1: Individuals L2a: Households L2b: Districts
Langford and Bentham (1996)	Social Science & Medicine	Age standardized mortality ratio (SMR)	England and Wales	Not Reported	Unspecified nationally-representative data (1989–1991)	L1: Districts L2a: Regions L2b: ACORN Sociodemographic Category
Milliren, Richmond, Evans, Dunn, and Johnson (2017)	Substance Abuse: Research and Treatment	Marijuana Use	USA	N = 18,329	Longitudinal Study of Adolescent to Adult Health (Add Health)	L1: Individuals L2a: Schools L2b: Neighborhoods
Moore et al. (2013)	Journal of Epidemiology and Community Health	Body mass index (BMI); Waist circumference	USA	N = 1503	Multi-Ethnic Study of Atherosclerosis	L1: Individuals L2a: Workplace Environments L2b: Neighborhoods
Muntaner et al. (2004)	International Journal of Occupational and Environmental Health	Depression	USA	N = 473	Primary Data Collection	L1: Individuals L2a: Nursing Home Workplaces L2b: County of Residence
Muntaner et al., 2006	Health & Place	Depressive symptoms	USA	N = 241	Primary Data Collection	L1: Observation Occasion L2a: Nursing Assistants L2b: Nursing Home Workplaces L3: County of Residence
Muntaner, Li, Xue, Thompson, Chung, et al. (2006)	Social Science & Medicine	Depressive symptoms	USA	N = 341	Primary Data Collection	L1: Observation Occasion L2a: Nursing Assistants L2b: Nursing Home Workplaces L3: County of Residence
Muntaner, Li, Ng, Benach, and Chung (2011)	International Journal of Health Services	Self-reported health; Activity limitations; Alcohol use; Caffeine consumption	USA	N = 868	Primary Data Collection	L1: Observation Occasion L2a: Nursing Assistants L2b: County of Residence
Pedan, Varasteh, and Schneeweiss (2007)	Journal of Managed Care Pharmacy	Adherence to statins	USA	N = 6436	Blinded Computerized Pharmacy Prescription Records	L1: Individuals L2a: Physicians L2b: Pharmacies
Pedersen (2017)	Journal of Youth and Adolescence	Alcohol use; Heavy episodic drinking	Norway	N = 10,038	Young in Oslo Study	L1: Individuals L2a: Schools L2b: Neighborhoods
Penfold, Deena, Benedict, and Kelleher (2008)	International Journal of Health Geographics	Risk of perforated appendicitis	USA	N = 8086	Ohio Hospital Association	L1: Individuals L2a: Zip Code of Residence L2b: Hospitals
Pilbery, Teare, Goodacre, and Morris (2016)	Emergency Medicine Journal	Correct reading of an ECG	UK	N = 254	Recognition of STEMI (ST-segment elevation myocardial infarction) by Paramedics and the Effect of Computer Interpretation Feasibility Study	L1: Observation Occasion L2a: Individuals L2b: ECG Machines
Pruitt et al. (2014)	Cancer Epidemiology, Biomarkers & Prevention	Colorectal cancer screening	USA	N = 3898	John Peter Smith Health System	L1: Individuals L2a: Primary Care Physicians L2b: Clinics L2c: Neighborhoods
Ratnapradipa et al. (2017)	Diseases of the Colon & Rectum	Laparoscopic colon cancer resection	USA	N = 10,618	National Cancer Institute's Surveillance, Epidemiology, and End Results database; Medicare claims Data (2008–2011); 2010 US Census; 2008–2012 American Community Survey	L1: Individuals L2a: Hospitals L2b: Counties
Richmond, Dunn, Milliren, Evans, and Subramanian (2016)	Obesity	Body mass index (BMI)	USA	N = 18,200	National Longitudinal Study of Adolescent to Adult Health (Add Health)	L1: Individuals L2a: Schools L2b: Neighborhoods
Riva, Gauvin, Apparicio, and Brodeur (2009)	Social Science & Medicine	Physical activity	Canada	N = 2716	Unnamed larger study in Montreal	L1: Individuals L2a: Census Tracts L2b: Active Living Potential Walkability Zones
Schootman et al. (2014)	Annals of Surgical Oncology	Colorectal surgery outcomes	USA	N = 35,946	2000–2005 National Cancer Institute's Surveillance, Epidemiology and End Results; 1999–2005 Medicare Claims	L1: Individuals L2a: Hospitals L2b: Census Tracts
Teitler and Weiss (2000)	Sociology of Education	Sexual activity	USA	N = 2080	Philadelphia Teen Survey 1993 Wave	L1: Individuals L2a: Schools L2b: Neighborhoods
Thomas, Rahman, Mor, and Intrator (2014)	American Journal of Managed Care	Rehospitalization	USA	N =  1,382,477	Medicare Claims and Enrollment Records; Minimum Data Set; Online Survey Certification and Reporting Dataset; Hospital Compare; American Hospital Association Database	L1: Patients L2a: Hospitals L2b: Nursing Home
Townsend (2012)	International Journal of Obesity	Body mass index (BMI); Waist circumference	England	N =  788,525	National Child Measurement Program	L1: Individuals L2a: Schools L2b: Neighborhoods
Tsugawa (2018)	Dissertation	Health care spending	USA	N =  434,616	Medicare Beneficiaries	L1: Hospitalizations L2a: Hospitals L2b: Physicians
Utter (2011)	American Journal of Public Health	Physical activity	New Zealand	N = 9107	Youth 2007 Health Survey	L1: Individuals L2a: Schools L2b: Neighborhoods
Virtanen et al. (2010)	American Journal of Epidemiology	Long-term sick leave	Finland	N = 3063	Finnish 10-Town Study	L1: Individuals L2a: Communities L2b: School Workplaces
Weich et al. (2017)	The Lancet Psychiatry	Rates of compulsory admission to inpatient psychiatric beds	England	N =  1,238,188	2010–11 Mental Health Minimum Data Set (MHMDS)	L1: Patients L2a: Hospitals L2b: Neighborhoods
West, Sweeting, and Leyland (2004)	Research Papers in Education	Drinking; Smoking; Illicit drug use; Unhealthy eating	Scotland	N = 2000+	West of Scotland 11 to 16 Study	L1: Individuals L2a: Schools L2b: Neighborhoods
Wilk et al. (2018)	SSM - Population Health	Physical activity	England	N = 1517	Grade 5 ACT-i-Pass	L1: Individuals L2a: Schools L2b: Neighborhoods
Williams et al., 2016	PLoS ONE	Body mass index (BMI)	USA	N = 16,956	National Child Measurement Program	L1: Individuals L2a: Residential Neighborhoods L2b: School Neighborhoods
Young (2010)	International Journal of Health Geographics	Birth weight	USA	N = 1449	Cape Cod Family Healthy Study	L1: Individuals L2a: Communities L2b: Families
Reason for Use of CCMM: Account for Data Structure
Aerenhouts, Clarys, Taeymans, and Cauwenberg (2015)	PLoS ONE	Body fat percentage	Belgium	N = 69	Primary Data Collection	L1: Observation Occasions L2a: Individuals L2b: Measurement Instruments
Ali et al. (2007)	Health & Place	Vaccine uptake	Vietnam	N = 56,076	Diseases of the Most Impoverished program	L1: Individuals L2a: Schools L2b: Neighborhoods
Atkin, Sharp, Harrison, Brage, and Van Sluijs (2016)	Medicine and Science in Sports and Exercise	Physical activity	UK	N = 704	Millennium Cohort Study	L1: Observation Occasions L2a: Individuals L2b: Seasons
Atkin, Sharp, et al. (2016)	PLoS ONE	Sedentary Behavior	UK	N = 264	Sport, Physical Activity, and Eating Behavior: Environmental Determinants in Young People (SPEEDY study)	L1: Individuals L2a: Primary Schools L2b: Secondary Schools
Becker et al. (2016)	Journal of Hand Surgery	Classification of severity of trapeziometacarpal (TMC) arthrosis	Global	N = 92	Science of Variation Group 8, 2014-5	L1: Observation Occasion L2a: Patients' Radiographs L2b: Hand Surgeons
Block, Christakis, O'Malley, and Subramanian (2011)	Qual Quant	Body mass index (BMI)	USA	N = 3113	Framingham Heart Study (Offspring Cohort)	L1: Observation Occasion L2a: Individuals L2b: Neighborhoods
De Meester, Van Dyck, De Bourdeaudhuij, Deforche, and Cardon (2014)	BMC Public Health	Physical activity	Belgium	N = 736	Primary Data Collection	L1: Individuals L2a: Primary Schools L2b: Secondary Schools
D'Haese, Dyck, Bourdeaudhuij, Deforche, and Cardon (2014)	International Journal of Behavioral Nutrition and Physical Activity	Physical activity	Belgium	N = 606	Belgian Environmental Physical Activity Study (BEPAS-child)	L1: Individuals L2a: Schools L2b: Neighborhoods
D'Haese et al. (2016)	International Journal of Behavioral Nutrition and Physical Activity	Physical activity	Belgium	N = 494	Belgian Environmental Physical Activity Study (BEPAS-child)	L1: Individuals L2a: Schools L2b: Neighborhoods
Doumouras et al. (2017)	British Journal of Surgery	Non-technical skills of surgeons and anesthetists	USA	N = 26	Primary Data Collection	L1: Individuals L2a: Simulation Raters L2b: Training Simulation
Hooiveld et al. (2016)	Environmental Health	Morbidity	Netherlands	N =  197,096	Dutch Agricultural Geographic Information System & electronic medical record data registered by Dutch general practitioners	L1: Individuals L2a: General Practitioner Practices L2b: Neighborhoods
Kirby (2008)	Social Forces	Access to health care	USA	N = 22,682	Medical Expenditure Panel Survey (MEPS); U.S. Census; Health Resources and Services Administration	L1: Individuals L2a: Census Block Groups L2b: Primary Care Service Areas
Li et al. (2015)	Academic Emergency Medicine	Length of stay in emergency department	Australia	N = 27,656	Linked Emergency Department-Laboratory Information Systems database	L1: Individuals L2a: Emergency Departments L2b: Years
Linton, Jennings, Latkin, Kirk, and Mehta (2014)	Health & Place	Injection drug use	USA	N = 1510	AIDS Linked to the Intravenous Study (ALIVE)	L1: Individuals L2a: Neighborhoods at Time 1 L2b: Neighborhoods at Time 2
Meunier, Bisceglia, and Jenkins (2012)	Developmental Psychology	Children's oppositional and emotional problems	Canada	N = 809	Primary Data Collection	Not reported
Schofield, Das-Munshi, Mathur, Congdon, and Hull (2016)	Psychological Medicine	Depression	UK	N =  410,541	General Practitioner Patient Health Records from four ethnically diverse London boroughs: Lambeth, Hackney, Tower Hamlets and Newham	L1: Individuals L2a: Neighborhoods L2b: General Practitioner Practices
Sharp, Denney, and Kimbro (2015)	Social Science & Medicine	Self-rated health	USA	N = 1147	Los Angeles Family and Neighborhood Survey (LAFANS)	L1: Observation Occasion L2a: Individuals L2b: Neighborhoods
Sink, Hope, and Hagadorn (2011)	Archives of Disease in Childhood - Fetal and Neonatal Edition	Oxygen saturation target achievement	USA	N = 1019	Quality Improvement Oximeter Data and Neonatal Intensive Care Unit Nurse Administrative Data and Bedside Patient Flowsheets	L1: Monitoring Period L2a: Infants L2b: Nurses
Vigil et al. (2017)	Pain	Pain intensity & emergency room prioritization	USA	N =  129,991	Veterans' Health Administration Corporate Data Warehouse	Not reported
Winpenny et al. (2017)	Appetite	Dietary intake	UK	N = 351	Sport, Physical Activity and Eating behavior: Environmental Determinants in Young People	L1: Individuals L2a: Primary Schools L2b: Secondary Schools
Reason for Use of CCMM: Contexts Over Time
Gustafsson et al. (2017)	Health & Place	Functional somatic symptoms	Sweden	N = 920	Northern Swedish Cohort Study	L1: Individuals L2a: Neighborhoods at 1981 L2b: Neighborhoods at 1986 L2c: Neighborhoods at 1995 L2d: Neighborhoods at 2007
Huijts and Kraaykamp (2012)	International Migration Review	Self-rated health	31 European Countries	N = 19,210	European Social Surveys (2002–2008)	L1: Individuals L2a: Country of Origin L2b: Country of Destination L2c: Immigrant Community
Leyland and Naess (2009)	Journal of the Royal Statistical Society: Series A	Mortality	Norway	N = 49,736	Oslo Census Information from 1960, 1970, 1980 and 1991	L1: Individuals L2a: Area in 1960 L2b: Area in 1970 L2c: Area in 1980 L2d: Area in 1990
Morton et al. (2016)	International Journal of Behavioral Nutrition and Physical Activity	Physical activity	England	N = 325	Sport, Physical Activity, and Eating Behavior: Environmental Determinants in Young People	L1: Individuals L2a: Primary Schools L2b: Secondary Schools
Murray et al. (2013)	American Journal of Epidemiology	Physical capability (chair rise, grip strength, balance)	England	N = 2566	National Survey of Health and Development	L1: Individuals L2a: Residential Area (age 4) L2b: Residential Area (age 26) L2c: Residential Area (age 53)
Ohlsson and Merlo (2011)	Social Science & Medicine	All-cause mortality; Ischemic heart disease mortality and morbidity; Cancer mortality and morbidity; Respiratory diseases and related mortality	Sweden	Not reported	National Registers	L1: Individuals L2a: Parish of Residence in 1969 L2b: Parish of Residence in 1979 L2c: Parish of Residence in 1989 L2d: Parish of Residence in 1999
Urquia et al. (2009)	American Journal of Public Health	Birth weight	Canada	N = 22,189	Canadian Institute for Health Information (1993–2000)	L1: Individuals L2a: Census Tracts L2b: Country of Origin
Urquia, Frank, and Glazier (2010)	Social Science & Medicine	Birth weight	Canada	N =  320,398	Discharge Abstract Database of the Canadian Institute for Health Information	L1: Infants L2a: Migrant Mother's Country of Birth L2b: Migrant Mother's Country of Last Permanent Residence
Urquia, Frank, Moineddin, and Glazier (2011)	Journal of Urban Health	Preterm birth	Canada	N =  397,470	Discharge Abstract Database of the Canadian Institute for Health Information	L1: Births L2a: Neighborhoods L2b: Maternal Country of Origin
Zaccarin and Rivellini (2002)	Statistical Methods & Applications	Decision to have second child	Italy	N = 1092	World Fertility Survey Project of the International Statistical Institute and Fertility and Family Survey Project of the European Economic Commission	L1: Individuals L2a: Birth Place L2b: Current Residence
Reason for Use of CCMM: Innovative Applications
Bell, 2014b	Social Science & Medicine	Mental Health	UK	N = 21,142	British Household Panel Survey, 1991–2009	L1: Observation Occasion L2a: Household Year L2b: Period L2c: Individual L3a: Cohorts L3b: Local Authority District
Congdon and Best (2000)	Applied Statistics	Emergency in-patient admissions	UK	N = 335 ward-practice catchment zones	Hospital records from Barking and Havering Health Authority (UK); unspecified ward-level data	L1: Hospital Admissions L2a: Ward-Practice Catchment Zones L2b: Primary Care Practices L2c: Hospitals
Evans et al. (2016)	Social Science & Medicine	Body mass index (BMI)	USA	N = 14,144	National Longitudinal Study of Adolescent to Adult Health (Add Health)	L1: Individuals L2a: Social Network Communities L2b: Schools L2c: Neighborhoods
Fischbach et al. (2002)	International Journal of Epidemiology	Helicobacter pylori treatment success	Global	N = 618 treatment groups	Meta-analysis of studies examining treatment regimens to eliminate H. pylori from human subjects	L1: Treatment Groups L2a: Studies L2b: Treatment Regimens

^aSample size unit is human respondents, unless otherwise noted.

^bBell, 2014 is classified as an Innovative Methods paper and so is included in Table 1. However it is also an APC paper.

Table 2.

Summary of empirical health studies examining Age-Period-Cohort (APC) effects.

Authors	Journal	Outcome	Country	Data Source	Did the Authors Address the Controversies over APC Models, particularly those raised by Bell and Jones (2014)?
Ananth, Keyes, and Wapner (2013)	BMJ	Pre-eclampsia	USA	US Centers for Disease Control and Prevention National Hospital Discharge Survey Data	NO: published before Bell and Jones (2014). Did acknowledge the identification problem and used two approaches. CCMM was a robustness check on the first approach.
Bardo 2015a	Dissertation	Self-rated health	USA	General Social Survey	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Beck, Finch, Lin, Hummer, and Masters (2014)	Social Science & Medicine	Self-rated health	USA	National Health Interview Survey	NO: published concurrently with the Bell and Jones (2014).
Bell and Jones (2018)	Qual Quant	Body mass index (BMI)	USA	National Health Interview Survey	YES: this is a paper in the dialogue between the two factions and uses the approach only to illustrate issues of concern.
Chaurasiya (2018)	Asian Journal of Epidemiology	Non-communicable disease	India	National Sample Survey (NSS) in India	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Delaruelle, Buffel, and Bracke (2015)	Social Science & Medicine		32 European countries	European Social Survey	YES: mentions the identification problem and cites Bell and Jones (2014); attempts to address the issue through theory-informed sensitivity analyses.
Fu and Land (2015)	Population Research and Policy Review	Overweight; Obesity	China	China Health and Nutrition Survey	YES: acknowledges the controversy and cites Bell and Jones (2014); attempts to address the issue through sensitivity analyses.
Giordano et al. (2014)	Drug and Alcohol Dependence	Hospitalization for drug abuse	Sweden	National Swedish Hospital Discharge Register	NO: published concurrently with the Bell and Jones (2014).
Hidehiro, Ken, Yoko, Shizuko, and Masaya (2016)	Population Health Metrics	Self-rated health	Japan	Comprehensive Survey of Living Conditions	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Keyes, Gary, O'Malley, Hamilton, and Schulenberg (2019)	Social Psychiatry & Psychiatric Epidemiology	Depressive symptoms	USA	Monitoring the Future	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Kraus et al. (2015)	Alcohol and Alcoholism	Alcohol use	Sweden	1979 Scandinavian Drinking Survey (SDS-79), the 1995 Nordic Survey of Alcohol and Narcotics (NSAN-95), the 2003 Alcohol and Narcotics Survey (ANS-03) and the 2005, 2007, 2009 and 2011 Swedish Alcohol Monitoring Survey (AMS-05/11)	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Krieger et al. (2014)	Epidemiology	Mortality	USA	National Center fro Health Statistics Mortality Data, and US Census Data	NO: published concurrently with the Bell and Jones (2014).
Kwon and Schafer (2016)	SSM -Population Health	Self-rated health	China	World Value Survey-China	YES: acknowledges the controversy and cites Bell and Jones (2014); attempts to address the issue through sensitivity analyses.
Lin, Beck, and Finch (2014)	Journals of Gerontology, Series B: Psychological Sciences and Social Sciences	Disability	USA	National Health Interview Survey	NO: published concurrently with the Bell and Jones (2014).
Lin, Beck, and Finch (2016)	Disability and Health Journal	Disability	USA	National Health and Nutrition Examination Survey (NHANES)	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Livingston et al. (2016)	Addiction	Alcohol use	Australia	Australian National Drug Strategy Household Survey (NDSHS)	YES: cite the Bell and Jones paper; argue that linear effects are less likely to be an issue in this case (based on supplemental analyses).
Luo, Pan, Sloan, Feinglos, and Wu (2015)	Preventing Chronic Disease	Tooth loss	USA	National Health and Nutrition Examination Survey (NHANES)	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Masters (2012)	Demography	Mortality	USA	National Health Interview Survey-Linked Mortality Files	NO: published prior to controversy.
Masters, Hummer, and Powers (2012)	American Sociological Review	Mortality	USA	National Health Interview Survey-Linked Mortality Files	NO: published prior to controversy.
Masters et al. (2013)	American Journal of Public Health	Mortality	USA	National Health Interview Survey-Linked Mortality Files	NO: published prior to controversy.
Piontek, Kraus, Müller, and Pabst (2010)	Sucht: Zeitschrift für Wissenschaft und Praxis	Cigarette use prevalence and frequency	Germany	German Epidemiological Survey of Substance Abuse	NO: published prior to controversy.
Piontek, Kraus, Pabst, and Legleye (2012)	Journal of Epidemiology & Community Health	Cannabis use prevalence and frequency	Germany	German Epidemiological Survey of Substance Abuse	NO: published prior to controversy.
Reither, Hauser, and Yang (2009)	Social Science & Medicine	Body mass index (BMI)	USA	National Health Interview Survey	NO: published prior to controversy.
Ryan-Ibarra et al. (2016)	Annals of Epidemiology	Lifetime asthma diagnosis	USA	California Behavioral Risk Factor Surveillance System	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Sugisawa et al. (2016)	International Journal of Nephrology & Renovascular Disease	Dialysis complications and depressive symptoms in dialysis patients	USA	Original Data Collected	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Tang (2014)	Social Indicators Research	Subjective well-being	China	World Values Survey	NO: published concurrently with the Bell and Jones (2014).
Teisl, Lando, Levy, and Noblet (2016)	Food Control	In-home food preparation safety	USA	The Food Safety Survey	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Thibodeau (2015)	Population Review	Suicide mortality	Canada	Statistics Canada	YES: acknowledged as a limitation.
Thorpe et al. (2016)	Journal of Urban Health	Hypertension; Diabetes; Stroke; Cardiovascular Disease	USA	National Health Interview Survey	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Twenge, Sherman, and Wells (2017)	Archives of Sexual Behavior	Number of sexual partners since age 18	USA	General Social Survey	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Volken, Wieber, Ruesch, Huber, and Crawford (2017)	Public Health	Self-rated health	Switzerland	Swiss Health Survey (1997, 2002, 2007 and 2012)	YES: acknowledges the controversy and referenced Bell and Jones (2014) book chapter which suggested modifications to the HAPC approach and strong assumptions that would enable the analysis to be conducted.
Wilk, Maltby, and Cooke (2017)	Canadian Studies in Population	Body mass index (BMI)	Canada	Canadian Community Health Survey (CCHS)	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Willson and Abbott (2018)	Australian and New Zealand Journal of Public Health	Body mass index (BMI)	New Zealand	New Zealand National Health Surveys	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.
Yu et al. (2016)	BMJ Open	Disability	Hong Kong	Hong Kong Elderly Health Centers (EHCs) of the Department of Health	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Zhang (2017)	Journal of Religion and Health	Self-rated health	USA	General Social Survey	NO: published after Bell and Jones (2014) but does not cite them; does discuss the identification problem but claims CCMM addresses the issue.
Zheng, Yang, and Land (2011)	American Sociological Review	Self-rated health	USA	National Health Interview Survey	NO: published prior to controversy.
Zheng et al. 2016b	Population Research and Policy Review	Mortality	Australia, Belgium, Denmark, England and Wales, Finland, France, Iceland, Italy, Japan, Netherlands, New Zealand, Norway, Sweden, Switzerland, and U.S.A.	Human Mortality Database	NO: published after Bell and Jones (2014) but does not cite them; also does not mention the identification problem.

^aBardo, 2015 is a dissertation that includes a variety of analyses. Bardo examined “domain satisfaction” (which is related to subjective well being but focuses on specific domains, including satisfaction with residence, hobbies, family, friends and health. Because health was examined we included this in our review even though it is not exclusively focused on health). In Chapter 3 these outcomes are treated separately, including a CCMM specifically for the health satisfaction outcome. This is the main model we reviewed.

^bZheng, Yang, & Land, 2016 (Popul Res Policy Rev) conduct an atypical APC analysis where level 1 units are mortality rates in groups rather than outcomes for individual respondents.

3.2. Increasing but limited use of CCMM, and patterns in the literature

As shown in Fig. 4, heath researchers have increasingly adopted CCMM in empirical studies. Since the publication of the seminal CCMM paper in 1994 by Goldstein, a total of 118 empirical health articles had been published by the time of our scoping review. Most papers (n = 71; 60%) were published since 2014.

Fig. 4.

CCMM Empirical Health Publications by Year*. The main review was conducted through Feb 1, 2018, however several additional APC papers were identified in the APC secondary literature search after this date, including one study from 2019.

The 118 papers came from 54 different journals and a range of health sub-disciplines: medicine, public health, and the social sciences. A large variety of data sources were used, from large nationally-representative cohort studies to smaller data sets collected to answer specific research questions. The countries in which the data were collected largely represent the global north, especially English-speaking nations, which likely reflects both our inclusion criteria of English-language articles, and data availability within these countries.

Across reviewed studies, the language used to describe data structures was highly non-standardized. For example, some authors would describe the data structure represented in Fig. 2, with children nested in schools and cross-classified by neighborhoods, as a two-level model, while others would describe this as a three-level model. These inconsistencies in description became particularly challenging when data structures were especially complex. There were also considerable inconsistencies in reporting of other vital information, such as sample sizes for each level of analysis.

3.3. CCMM used across multiple health domains

The 118 studies covered topics that fell into 17 health domains, as presented in Table 3. The most-often examined health outcome in CCMM studies is body weight (n = 14), using measures of body mass index (BMI) and waist circumference. The next most-often examined health outcomes were self-rated individual health or subjective wellbeing (‘general health’) (n = 12) and substance use (n = 11). As an example of a general health study, Aminzadeh et al. (2013) used CCMM to examine the relationship between neighborhood social capital and adolescent subjective wellbeing, as measured by questions on general mood, life satisfaction and the WHO-5 Wellbeing Index. Mental health (n = 10) was the next most examined health outcome. These four domains comprised 40% of the total studies included in the review. Other health domains covered by CCMM studies include: physical activity (n = 9); medical services (n = 9); medical care quality (n = 9); mortality (n = 8); morbidity and disease outcomes (n = 8); sexual and reproductive health (n = 5); infant health (n = 5); physical capability (n = 4); pregnancy or other reproductive health (n = 3); adherence to treatment (n = 3); sexual activity (n = 2); diet (n = 1); a combination of these topics (n = 5); and other topics (n = 5).

Table 3.

Type of health outcome evaluated by reviewed studies.

Outcome Type	Frequency	Percent
Body Weight	14	11.9
General Health/Self-rated Health	12	10.2
Substance Use	11	9.3
Mental Health	10	8.5
Physical Activity	9	7.6
Medical Services	9	7.6
Medical Care Quality	9	7.6
Mortality	8	6.8
Morbidity & Specific Disease Outcomes	8	6.8
Sexual and Reproductive Health	5	4.2
Infant Health	5	4.2
Other	5	4.2
Combinations of Other Categories	5	4.2
Physical Capability	4	3.4
Adherence to Treatment	3	2.5
Diet	1	0.8
Total	118	100

Common levels of analysis included in these studies include individuals (e.g., patients, survey respondents, students), institutions (e.g., schools, hospitals), area (e.g., neighborhood, district), and time (measurement occasion or cohort).

3.4. Five rationales for use of CCMM

Across all 118 studies included in the review, cross-classified multilevel modeling was correctly chosen as an analytic technique to handle clustering or grouping in non-hierarchical data. The research questions asked by investigators using clustered data did, however, vary in five key ways. These reflect different rationales for and uses of CCMM in empirical health research, as outlined below. We note that researchers may have multiple research questions in each study and therefore may use CCMM for multiple reasons. We chose to categorize the studies by their primary reason for use of CCMM, as shown in Table 3.

3.4.1. Examining concurrent contextual effects

Some authors had substantive interest in the clustering within the data, and as such, an examination of contextual effects was a key aim of the study. As shown in Fig. 5, this was the most common reason for use of CCMM (40%) (i.e., to assess the contributions that concurrent contexts (e.g., schools and neighborhoods) have on individual-level health outcomes). For example, Di Martino et al. (2016) used CCMM to assess the extent to which variation in patient treatment adherence (Level 1) after myocardial infarction was attributable to hospitals (Level 2a) or primary care providers (Level 2 b). In studies such as these, results from the random effects, or variance components, of the CCMM are of primary consideration.

Fig. 5.

Uses of CCMM in empirical health literature.

3.4.2. Accounting for non-independence in the data structure

Other authors had no substantive interest in the ways in which observations were related to each other, but used CCMM to account for data structure, namely the non-independence of data values. In these instances, the authors chose CCMM as a way to adjust standard errors to account for clustering. Nearly a fifth (17%) of CCMM studies use the technique to account for clustering in the data structure without a particular focus on the random effect estimates. For instance, Ali et al. (2007) examined vaccine uptake among students in Vietnam. The authors accounted for cross-classification of individuals (Level 1) within both schools (Level 2a) and neighborhoods (Level 2b) to address the data's non-hierarchal structure and to account for the potential variation in vaccine uptake by school and neighborhood. Previous research indicates that ignoring one of these levels does not drastically affect fixed-effect point estimates, but importantly, could cause standard errors to be underestimated (Fielding, 2002; Raudenbush & Bryk, 2002). This impacts the statistical significance inferences associated with the fixed-effect parameter estimates and may lead to increased Type I error (Meyers & Beretvas, 2006). However, as opposed to reporting results from the variance components of the models, many authors included only fixed-effects coefficients from the independent variables included in models. In this sense, authors employing CCMM to account for non-independence in the data structure seek to account for contextual effects rather than to explore them substantively.

3.4.3. Examining contextual effects over time

CCMM are also used to examine longitudinal relationships that vary across time or by measurement occasion, including in studies adopting a life course perspective. We define life course studies as those in which the predictor variable was measured at multiple points in time (e.g., socioeconomic status measured in childhood, adolescence, adulthood). Longitudinal studies are those in which the outcome was measured at multiple occasions. In longitudinal studies with strict hierarchical levels, observation occasions (Level 1) are nested cleanly within individuals (Level 2). However, this strict hierarchical structure is not always present in the data. For example, in a Swedish study by Gustafsson, Bozorgmehr, Hammarstrom, and San Sebastian (2017), individuals were assessed in 1981, 1986, 1995, and 2007, with some respondents residing in different neighborhoods at each time point. Given individuals were not nested in the same neighborhood at each of the four time points, CCMM were necessary to accurately model the cross-classified data structure.

3.4.4. Age-period-cohort models

Age-Period-Cohort analyses are the second-most popular use for CCMM (31%). Generally, APC analyses attempt to disentangle the contributions of the age of individuals, the period of time they are surveyed in, and their birth cohort to explain individual-level outcomes. In CCMM APC models, individuals (Level 1) are nested in periods (Level 2a) cross-classified by cohorts (Level 2 b), and age is treated as a level 1 fixed effect. For instance, in a Chinese age-period-cohort model fit by Tang (2013), respondents were surveyed in four time periods: 1990, 1995, 2001, and 2007. Tang defined seven cohorts: Children of Old China (born 1908–1938), Children of New China (born 1939–1946), the “Lost” Generation (1947–1955), Children of the Early Cultural Revolution (1956–1960), Children of the Late Cultural Revolution (1961–1966), Children of Economic Reform (1967–1976), and Children of Opening-Up (born 1977 and later). Individuals were then cross-classified according to which cohort they belonged to and what time (or period) they were surveyed in. Their age at the time of the survey was included in the model as an individual-level fixed effect.

For decades, a notorious problem in the APC literature has been the “identification problem” (e.g., Fienberg & Mason, 1979; Glenn, 1976), whereby an exact linear dependency exists between the three measures of temporal effects: Period = Age + Cohort. Yang proposed CCMM as a solution to this issue because the model setup would break the dependency between the three factors by treating two as random effects and one as a fixed effect. However, critiques of Yang and Land's (2006) claim soon emerged, beginning notably with a critique by Bell and Jones (2014). A long exchange has now occurred in the literature between proponents and opponents of the CCMM approach (see Bell & Jones, 2018 for a succinct summary). Briefly, opponents of the CCMM approach show that results can vary strongly as an artefact of how the data were collected; when there is a large ratio of cohorts to periods, or vice versa, random effects can vary dramatically in favor of one or the other (Bell & Jones, 2018). Thus, study findings may not be attributable to substantive APC processes, but rather to the way in which the data were collected (Bell & Jones, 2018).

Results from this review indicate that, of 37 CCMM APC studies captured in our review, 24 were published after the 2014 critique by Bell and Jones (see Fig. 6).

Fig. 6.

Age-Period Cohort CCMM Studies in the Empirical Health Literature by Year. * The main review was conducted through Feb 1, 2018, however several additional APC papers were identified in the APC secondary literature search after this date, including one study from 2019.

Of the 24 CCMM studies published after the critique by Bell and Jones, only 7 (29%) acknowledged the controversy, and one of these was a follow-up paper by Bell and Jones (2018). Studies that do acknowledge the controversy often do not do so satisfactorily. We find, for example, that they may reference this substantive methodological debate only within the limitations section of the manuscript rather than meaningfully describing the ways in which findings could be affected within the methods or discussion sections. Of the remaining 17 studies (74%), more than half (n = 9) failed to acknowledge the identification problem and the rest seemed to assume that CCMM would resolve the issue. This is concerning from a methodological perspective because it indicates both that CCMM has become very popular for APC analysis in the empirical health literature, and that most scholars are either unaware of the methodological debate or are not engaging with the debate in a thoughtful manner.

3.4.5. Innovative applications of CCMM

CCMM are often applied to novel research questions and most studies that use this statistical approach could be considered innovative. A small set of studies (3%) did, however, apply the method in unique or unusual ways. These highly innovative applications included combining CCMM with other modeling techniques, as in Congdon and Best (2000) who integrated elements of gravity models to represent flows of patients between area-practice combinations and hospital units into CCMM. Fischbach, Goodman, Feldman, and Aragaki (2002) applied cross-classified modeling in a meta-analysis of clinical trials, in which treatment groups (level 1) were cross-classified by study (Level 2) and treatment regimen (Level 2). Further information about each study that used CCMM in an innovative manner is provided in Table 1.

4. Discussion

Motivated by both theoretical and methodological concerns, health researchers have increasingly employed cross-classified multilevel modeling methods. These advanced statistical methods are highly flexible tools for modeling data with complex structures and are appropriate and advantageous for addressing a wide range of substantive and methodological issues. Additionally, the approach can allay concerns about omitted context bias (Goldstein, 1994; Meyers & Beretvas, 2006). Despite its many advantages and the fact that it has been nearly three decades since the approach was proposed, we found only 118 empirical health studies that used this approach.

The range of topics examined within these empirical CCMM studies is diverse, with some health outcomes examined to a greater extent than others. Of the 118 reviewed studies, the health topics most commonly examined were body weight, self-rated health and wellbeing, substance use, and mental health. Other health outcomes, such as treatment adherence and diet were the least-examined topics. Treatment adherence and diet are widely discussed in the health literature, but our findings highlight a paucity of research designed to understand the multiple and intersecting contexts that influence an individual's ability to adhere to medical treatment and select the foods they consume. The relatively limited use of CCMM in empirical health studies indicates a fruitful avenue for future research, and its increased application is possible in a number of statistical packages including: SAS (proc mixed) (Hox, 2010); SPSS (mixed) (IDRE, 2020); Stata (xtmixed) (Hox, 2010); R (lmer) (Bolker, 2020); and MLwiN (Leckie & Bell, 2013); among others. For researchers wishing to use CCMM, we further provide a number of recommendations and best practices to improve clarity and standardize reporting within CCMM studies.

4.1. Recommendations

4.1.1. Explicitly state rationale for use of CCMM and sample sizes at all levels

Authors should indicate whether their use of CCMM is “out of necessity” (i.e., to account for clustering in the data) or “out of substantive interest” (i.e., to examine variation explained in a health outcome by multiple and intersecting contexts), or both. This is, surprisingly, not always clear in some studies, particularly in brief descriptions provided in abstracts. Clarity on this issue will enable readers to more easily determine the purpose(s) of a study.

Similarly, information should be provided about the sample size at all levels (e.g., number of individuals, number of schools, number of neighborhoods) regardless of the stated rationale for use of CCMM. Sample size information was inconsistently reported, yet this information is vital to the evaluation of potential study limitations in multilevel models and CCMM (Dedrick, Ferron et al., 2009; McNeish & Stapleton, 2016; Milliren, Evans, Richmond, & Dunn, 2018).

4.1.2. Describe cross-classified data structure

Regardless of the rationale for use of CCMM, it is essential to clearly communicate the multilevel structure of the data and the model(s). Three main strategies may be used to communicate the data structure. At minimum, one of these three strategies should be used, however we recommend that authors consider using all three.

First, the use of figures such as unit and classification diagrams (shown in Fig. 1, XFig. 2) is an effective and efficient way to communicate data structures. For more complex data structures, such as three-level models where only two contexts are cross-classified, a diagram is the most appropriate tool for communicating model structure because even carefully worded textual descriptions may be unclear. For example, in the study by Bell (2014) the data structure was highly complex; the visualization used in the study was a clear and effective way to communicate the data structure.

Second, we recommend including a full equation in classification notation (Browne, Goldstein, & Rasbash, 2001; Fielding & Goldstein, 2006). Although CCMM equations can sometimes be quite complex, they accurately reflect the cross-classified nature of the data and efficiently document subtle modeling choices made by researchers. Please see Appendix A for an example for a CCMM equation using classification notation.

Finally, we encourage a clear description of the data and model structure within the text, noting both the clustering units and the level at which each cluster is nested. Variation in this practice exists and we therefore encourage standardization around the description provided by Goldstein (1994) in his original paper on CCMM. Goldstein referred to cross-classified contexts as residing at the same level (e.g., cross-classified at level 2), rather than as being separate levels. This helps to distinguish between strict hierarchical data structures found in traditional multilevel modeling and the less strict cross-classified hierarchies present in data structures used in CCMM. In the strict hierarchical data structure visualized in Fig. 1, three levels are present: students (Level 1) nested in schools (Level 2), which are in turn nested in neighborhoods (Level 3). In the case of cross-classified data structures, as in Fig. 2 where there are three distinct types of units, this should be considered a two-way cross-classification, with two units of analysis (schools and neighborhoods) both existing at Level 2 because they are each a single “step” away from the lowest unit of analysis (Level 1, individuals). In this instance, we would recommend describing the data structure using language similar to the following: individuals (Level 1) are nested within a two-way cross-classification of schools (Level 2a) by neighborhoods (Level 2b). In both the strict hierarchical and less-strict cross-classified data structures, the numeric value for higher levels (or clustering units) corresponds with the number of steps away from the lowest unit of analysis.

Descriptions such as these will greatly clarify for readers of CCMM manuscripts the cross-classified nature of the data used in analysis. We hasten to add that not all statistical software for CCMM analysis uses this naming convention suggested by Goldstein, perhaps due to the software's original development for use in traditional multilevel models with strict hierarchical data structures. Depending on the software used, researchers specifying a cross-classified multilevel model may need to code “Level 2b” of the cross-classified data as “Level 3.” This misalignment between the recommendations in the literature and the practical considerations of coding language is perhaps not surprising from a software development perspective, but may nonetheless have fueled some of the inconsistencies in the ways in which CCMM model descriptions are described in the literature.

4.1.3. Report variance estimates and other relevant summary statistics

All multilevel models, regardless of whether they are hierarchical or non-hierarchical, allow for a decomposition of the total variance across levels and clustering units. These variance components should be reported for all contextual levels when using CCMM, even if the primary reason for use of CCMM is to account for clustering in the data. The Intraclass Correlation Coefficient (ICC) (also termed Variance Partition Coefficients (VPC) in its most general form) measures the proportion of total variance in the outcome that is attributable to the area or contextual-level (i.e., the correlation between two observations within the same cluster) (Merlo, Chaix et al., 2006). For linear CCMM, this is calculated as VPC = V_A/(V_A + V₁), where V_A is the area level variance and V₁ corresponds to the individual-level variance. As compared to linear models, in CCMM for binary and other discrete responses, there is no single ICC or VPC value as the level 1 variance is a function of the mean (Leckie & Charlton, 2013). A popular solution is to formulate the model in terms of a continuous latent response variable (i.e., the linear threshold model method) which underlies the observed binary response (Leckie & Charlton, 2013; Merlo et al., 2006). In this case, VPC is calculated as: ${\sigma }_u^2/({\sigma }_u^2+{\pi }^2/3)$.

In models with more than one area or contextual level unit of analysis, a separate VPC estimate is calculated for each clustering unit (e.g., one for classrooms, one for schools, one for neighborhoods). The VPC estimates for each area level included in a model should be reported in the manuscript, regardless of whether the area level was of substantive research interest to the investigators. In other words, even if CCMM was used only to account for clustering in the data structure, investigators using CCMM should report VPC estimates for each unit of analysis at the higher level(s).

4.1.4. Engage with the APC controversy

As discussed above, a substantial controversy has surrounded APC analysis in general and the use of CCMM for APC analysis in particular. Bell and Jones (2018) have done much to explicate the debate, the ‘identification problem,’ and the methodological concerns. Despite this, the vast majority of researchers continue to employ CCMM for APC analysis without reference to the identification problem, the controversy itself, or any of the latest recommendations for best practices. Those that do refer to the identification problem often note this only within the limitations section of the manuscript. In light of the ongoing debate surrounding these methods, however, we urge substantial caution when conducting APC analysis and recommend a more meaningful engagement with the logic underlying the controversy. If researchers decide to apply the CCMM approach to the “identification problem,” they should—at a minimum—engage seriously with this literature, specify how their data structure may influence their findings, and detail their reasons for believing that the analysis they are conducting yields unbiased estimates. Researchers are also encouraged to stay abreast of the controversy and to consult recently published recommendations that center on this topic (Bell, 2020).

4.2. Conclusion

Health and human behavior are shaped by multiple and intersecting social and physical contexts—contexts that rarely form perfect hierarchies. Cross-classified multilevel modeling allows researchers to model these complex realities. This statistical technique may be and is used to: account for complex data structures, answer research questions related to concurrent geographic and social contexts, and contexts over time, and to incorporate a hybrid of statistical applications within one model. Results from this review indicate that CCMM are used to examine a wide range of health topics and domains and that the use of CCMM in health research has expanded in recent years. Despite its increased use, this flexible approach remains relatively underutilized. This leaves much room for research investigations that employ this technique to more precisely model the complex causal architecture of individual health outcomes. Recommendations proposed in this review can improve clarity and standardization of CCMM studies that seek to comprehensively understand the causes of health and disease in human populations.

Ethics statement

This review article did not involve human subjects research. As such, this research did not require ethical approval from an Institutional Review Board.

Appendix A. Equation in classification notation

The following equation uses notation for cross-classified multilevel models proposed by Fielding and Goldstein (2006). It represents a random intercepts cross-classified model predicting outcome (y) that nests individuals (Level 1, denoted as i) within a three-way cross-classification of context 1 (Level 2a, denoted as j₁), by context 2 (Level 2 b, denoted as j₂), and bycontext 3 (Level 2c, denoted as j₃):

$$y_{i\left(j_{1,}{j}_{2,}{j}_3\right)}={\beta }_0{X}_{0i\left(j_{1,}{j}_{2,}{j}_3\right)}+{\beta }_x{X}_{xi\left(j_{1,}{j}_{2,}{j}_3\right)}+\left(u_{0\left(j_1\right)}+u_{0\left(j_2\right)}+u_{0\left(j_3\right)}+e_{0i\left(j_{1,}{j}_{2,}{j}_3\right)}\right)$$

where:

• $y_{i\left(j_{1,}{j}_{2,}{j}_3\right)}$is the outcome of individual i who is nested in cross-classified context at level 2a (j₁), context at level 2 b (j₂), and context level 2c (j₃);

• ${\beta }_0{X}_{0i\left(j_{1,}{j}_{2,}{j}_3\right)}$is the precision-weighted grand mean of the outcome across all three contexts, holding covariates constant;

• ${\beta }_x{X}_{xi\left(j_{1,}{j}_{2,}{j}_3\right)}$is a vector of individual-level covariates and their associated parameter values;u_0(j1) is the random effect parameter for context 2a-level variance (u_0(j1) ~ N (0${\sigma }_{u0}^2$));

• u_0(j2) is the random effect parameter for context 2 b-level variance (u_0(j2) ~ N (0${\sigma }_{u0}^2$));

• u_0(j3) is the random effect parameter for context 2c-level variance (u_0(j3) ~ N (0${\sigma }_{u0}^2$));

• $e_{0i\left(j_{1,}{j}_{2,}{j}_3\right)}$is the random effect for the individual (e_0i ~ N (0${\sigma }_{e0}^2$))