Developmental Continuity of Oppositional Defiant Disorder Subdimensions at Ages 8, 10, and 13 Years and Their Distinct Psychiatric Outcomes at Age 16 Years

Abstract

Objective

To test the developmental continuity, interrelationships, and predictive associations of the oppositional defiant disorder (ODD) subdimensions of irritable, headstrong, and hurtful.

Method

Data were collected from 6,328 mother–child pairs participating in the Avon Longitudinal Study of Parents and Children (United Kingdom).

Results

Developmental continuity for each subdimension was strong and interrelationships indicated that headstrong was associated mainly with irritable, whereas irritable did not cross associate with other ODD subdimensions; and hurtful was associated with lower levels of headstrong. With regard to associations at age 16 years, irritable at age 13 years was associated with depression, whereas headstrong at 13 was associated with delinquency and callous attitude; at age 13, hurtful failed to associate with any of the 3 age 16 outcomes.

Conclusions

The results suggest that the ODD headstrong and irritable subdimensions are developmentally distinct, with small cross-over (i.e., headstrong to irritable), and are associated with unique outcomes. Hurtful does not appear to be associated with future maladjustment in children.

Oppositional defiance in youth is a highly prevalent psychiatric condition that strongly associates with a wide range of psychiatric illness, including both emotional (e.g., depression) and externalizing disorders (e.g., conduct disorder, and callous-unemotional traits).^1-3 Because oppositional defiant disorder (ODD) predicts to such a wide range of adjustment difficulties in children, it has been proposed that ODD may be composed of distinct subdimensions that may have different psychiatric outcomes.^4-6

Along these lines, Stringaris and Goodman⁶ proposed and defined 3 a priori subdimensions of ODD: irritable (i.e., temper outbursts, easily annoyed, angry/resentful), headstrong (i.e., argued with grown-ups, rule violations, purposefully annoyed others, blamed others), and hurtful (i.e., been spiteful, tried to get his/her own back on people [a colloquial British expression for vindictive behavior]). Stringaris and Goodman⁶ found that the irritable (or affective) subdimension prospectively associated with emotional problems, peer problems, and, to a lesser extent, conduct problems and a callous disposition toward others, whereas the headstrong (or opposition) subdimension related more strongly to conduct problems and hyperactivity; hurtful (or spitefulness) related more strongly to callousness. It has been proposed that identifying such distinct dimensions may improve clinical prediction of later outcomes and may help to tailor treatments for children with ODD.⁶

Following the Stringaris and Goodman⁶ study, several studies have found support for models that distinguish dimensions within ODD. Some have identified 2 such dimensions (irritable and headstrong; Rowe et al.⁵), whereas others have identified 3 dimensions with either identical or a slightly different symptom structure to that originally described by Stringaris and Goodman,⁶ such as Burke et al.^4,7,8 Most recently, a study compared the proposed models and found strongest support for the 3-factor structure originally proposed by Stringaris and Goodman⁶ and adopted by the DSM-5.⁹

Based on the studies outlined above, it seems like there is good evidence for at least 2 ODD subdimensions (irritable and headstrong); however the developmental distinctiveness of these subdimensions has not been established. For example, the Stringaris and Goodman⁶ study collapsed across ages 5 to 16 years, and therefore did not identify the subdimensions at the respective ages. Similarly, Burke et al.⁷ examined the subdimensions by collapsing data across ages 5 and 8, whereas Rowe et al.⁵ analyzed an accelerated cohort sequential study, and thereby collapsed data at the first wave for participants 9, 11, and 13 years of age. In addition, although Stringaris et al.¹⁰ did confirm ODD subdimensions, the analyses were conducted on waves of data that collapsed different ages. Finally, Ezpeleta et al.,¹¹ and Krieger et al.⁹ performed cross-sectional studies rather than longitudinal studies, with a mean age of 3 years for the former and an age range of 6 to 12 years for the latter.

The current study sought to confirm evidence for the subdimensions of ODD as proposed by Stringaris and Goodman⁶ from late childhood through to early adolescence. More specifically, we investigated, in parallel with the studies outlined earlier, the following: first, the reliability of the factor structure of the ODD subdimensions of irritable, headstrong, and hurtful at ages 8, 10, and 13 years; if these dimensions are to be used in future psychiatric classification it is important to know that they can be reliably measured; second, the degree to which developmental interrelationships of ODD subdimensions demonstrate continuity across this age range; it is important for clinicians and researchers to know whether children who are, say, irritable, will continue to be so over time or whether the boundaries between the dimensions are fluid over time; and third, the degree to which the ODD subdimensions at age 13 related to the age 16 outcomes of depression, conduct problems and callous attitude, controlling for ODD subdimensions at ages 8 and 10; this information is crucial for prediction and for further research into possible interventions.

METHOD

Sample

The Avon Longitudinal Study of Children and Parents (ALSPAC) was established to understand how genetic and environmental characteristics influence health and development in parents and children. All pregnant women resident in a defined area in the southwestern part of England, with an expected date of delivery between April 1, 1991, and December 31, 1992, were eligible and 13,761 women (contributing 13,867 pregnancies) were recruited. These women have been followed up over the last 19 to 22 years.¹² When compared with 1991 National Census Data, the ALSPAC sample was found to be similar to the UK population as a whole.¹³ Ethical approval for the study was obtained from the ALSPAC Law and Ethics Committee and the local research ethics committees. (More detailed information on ALSPAC is available at http://www.bris.ac.uk/alspac/.)

Measures

Irritable, Headstrong, and Hurtful Subdimensions at Ages 8, 10, and 13 Years

Indicators of the 3 potential ODD subdimensions were derived from the Development and Well Being Assessment (DAWBA), a well-validated measure developed for the British Child Mental Health surveys,¹⁴ which was rated by teachers and parents. In addition to generating binary (yes/no) diagnostic indicators, DAWBA algorithms have recently been developed to generate 6-level ordered-categorical measures of the probability of disorder for each of the individual items underlying the diagnoses, ranging from <0.1% to >70%.¹⁵ Evaluated in 2 large-scale national samples, these DAWBA “bands” functioned well as ordered-categorical measures, showed dose–response associations with mental health service contacts, and showed associations with potential risk factors very similar to those of clinician-rated diagnoses.¹⁶

The DAWBA asks 9 separate symptoms of ODD. Each parent- and teacher-rated question is introduced with the stem: “Over the last 6 months, and as compared with other children the same age, has s/he often … .” followed by the specific clause. Children were assigned a diagnosis only if their symptoms were causing significant distress or social impairment. Following the lead of Stringaris and Goodman,⁶ irritable was defined by the following 3 symptoms: has temper outbursts; has been touchy or easily annoyed; and has been angry or resentful (age 8, α = 0.81; age 10, α = 0.83; age 13, α = 0.85). Headstrong was defined by the following 4 symptoms: argued with grown-ups; takes no notice of rules/refused to do as s/he is told; 3) seemed to do things to annoy other people on purpose; and blamed others for his/her own mistakes or bad behavior (age 8, α = 0.86; age 10, α = 0.87; age 13, α = 0.87). ODD hurtful was defined by these symptoms: been spiteful; tried to get his/her own back on people (i.e., been vindictive) (age 8, α = 0.80; age 10, α = 0.82; age 13, α = 0.80).

Depression at age 16 years was derived from the adolescent-reported Mood and Feelings Questionnaire Short Form (SMFQ).¹⁶ The SMFQ is a 13-item self-report questionnaire of symptoms experienced in the previous 2 weeks. Symptoms are coded on a 3-point scale (“true,” “sometimes true,” “not true”), with a range of 0 to 26 (α = 0.91). This scale has been found to have high reliability and validity, and the short form is made up of items that best discriminated depressed and nondepressed children in field trials using structured psychiatric interviews.¹⁷

Conduct problems at age 16 years were measured by mother reports on the Strengths and Difficulties Questionnaire,¹⁸ with the following 4 items: generally obedient, usually does what adults request (reverse coded); often fights with other children or bullies them; often lies or cheats; and steals from home, school, or elsewhere. Items were coded as a 3- point scale (“not true,” “somewhat true,” and “certainly true”) (α = 0.43). It should be noted that the temper outburst item is typically the final measure for the SDQ for conduct problems; however, in the present study, this item was removed to avoid item overlap between this item and the irritable temper tantrum/outburst item.

Callous attitude at age 16 years was measured by mother reports on the Strengths and Difficulties Questionnaire¹⁸ by reverse coding 4 items (i.e., helps others, has 1 good friend, considerate to others, kind to younger children) on the prosocial SDQ scale.¹⁸ Items were coded as a 3-point scale (“not true,” “somewhat true,” “certainly true”) (α = 0.73). These specific items have previously been used as part of a a assessment of callous-unemotional trait in children,¹⁹ but it should be noted that commonly recognized components of callousness are not included in this construct.

Control Variables

Socioeconomic status (SES), partnership status, and age of mother at the birth of the child were reported at 18 weeks postnatal. SES was coded via the Registrar General’s social class scale²⁰; we compared mothers in classes IV and V (low SES) with those in classes I, II, and III. Partnership status reflected the following: no partner; and has a partner. Age of mother was dichotomized to age 19 and younger (coded 1; 4.7% of sample) with all older mothers (coded 0). Maternal education was coded (at 32 weeks antenatal) as none, or CSE or vocational qualifications only (basic school-leaving/vocational qualifications) versus all higher qualification levels; conduct disorder and depression at age 7 years were derived from the DAWBA “bands.” Conduct disorder was reported by parents and teachers and depression by parents only.¹⁵ These diagnoses have been validly associated by environmental risks and psychopathology in the caregiver.^21,22 Children were assigned a diagnosis only if their symptoms were causing significant distress or social impairment.

Attrition and Missing Data

Participants with data for depression, conduct problems, and callous attitude at 16 years were selected for the analysis (n = 6,328). In a multivariate model, we tested the extent to which sex of the child (OR = 0.99; 95% CI = 0.92–1.05), partner status (OR = 2.25; 95% CI = 1.91–2.66), low SES (OR = 1.84; 95% CI = 1.63–2.08), teen pregnancy (OR = 2.61; 95% CI = 2.39–2.85), and maternal education (OR = 2.42; 95% CI = 2.24–2.62) were associated with exclusion from the current analysis, and found that all of these variables were significantly associated with child exclusion in the present analysis. To reduce bias, all of these variables were included as controls in the analysis,²³ as was the sex of the child, as females are more likely to experience depression than are males, and males are more likely to develop antisocial behaviors.²⁴

Statistical Analysis

Analyses proceeded in 3 steps. In step 1, we examined the extent to which the 3 subdimensions fit an overall second-order latent construct of ODD in a confirmatory factor analysis, at ages 8, 10, and 13, years, respectively. We then tested, via nested model comparisons, the degree to which the 3 subdimensions could be differentiated from the overall latent ODD factor at each age.

In step 2, we examined the developmental continuity and interrelationships among the 3 subdimensions in a latent autoregressive cross-lag model. In this modeling approach, each variable in the model is regressed on all of the variables that precede it in time. The autoregressions examined continuity in the same subdimensions at different time points, for example, headstrong at 8, 10, and 13 years. The cross-lags examined the interrelationships among different subdimensions at different time-points, for example, headstrong at 8 years associating with irritable at 10 years. Within-time covariances were also included, for example headstrong and irritable at 8 years.

In step 3, we examined associations from the ODD subdimensions to the age 16 outcomes (depression, conduct problems, and callous attitude). The analytic steps were conducted in Mplus Version 6.21.²⁵ To provide robust estimates and to account for missing values, a full information maximum likelihood estimation with robust standard errors (MLR) was used. Individual model fit was determined through the comparative fit index (CFI) and Tucker-Lewis Index (TLI; acceptable fit > 0.90)²⁶ and root mean square error of approximation (RMSEA; acceptable fit < 0.08).²⁷ Satorra-Bentler scaled χ² difference tests²⁸ were used to test nested model comparisons.

RESULTS

Descriptive Statistics

Study variables were significantly correlated within time. For example, at age 8 years, irritable and headstrong were significantly correlated, as were irritable and hurtful, and headstrong and hurtful. Significant between-time correlations also existed, for example, between headstrong at age 8 years and all 3 ODD subdimensions at age 10 and also between ages 10 and 13.

Step1: Latent Confirmatory Factor Analysis

Table 1 contains the overall model and the nested models tested in this step. In brief, model 1 is the overall ODD latent construct with the 3 latent subdimensions as the indicators of ODD. Model 2a is nested in model 1 and tests whether irritable and headstrong are the same subdimension; the same nested tests were conducted for 2 additional factor models, model 2b (irritable constrained to be the same as hurtful), and model 2c (headstrong constrained to be the same as hurtful). These analyses were performed at ages 8, 10, and 13 years, respectively.

TABLE 1. Fit Statistics and Nested Model Comparisons at Ages 8, 10, and 13 Years (Sex Controlled).

	Difference Test of Relative Fit
Model	df	c	χ 2	CFI	TLI	RMSEA	Model Comparison	cd	Δ χ 2	Δdf	p
Age 8
1. Freely estimated overall ODD	80	1.836	697.803	0.962	0.952	0.035
2a. Constrained: 2 subdimensions (irritable and headstrong)	81	1.851	701.405	0.961	0.953	0.035	2a vs. 1	3.0510	5.6160	1	<.05
2b. Constrained: 2 subdimensions (irritable and hurtful)	81	1.855	792.328	0.956	0.946	0.037	2b vs. 1	3.3750	55.8821	1	<.01
2c. Constrained: 2 subdimensions (headstrong and hurtful)	81	1.846	868.919	0.951	0.940	0.039	2c vs. 1	2.6460	122.0174	1	<.01
Age 10
1. Freely estimated overall ODD		80	1.904	519.587	0.967	0.959	0.032	2a vs. 1
2a. Constrained: 2 subdimensions (irritable and headstrong)	81	1.923	542.248	0.966	0.958	0.033	2b vs. 1	3.4430	15.5240	1	<.01
2b. Constrained: 2 subdimensions (irritable and hurtful)	81	1.936	614.948	0.960	0.951	0.036	2c vs. 1	4.4960	44.7610	1	<.01
2c. Constrained: 2 subdimensions (headstrong and hurtful)	81	1.910	758.522	0.950	0.938	0.040	2a vs. 1	2.3900	192.2525	1	<.01
Age 13
1. Freely estimated overall ODD	80	2.064	477.548	0.965	0.957	0.032	2a vs. 1
2a. Constrained: 2 subdimensions (irritable and headstrong)	81	2.092	475.763	0.966	0.958	0.032	2b vs. 1	2.2246	4.3320	1	<.05
2b. Constrained: 2 subdimensions (irritable and hurtful)	81	2.087	628.450	0.952	0.942	0.037	2c vs. 1	3.9270	82.9937	1	<.01
2c. Constrained: 2 subdimensions (headstrong and hurtful)	81	2.064	695.438	0.947	0.935	0.040	2a vs. 1	2.0640	217.8900	1	<.01

Note: Model 1 is a freely estimated second-order oppositional defiant disorder (ODD) model; model 2a tests whether irritable and headstrong are distinct; model 2b tests whether irritable and hurtful are distinct; model 2c tests whether headstrong and hurtful are distinct. The letter c denotes weighting constant for computing the χ² statistic using robust estimation method; cd denotes weighting constant for difference between 2 χ² values using robust estimation. CFI = comparative fit index; RMSEA = root mean squared error of approximation; TLI = Tucker Lewis Index.

The overall ODD latent factor (model 1) presented adequate fit to the data at the following ages: age 8 years (χ² [80] = 697.803, p < .001; CFI = 0.962; TLI = 0.952; RMSEA = 0.035); age 10 (χ² [80] = 519.587, p < .001; CFI = 0.967; TLI = 0.959; RMSEA = 0.032) and age 13 (χ² [80] = 477.548, p < .001; CFI = 0.965; TLI = 0.957; RMSEA = 0.032). As can be seen in Table 1, nested models 2a, 2b, and 2c resulted in differentiated subdimensions (e.g., model 2a, irritable differs from headstrong at all ages) from the overall ODD latent construct. At ages 8, 10, and 13 years, each of the 3 subdimensions were differentiated from the overall ODD latent model (Table 1).

Step 2: Examining the Developmental Interrelationships of Subdimensions

We examined a latent autoregressive cross lag (ACRL), to assess the degree to which the subdimensions associate with each other (i.e., cross-lags), above and beyond developmental continuity (i.e., autoregressions). The model showed acceptable fit on 3 fit indices (χ² [1,347] = 6,624.877, p < .001; CFI = 0.929; TLI = 0.920; RMSEA = 0.025). This model was the comparison model for all tests presented below.

Figure 1 shows the significant path coefficients in the ARCL model. We highlight 2 main results. First, with regard to autoregressions, the following associations were observed: irritable at 8 years was associated with irritable at age 10 (β = 0.58), which was associated with irritable at age 13 (β = 0.43); headstrong at 8 years was associated with headstrong at age 10 (β = 0.65), which was associated with headstrong at age 13 (β = 0.78); and hurtful at 8 was associated with hurtful at age 10 (β = 0.42), which was associated with hurtful at age 13 (β 0.34). Second, with regard to cross-lags, headstrong at age 10 years was associated with irritable at age 13 (β = 0.21); hurtful at age 8 was significantly associated with lower levels of headstrong at age 13 (β = −0.18); and irritable did not relate to either headstrong or hurtful.

FIGURE 1. Multivariate autoregressive cross-lagged model of longitudinal relationships between subdimensions of oppositional defiant disorder (ODD) and adolescent outcomes of depression, conduct problems, and callous attitude.

Note: Circles denote latent variables. Controls: partnership status, low socioeconomic status (SES), teen pregnancy, maternal education, sex, and age 7 Development and Well Being Assessment (DAWBA) depression and conduct disorder. The resulting population effect sizes are interpreted using the Cohen³⁵ guidelines: an effect of 0.10 is a small effect, an effect of 0.24 is a medium effect, and an effect of 0.37 is a large effect. Significant (*) results only are shown. The numbers 8, 10, 13 denote age in years. Callous16 = callous attitude at 16 years; CP16 = conduct problems at 16 years; Dep16 = depression at 16 years; Head = headstrong; Hurt = hurtful; Irrit = irritable.

Step 3: Associations With Age 16 Outcomes

Figure 1 shows associations from the age 13 subdimensions associating with the age 16 outcomes. irritable at age 13 years was associated with depression; and headstrong at age 13 was associated with conduct problems and callous attitude. Of note, hurtful at age 13 was not associated with any of the 3 age 16 outcomes. In addition to the above steps, we tested for the presence of a sex-by-outcome interaction effect, and found no differences between males and females for irritable, headstrong, and hurtful at 13 years and the age 16 outcomes (i.e., depression [Δχ² = 0.36, Δdf = 1, p = .55]), conduct problems Δχ² = 1.08, Δdf = 1, p = 0.30) and callous attitude (Δχ² = 0.52, Δdf = 1, p = .47).

DISCUSSION

The current study examined, and provided evidence for, the developmental distinctiveness of the ODD subdimensions irritable, headstrong, and hurtful from middle childhood to early adolescence. Therefore, the substructure of ODD may not be age dependent, which is important information for both the identification and treatment of these subdimensions. Moreover, the developmental interrelationships demonstrated that headstrong was associated with irritable, and hurtful was associated with lower levels of headstrong. irritable was uniquely associated with age 16 depression, and headstrong was associated uniquely with age 16 conduct problems and callous attitude, but hurtful was not associated with age 16 outcomes. These results increase our knowledge of ODD subdimensions in 2 main ways.

First, unlike previous studies, we examined the development and interrelationships of the ODD subdimensions through latent variable modeling. We first demonstrated age-dependent reliability (i.e., the confirmatory latent structure) of the ODD subdimensions in the DAWBA clinical assessment. These results therefore support and extend results from previous studies. Each subdimension also showed high developmental continuity. In addition, there were not many developmental interrelationships, which may further highlight the distinctiveness of the ODD subdimensions. We did, however, identify 2 relationships. First, at age 10 years, headstrong was associated with age 13 irritable. Hence it may be the case that headstrong may be “driving” at least some of the variability of irritability in adolescence, which could provide evidence that if headstrong is successfully targeted and treated, then subsequent levels of irritability (and perhaps age 16 depression) can also be reduced.

The second developmental interrelationship was that hurtful (e.g., planning revenge) at age 10 years was associated with lower rates of headstrong at age 13. Moreover, hurtful did not associate with the age 16 outcomes of conduct problems, depression, or callous attitude. Although this may appear to suggest that hurtful may not relate to the other ODD dimensions or outcomes, it may also be the case that relevant outcomes were not assessed, for example, bullying of other youth, especially where instrumental acts of aggression are perpetrated toward peers. These outcomes may associate highly with the hurtful subdimension, above and beyond the other ODD subdimensions and controls included in the present study.

However, headstrong, but not hurtful, was uniquely associated with callous attitude at age 16. These results differ from those of Stringaris and Goodman,⁶ who found that hurtful was primarily associated with callous-unemotional traits. This may in part relate to differences between the studies; although we tested latent structures and controlled for prior ODD subdimensions, previous studies did not. It may also be that different and more temperamental measures, partially included in the ODD subdimensions, may be better indicators of callous-unemotional trait risk. For example, fearless temperament composed of measures of defiance (i.e., headstrong) and boldness (i.e., low fear of novel situations/persons) was associated with conduct problems and callous-unemotional traits.^29,30 Another reason for the inconsistency may be that the measure of callous attitude differed between the studies. For example, the measure used here did not include certain items such as the callous use of others. Therefore, additional research will need to replicate the current study results with a view to examining the validity of the link between hurtful and callous traits.

Although a number of study strengths exist, such as its large sample size, broad scope, longitudinal focus, and inclusion of cross-informant predictions, the findings also need to be viewed in the light of 3 main limitations. First, the measures were brief and could have benefited from more detail. For example, although the our measure of callous attitude included items previously used as part of a callous-unemotional trait assessment in children,²⁰ it should be noted that commonly recognized components of callousness were not included in this construct (e.g., callous use of others). Future research may want to examine whether the present results can be replicated when using a more complete callous-unemotional measure. Second, as most of our measures (including 2 of the outcomes, conduct problems and callous attitude) were based on maternal reports, this raises the possibility of shared method variance. Third, as with most longitudinal cohorts, attrition has occurred in ALSPAC over time. For example, as expected, younger and more socially disadvantaged mothers were more likely to be lost to follow-up. As these predictors of attrition also predict childhood psychopathology, our sample is highly likely to underrepresent the most severely affected children. Of note, a recent ALSPAC cohort study showed that although attrition affected prevalence, rates of anti-social behavior, and related disorders, associations between risks and outcomes remained present, albeit conservative estimates of the likely true effects.³¹

In summary, using a latent autoregressive cross-lag model, we found evidence for 3 ODD subdimensions at ages 8 and 10, and at age 13 years, where developmental interrelationships showed that headstrong was associated with irritable more than the converse. With regard to age 16 outcomes, irritable and headstrong subdimensions at age 13 years were associated with distinct outcomes at age 16 years (i.e., depression and conduct problems/callous attitude, respectively), and hurtful was nonpredictive at each age.

The DSM-5 taskforce has recently been considering adapting nosology to integrate ODD subdimensionality with the intention of improving predictive validity and prognosis. Taken together, current research suggests that ODD is a complex psychiatric problem and that interventions may need to consider both irritable and headstrong in childhood with a view to preventing young people following the course of differential pathways to different negative psychiatric outcomes. This means that clinicians treating ODD and service planners may do well to assess the 3 dimensions described here to predict and tailor the most appropriate treatment for each individual. For example, children scoring high on the irritable subdimension may benefit from tailored early interventions utilising cognitive behavioral therapy, or, where relevant, selective serotonin reuptake inhibitors to reduce their future risk for mood disorders (i.e., depression).

As ODD in youth is associated with adult psychopathology,^32-34 and as DSM-5 proposal recommendations have stressed the need for a developmental understanding of the precursors of adult mental illness, our study results may help to inform lifespan psychiatric research. Further studies testing different theories of ODD subdimensions and their developmental relationships are recommended to help clinicians draw upon replicated results and translate these into evidence-informed treatment approaches.

Clinical Guidance.

• We found that the substructure of oppositional defiant disorder (ODD) may not be age dependent, and this is important for both identification and treatment.

• In addition, current results indicated that, at age 13, highly irritable and headstrong youth are at risk for depression, delinquency, and callous attitude.

• The DSM-5 taskforce has recently been considering adapting nosology to integrate recent findings on ODD subdimensionality with the intention of improving predictive validity and prognosis.

• A recent Brazilian study using an independent sample provides further confirmatory evidence for the same model.

• Individuals may thus require differential clinical interventions, and clinicians may want to assess the 3 dimensions described here to predict and tailor the most appropriate treatment.

• DSM-5 proposal recommendations have stressed the need for a developmental understanding of the precursors of adult mental illness, and our results may inform lifespan psychiatric research.