Measuring Orthodontic Treatment Satisfaction: Questionnaire Development and Preliminary Validation

Abstract

Objectives

The aims of this study were to develop a reliable self-report measure of consumer satisfaction with orthodontic treatment, and to preliminarily assess its validity.

Method

Transcripts of qualitative interviews with patients, their parents, and practicing orthodontists together with items from existing dental satisfaction questionnaires were used to develop a pool of 41 items assessing satisfaction with various aspects of orthodontic care. These items were paired with five-point Likert scales (1=strongly disagree, 5=strongly agree) and were administered to 299 parents of children who had completed orthodontic treatment at two university-based clinics.

Results

Factor analyses and reliability analyses identified three main subscales with high reliabilities: 13 items assessing satisfaction with treatment process (Cronbach’s alpha=.92), seven items assessing satisfaction with psychosocial effects of treatment (Cronbach’s alpha=.87), and five items assessing satisfaction with overall treatment outcome (Cronbach’s alpha=. 79). Relationships among these three subscales and pre- and posttreatment variables were examined in a subset of 86 parent/patients. Forward stepwise regression with backward overlook revealed no significant relationships between any satisfaction subscale and demographic variables. Posttreatment overjet was inversely related to parental satisfaction with orthodontic treatment process (R²=.13; P<.001), and parent satisfaction with treatment outcome (R²=.28; P<.0001). Improvement in esthetics as measured by improvement in IOTN Aesthetic Component scores was positively related to satisfaction with psychosocial outcomes (R²=.28; P<.0001).

Conclusions

The present instrument is reliable and can be used to assess three dimensions of parental satisfaction with their child‘s orthodontic treatment. Relationships between visible orthodontic outcome variables and parent satisfaction provide preliminary validity support for the instrument.

Interest in patient satisfaction with various aspects of health care has grown over the past 25 years. The rise in the consumer rights movement in the United States during the 1970s provided a strong impetus for assessment of patient satisfaction (1). Managed health care has driven the development of satisfaction measures in almost every patient care arena because purchasers of health care plans need evidence that their employees are indeed satisfied with the health care they receive (2). Evidence shows that satisfied patients are more likely to be compliant with their treatment regimens, which should lead to more complete recovery (1). In addition, increased marketing of all health services has led to a focus on patient satisfaction as part of “practice building” (i.e., as a means to increase patient retention and attraction) (3).

In keeping with the increased focus on patient satisfaction, several large-scale studies have been published outlining the major dimensions of patient satisfaction with dental care. Specifically, Murray and Kaplin (4) reported six main dimensions of dental patient satisfaction: general treatment, staff performance, organization/efficiency, convenience, pain, and patient-personnel interaction. In 1982 Davies and Ware (5) developed a 19-item dental satisfaction questionnaire modeled on studies of satisfaction with medical care. The scope of the Davies and Ware instrument was later expanded by increasing the dental satisfaction questionnaire to 52 items (6). Their factor analysis of over 6,000 patient questionnaires produced 13 subscales, ranging from satisfaction with pain control to office atmosphere. Similar work focusing more heavily on satisfaction with the dentist-patient relationship resulted in the dental visit satisfaction questionnaire (7).

Despite the development of patient satisfaction measures in general dentistry and in some dental subspecialties, such as implants and prosthetics (8), work is needed to develop patient satisfaction measures for orthodontic treatment. The European orthodontic community has highlighted satisfaction as an important component of quality of care (9) and has developed a measure of orthodontic treatment satisfaction (10); however, data on the validity of this instrument have not yet been published. Even when the Euro-Qual validity data become available, Euro-Qual measures may be only of limited use to American orthodontists because of significant differences in the delivery of orthodontic services in the United States versus Europe.

Measuring satisfaction with orthodontic treatment is a complex task. Orthodontists wishing to measure satisfaction with a single aspect of orthodontic treatment might consider adaptation of the Dental Visit Satisfaction Questionnaire (11, 12). Those who wish to measure patient or parent satisfaction with the final treatment result might use a simple global question such as “How satisfied are you with your treatment result?” However, this approach has some inherent problems. Orthodontic treatment, like general dental treatment, is likely to be a multifaceted construct and single-item assessments such as this would fail to detect important subtleties. For example, parents might report being satisfied overall with their child’s treatment, but might be dissatisfied with some specific aspects of treatment, for example office staff or appointment hours and parking. A significant risk of single or simple assessments is that by omitting any dimension of orthodontic treatment satisfaction, generalizations about satisfaction (or dissatisfaction) can be inaccurately extended to all dimensions of treatment. To avoid such erroneous generalizations, measures that consider multiple dimensions of orthodontic treatment are needed. Apart from the Euro-Qual questionnaire in development (10), we could locate no instruments designed to assess satisfaction with multiple dimensions of orthodontic treatment.

To develop a more encompassing orthodontic treatment satisfaction questionnaire that would be suitable for use with parents, results from qualitative interviews with parents of orthodontic patients, orthodontic patients themselves, and existing dental satisfaction questionnaires were used to generate items for preliminary analyses. A variety of approaches were used to identify a universe of items that would encompass as many aspects of orthodontic treatment satisfaction as possible. The present study describes how items were created, tested, and grouped through the use of factor analytic techniques. Finally, detailed descriptions of reliability and validity studies are presented.

Methods

Phase I. Item Generation

This phase of questionnaire development began with 15 telephone interviews of parents of recently (within one year) debanded patients. We asked open-ended questions about the parents’ view of orthodontic treatment, using a method similar to “one-on-one” interviews used in market research (13). The interviewers recorded the responses verbatim and 53 closed-ended items were created using the transcripts of the interviews (14). In 1997 we conducted focus groups with recent (within 18 months) posttreatment patients, and reviewed those transcripts for content (15). As a final step, we conducted a thorough literature search for measures of treatment satisfaction in dentistry and orthodontics (5–7, 9–12, 16). We modified items from general dental satisfaction questionnaires for orthodontic use, and appended those items to the items created from our focus groups and one-on-one transcripts, thereby creating a master list of satisfaction items. After omitting redundant items, we presented the 41-item master list to practicing orthodontists, orthodontic faculty, and several parents of current orthodontic patients for comments on readability and flow. Each item was then paired with a five-point scale ranging from “strongly disagree” (1) to “strongly agree” (5). Four items were phrased in reverse fashion in an effort to diminish the formation of response sets.

Phase II: Questionnaire Distribution

After adding several demographic questions concerning parent age, sex, education level, insurance coverage for orthodontic treatment, and income, the questionnaire was mailed to parents of children younger than 18 years old who had completed orthodontic treatment within the last two years at the orthodontic clinics of Ohio State University (OSU) and University of North Carolina (UNC). We used a modification of the Dillman method (17) to maximize response rate. After two mailings, 112 of 173 (65%) and 108 of 220 (49%) questionnaires were returned at OSU and UNC, respectively. An additional 86 parents bringing their children under 18 years old to have orthodontic retainer checks at UNC completed satisfaction questionnaires at the clinic site as part of an ancillary study. After omitting seven questionnaires with extensive missing data, a total of 299 questionnaires were available for factor analysis.

Phase III: Validity Tests on Subsample

The subsample of 86 parents whose children were enrolled in an ancillary study was separated out from the main sample and the relationships between clinical orthodontic variables and treatment satisfaction explored in this subsample. The 86 children in the ancillary study group were selected without reference to either their initial malocclusion or treatment outcome. This group included children with a broad range of mild to very severe initial skeletal and dental problems. Some children achieved near ideal results, while for others treatment had been discontinued early for varying reasons.

Several measures were selected to describe the severity of the initial clinical condition, treatment change, and treatment outcome.

ANB and Overjet

The anteroposterior skeletal relationships are represented by the ANB angle and dental relationships by the incisor overjet in millimeters, both measured from the lateral skull cephalogram. A “normal” or Class I skeletal relationship generally has an ANB angle of 2–4 degrees and a “normal” or ideal incisor relationship would have an overjet of 1–4 mm. These measures have been shown to be moderately correlated with patients’ perceptions of attractiveness (18).

PAR

The PAR (Peer Assessment Rating) index is a summary score of deviations from a normal or ideal tooth relationship that synthesizes five major occlusal traits found in malocclusion. The index was developed to assess not only the severity of malocclusion, but the outcome of treatment (19), and has been tested and validated in both European and American settings (19, 20). A score of <5 would reflect a nearly ideal occlusion (generally the target of orthodontic treatment), a score of 5–10 would reflect a generally acceptable occlusion with only minor deviations from ideal, while a score of >40 would reflect a very severe malocclusion.

IOTN Aesthetic

The Aesthetic Component (AC) of the IOTN (Index of Orthodontic Treatment Need) is a patient- and parent-derived index that ranks the appearance of the teeth on a 1–10 scale in terms of dental esthetic impairment (21, 22).

Analysis Strategy

Data were entered using Microsoft Access and imported into SAS. All items (except demographics) were subjected to factor analysis using principal components and varimax rotation. Criteria for factor retention included eigenvalue greater than 1 and percent variance explained greater than 5. The data set had a cases-to-item ratio of over 7:1, which is in the acceptable range for factor analysis (23). The total sample size, 299, is considered “good” for exploratory factor analysis (24). Reliability was assessed using Cronbach’s alpha. Multiple regression was used to explore the relationships between each of the subscales and selected demographics on the entire sample of 299. Multiple regression was used to explore relationships between the subscales and measures of (1) pretreatment severity, (2) treatment result, and (3) treatment change on the subset of 86 parent/patients who were enrolled in a longitudinal study of treatment outcomes. Level of significance was set at .05.

Results

Overall Sample Characteristics

Nineteen percent of the respondents were men, and 30 percent of respondents had received orthodontic treatment themselves. For 72 percent of the sample, this was their first child to have orthodontic treatment. Forty-four percent of the sample had dental insurance that covered some portion of orthodontic fees. Fifty-two percent of mothers and 49 percent of fathers were college educated. Ninety percent of the sample was Caucasian, 6 percent were African American. Eighteen percent of the sample were single parents. Families were roughly evenly split between those with incomes between $20–$40K/year, $40K–$60K per year, and above $60K/year. Nearly 87 percent of parents were able to recall the name of their child’s orthodontist.

Subsample Comparisons

To explore possible differences between the UNC survey group (N=108) and the UNC subsample (N=86), bivariate analyses (t-tests for continuous and chi-squared tests for categorical variables) were conducted with Bonferroni corrections for multiple comparisons. No statistically significant differences on demographics were revealed, so the two UNC samples were combined in the overall analyses.

Thirty-seven percent of the sample was from the Ohio State University orthodontic graduate clinic. Bivariate analyses with Bonferroni corrections for multiple comparisons revealed few statistically significant differences between the OSU and UNC groups on demographics. Specifically, the two groups were compared on 17 variables, and differed significantly on five (see Table 1). Only two of these differences were demographic: more OSU parents reported that this was their first child in braces, and UNC parents drove farther to the clinic.

TABLE 1.

Comparison of Characteristics and Response Scores in OSU and UNC Samples

	OSU	UNC	P-value*
Respondents
% male	29	16	.008
% had ortho treatment	24	35	.055
% first child in ortho treatment	82	65	.002
% dental insurance	55	67	.003
% Caucasian	89	92	.377
% single parent	22	15	.151
% recall provider name	83	89	.082
% college degree mother	47	53	.072
% college degree father	49	48	.595
% income above $40K	59	66	.440
Mean # children (SD)	2.6 (1.2)	2.3 (.9)	.015
Mean miles to clinic (SD)	23.7 (22.4)	33.7 (25.8)	.001
Mean age (SD)	45.3 (7.3)	43.9 (5.3)	.046
Satisfaction scores
Process scaled mean (SD)	4.14 (.56)	4.34 (.48)	.001
Psychosocial scaled mean (SD)	3.35 (.64)	3.62 (.66)	.001
Outcome scaled mean (SD)	4.26 (.50)	4.41 (.54)	.020
Overall scaled mean (SD)	3.86 (.45)	4.06 (.41)	.001

^*Significance level was set at .0029 (.05/17) using Bonferroni correction for multiple comparisons.

Factor Analysis

The initial goal of the factor analysis was to winnow the 41-item pool down to a more manageable set of items that would assess treatment satisfaction. Results of the factor analysis yielded three main factors that together accounted for 44.9 percent of the variance. The main factors included: a “process” factor (factor 1, with an eigenvalue of 12.7 accounting for 31.1% of the variance); a “psychosocial” factor (factor 2, with an eigenvalue of 3.5 accounting for 8.5% of the variance); and an “outcome” factor (factor 3, with an eigenvalue of 2.1 accounting for 5.3% of the variance). Six other factors with eigenvalues above 1 are not reported here because they each consisted of too few items for meaningful interpretation and did not meet our factor retention criteria (each accounting for less than 5% variance).

Reliability and Descriptives

Subscales were created from each factor by averaging the items in each factor, i.e., a scaled mean (SM) was computed for each factor. Item descriptions, means, and standard deviations can be found in Table 2. The internal consistencies were computed using Cronbach’s alpha (Table 3). For this sample, alphas ranged between .79 and .92.

TABLE 2.

Items Identified in Three Subscales Together with Mean and Standard Deviation of Response Scores for Each Included Item

Factor	Items	Mean	SD
Process	Informed about costs before treatment	4.40	.61
	Orthodontist treated parent & child w/respect	4.40	.71
	Treatment carefully explained	4.38	.69
	Any questions answered promptly	4.36	.64
	Staff treated child and parent w/respect	4.35	.73
	Child liked orthodontist	4.35	.82
	Orthodontist gentle	4.29	.72
	Treatment area clean and sanitary	4.28	.57
	Plenty of time spent during appointments	4.26	.68
	Office procedures explained before treatment	4.13	.81
	Care could have been better*	4.12	.82
	Kept well informed of progress	4.11	.87
	Assistants were gentle	4.10	.71
Psychosocial	Child’s self-esteem improved	4.40	.87
	Child has more attractive face	4.16	.89
	Child better career opportunities due to ortho	3.36	.90
	Child more confident	3.60	.87
	Child more outgoing	3.40	.94
	Child more popular	3.10	.87
	Child’s academic performance better	2.77	.87
Outcome	Would seek ortho treatment again	4.59	.62
	Child has straighter teeth after treatment	4.50	.68
	Parent satisfied with result	4.41	.73
	Child has better bite	4.36	.70
	Treatment fees too high*	3.92	.82

^*Questionnaire items presented in reversed fashion to diminish formation of response sets.

TABLE 3.

Subscale Internal Consistencies, Scaled Means, Standard Deviations, and Ranges

Subscale	Items	Cronbach’s Alpha	Scaled Mean	SD	Min	Max	N
Process	13	0.92	4.13	.52	2.84	5.00	299
Psychosocial	7	0.87	3.71	.64	2.00	5.00	299
Outcome	5	0.79	4.42	.65	1.40	5.00	299

Relationship of Scales to Demographic Variables

Forward stepwise regression with backward overlook on the entire sample (N=299) revealed no significant relationships between any satisfaction subscale and demographic variables, including income, parent education, parent age, parent or patient sex, insurance status, or race. Treatment setting (OSU, UNC) was related to satisfaction with process, psychosocial outcomes, and overall satisfaction (see Table 1). However, mean differences were small and have little clinical significance.

Relationship of Scales to Clinical Orthodontic Variables

The range of malocclusion severity and treatment change is given in Table 4. For the 86 parent–patient pairs enrolled in the longitudinal study of orthodontic outcomes, results of a forward stepwise regression with backward overlook revealed inverse, significant relationships between posttreatment ovejet and parent satisfaction with treatment process (R²=.13; P<.001), and parent satisfaction with treatment outcome (R²=.28; P<.0001). Improvement in esthetics as measured by improvement in IOTN scores was positively related to satisfaction with psychosocial outcomes (R²=.28; P<.0001).

TABLE 4.

Mean, Standard Deviation, and Range of Clinical Orthodontic Variables Used in Regression Analysis of Relationship Among Satisfaction Scores and Pretreatment Severity, Treatment Result, or Treatment Change

	Pretreatment			Posttreatment			Change
	Mean	SD	Range	Mean	SD	Range	Mean	SD	Range
ANB angle	5.7	1.9	.9–10.5	4.0	1.9	.2–8.7	−1.7	1.6	−5.1 to −1.66
Overjet (mm)	7.5	2.4	2.2–13.1	3.5	1.6	.9–10.3	−4.0	2.8	−10.2 to 3.9
PAR	30.6	7.4	12–47	9.5	7.4	0–32	−21.3	10.3	−43 to 10
IOTN	7.4	1.4	5–10	2.1	1.6	0–8	−5.2	2.1	−9 to −1

Discussion

Factor Structure and Preliminary Validation

These data are useful from two perspectives. First, they help us understand at least three possible dimensions of orthodontic treatment satisfaction and thus inform us about the ways in which parents cognitively “construct” their experience with their child’s treatment. Although additional studies with larger, more representative samples are needed, the present university clinic data suggest that parents evaluate psychosocial outcomes, clinical outcomes, and treatment process distinctly. The differences in average satisfaction ratings between the subscales further suggest that parents are on average more satisfied with treatment outcome and process compared to psychosocial outcomes. This relative difference in satisfaction levels is consistent with data on orthodontic treatment expectations, which suggest that parents typically expect fewer psychosocial benefits compared to the more visible treatment outcome benefits (15).

These data are also useful in that they suggest that parents may be aware of some of the key visible orthodontic outcome variables, and that these visible attributes affect their satisfaction with orthodontic treatment. The finding that posttreatment overjet was related to lower outcome and process satisfaction levels suggests that parents are aware of suboptimal outcomes, even though we can’t say how parents acquired this awareness. In other words, it could be that parents noticed suboptimal outcomes themselves, or it could be that the treating orthodontist(s) pointed out relatively poor outcomes to patients or parents. Further, the relationship between esthetic improvement and satisfaction with psychosocial outcomes suggests that parents may be sensitive to (or simply concerned about) the possible link between dental attractiveness and social functioning.

Finally, the obtained factor structure is very similar to the structure of published dental satisfaction questionnaires. Most dental satisfaction questionnaires include subscales assessing office atmosphere, convenience/access, costs, and perceived quality (25). Our results differ mainly in the emphasis on appearance-based psychosocial outcomes and a deem-phasis on cost/access issues. This may have occurred because, unlike general dental care, orthodontic care is chosen and not motivated by pain or infection. It may be that access and cost become less salient when treatment is a choice. Although the Euro-Qual instrument (9) was developed on former patients and not their parents, the factor structure for that instrument was somewhat similar in that the two largest factors were outcome and process. No factor reflecting satisfaction with psychosocial outcomes was reported from Euro-Qual.

Satisfaction with Orthodontics vs General Dentistry

Parents’ satisfaction ratings were positively skewed for nearly all items, a finding that is somewhat inconsistent with satisfaction studies in general dentistry. Studies of the Dental Satisfaction Questionnaire (16) have suggested scaled means closer to neutral (i.e., 3 on a 5-point scale), while the present orthodontic satisfaction data generally indicate satisfaction scaled mean scores closer to 4 out of 5. This 20 percent difference could be attributable to several factors. The most obvious explanation might be the notable absence of pain in the typical orthodontic experience. While it is true that our qualitative studies of posttreatment patients revealed that pain was an issue for some, it was by no means the central issue that it is for general dental care. Studies using the Dental Satisfaction Questionnaire have consistently demonstrated that satisfaction with pain control is an important issue and the items reflecting this construct typically yield significantly lower satisfaction ratings. Since the single pain-related item did not load on any of the main factors in this analysis, it suggests that pain—a major detriment to patient satisfaction in general dentistry—does not occur with sufficient frequency to “pull down” the satisfaction of orthodontic consumers. This may not have been the case had actual patients—not their parents—been studied.

Furthermore, because parents of posttreatment orthodontic patients made a choice to purchase orthodontic care for their child, it may be nearly impossible to obtain negative impressions of orthodontic care. A child recently out of braces will almost always look better than when treatment began, even if by orthodontic standards the result was poor. Further, strong psychological forces mitigate against negative evaluation of a costly purchase. Specifically, Festinger’s theory (26) of dissonance reduction posits that when individuals are faced with conflicting attitudes or feelings, they are strongly motivated to reduce the internal conflict. They are particularly motivated to change their attitudes when the dissonant state results from free choice. Thus, when parents choose to pay large sums of money for orthodontic treatment, they are strongly motivated to seek out information that confirms the wisdom of their choice, and to ignore information that would suggest their choice was a poor one. Thus, information on the hassles of keeping appointments, long waits, or cost will be systematically ignored, and information on the attractiveness of the orthodontic result and other treatment benefits will be noticed.

Similarly, the assimilation-contrast model of Sherif and Hovland (27) predicts that expectations that are slightly higher than outcomes will increase satisfaction judgments. Only when there is a large discrepancy between expected and perceived performance will dissatisfaction result. A previous study of parents of pretreatment orthodontic patients suggested overall high expectations of orthodontic treatment (15). While only longitudinal work could confirm the assimilation-contrast model, the present data suggest that it may operate in the orthodontic consumer context.

This suggests that we might obtain a greater range of satisfaction scores from parents who felt they did not choose orthodontic treatment for their child (an unlikely scenario in any setting). We might also obtain more normal distribution of satisfaction ratings in settings where orthodontic treatment is paid for entirely by third parties. Also, patients themselves might be more able to generate a wider range of responses because they would find negative information harder to ignore (i.e., they experience pain and inconvenience firsthand) and they do not typically invest their own money in orthodontic care.

Study Limitations

Developing a psychometrically valid instrument is a long process that requires multiple studies of convergent and discriminant validity on multiple representative samples. The present measure appears to have acceptable reliability, and preliminary evidence for validity. However, the sample is not representative of Americans who purchase orthodontic treatment, as most orthodontic patients are treated in private offices by either orthodontists or pediatric dentists. Furthermore, these data cannot be generalized to university teaching institutions due to the overall low response rate (slightly better than half of eligible parents returned a completed questionnaire). Without additional studies with representative samples and higher response rates, it is impossible to know if the present sample represents parents who are different from the population of interest in some important way.

In addition, these data were not collected in a uniform manner. Some parents completed the questionnaire onsite (as was the case for most of the validity subsample), while other parents completed the questionnaire at home. Could the parents who completed the questionnaire on site have felt pressure to make more positive ratings? Perhaps, but our results suggest that parents whose children had suboptimal results felt free enough to make their ratings reflect this fact. Completion of the questionnaire at home might have provided even more striking relationships between satisfaction and outcome. Additional studies are needed to explore this possibility.

Future Directions

While the results of this preliminary study are promising, more research is needed. Representative samples of parents are needed in conjunction with larger samples of patients with complete clinical orthodontic data. Additionally, studies of patients themselves will be crucial in piecing together how orthodontist, stiff, parents, and patients come together to create an orthodontic experience that is rated as satisfying for both parents and patients.