Abstract
Objective:
Quality assessment is an essential part of orthodontic treatment. Most of the current indices are essentially based on occlusal assessment. However, an ideal occlusion is only one aspect of an ideal treatment. The aim of this article is to introduce a new prioritized commitment-based clinical assessment (PCCA) method and present its reliability and linear correlation test in comparison with the comprehensive clinical outcome assessment (CCA).
Methods:
One hundred treated cases were scored with the conventional assessment tool––the CCA––and the newly developed assessment tool––the PCCA––with 2 calibrated examiners at 2 different time intervals. These cases were randomly selected including equal numbers of the main malocclusions managed with fixed conventional edgewise appliances within the past 3 years and had complete pre-treatment and post-treatment routine records. The intraclass correlation coefficient (ICC) was used to assess the intra-examiner repeatability of the total scores of both methods. Pearson’s correlation coefficients were computed to assess the linear relationships between the CCA and PCCA scores.
Results:
The intra-examiner reliability assessed for CCA and PCCA showed high repeatability for both examiners (ICC: 0.93 and 0.945, respectively). The inter-examiner reliability values for CCA and PCCA, assessed by ICC, were 0.84 and 0.96, respectively. The linear correlation between the 2 methods, assessed by comparing the mean score of each case by the 2 examiners was significant, at 0.01.
Conclusion:
The PCCA method can be used for quality assessment in treated orthodontic patients. The preliminary test of the new method presented good inter- and intra-observer agreements and a significant linear correlation with the CCA method.
Keywords: Dental occlusion, calibration, malocclusion, orthodontic appliances, outcome assessment
Introduction
Since the introduction of orthodontics as a dental specialty, the pioneers of the profession made efforts to establish methods of classifying deviations from the so-called “ideal occlusion” and set treatment goals to achieve that. According to Kingsley, articulation of the teeth was considered as the second priority, following the facial appearance. However, in the early 20th century, Angle established a new concept stating that the optimal facial esthetics always coincided with ideal occlusion; therefore, strict rules to achieve ideal occlusal relationships were established and accepted until just the last decades.1
Although a concept shift has occurred, most of the available popular outcome assessment indices are based essentially on the previous concept of considering the final occlusion to assess the quality of final treatment outcome, and contain strict quantitative scoring systems involving intra-arch/inter-arch teeth positions compared with an ideal occlusion measured on dental casts. The most well-known assessment indices are the occlusal index,2 the peer assessment rating (PAR) index,3 and the American Board of Orthodontics Objective Grading System (ABO-OGS).4
Among these indices, the ABO-OGS, thanks to several comprehensive field tests by orthodontic experts, provides one of the most valid and reliable treatment outcome assessments.4 This index includes several criteria on dental casts and some other important aspects of treatment, like the clinician’s ability in case management, and final fulfillment of pre-treatment objectives. On the other hand, it has been found that only 32% of the cases treated by a group of orthodontists achieved scores less than 20 in the model analysis section, which usually would pass the board examination.5 Later on, the comprehensive clinical assessment (CCA) was established at Indiana University as a complement to the ABO-OGS, with the aim of developing a more comprehensive outcome method. The CCA considers the following criteria to provide a clinical score, more representative of quality of treatment in a semi-quantitative manner: facial esthetics, dental esthetics, vertical control, arch forms, periodontium management, root structure preservation, and treatment efficacy.6 However, it does not consider individualized priorities of these criteria at each specific case, nor does it assess the pre-treatment objectives.
On the other hand, considering the special social, economic, cultural, mental, and cosmetic concerns of each individual, compromised occlusion is an inevitable part of orthodontic treatments; therefore, in selected cases, compromising some aspects of ideal occlusion in order to reduce treatment costs and risks and fully satisfy patient compliance seems logical. Additionally, it seems also logical to consider some other aspects of a suitable treatment other than occlusion for assessing its success in a bolder fashion.7
The present methods of assessing treatment outcomes do not put enough emphasis on aspects of an ideal treatment other than occlusion,8 and generally compare treatment outcome with an ideal condition. Since the achievement of an ideal treatment outcome is almost impossible in many patients, in the current investigation, we attempted to develop a new quality assessment method that is based essentially on the clinician’s commitments in a prioritized manner at the start of the treatment ––the prioritized commitment-based assessment (PCCA)––and compared it to the available CCA for a preliminary overview of its applicability.
Methods
Development of a New Quality Assessment Method
A new treatment quality assessment method was designed using pre-treatment and post-treatment records of patients, namely extra-oral and intra-oral standard photographs, dental casts, panoramic radiographs, lateral cephalograms, and treatment progress notes. The standards considered for intra-oral photographs were that they be of good quality without any distortion and blurring of the images, with as much as possible of soft tissue retraction, and 5 standard views (upper and lower jaw occlusal views, front, left, and right teeth in occlusion views). The standards needed for extra-oral photographs were that they be of good quality without any distortion and blurring of the images in at least 4 standard views of face and neck and the upper parts of shoulders (frontal at rest, frontal at smile, profile, and three-quarter in rest).
The decision number of the ethics committee of the Research Institute for Dental Sciences of Shahid Beheshti University of Medical Sciences was EC1392-117. Informed consent was obtained from all the subjects whose documents were evaluated in the study, and the privacy of the records was strictly maintained by the investigators. To evaluate the quality of treatments received with the prioritized commitment-based clinical assessment (PCCA), we designed a customized problem/diagnosis list of problems that are common and frequently seen in most of the orthodontic patients. The list has 2 main parts: (1) facial appearance in vertical, antero-posterior, and transverse dimensions and (2) occlusion in transverse, antero-posterior, and vertical dimensions, which are shown in Figure 1.
Figure 1.
Table designed for prioritized commitment-based clinical assessment. It includes a problem diagnosis list containing common and frequently seen problems regarding facial appearance, occlusion criteria, and general criteria. It also has dedicated columns for the current status of each criterion according to available records, the priority, the orthodontist’s commitment regarding each criterion, the weight of each diagnosed problem according to its priority, the final status at treatment completion, the weighted score, and the total weighted score.
The method is mainly designed to work prospectively; however, it can be used to assess the quality of finished cases having the aforementioned records available, particularly with predetermined treatment objectives and clinician’s commitments available. The general pool of possible orthodontic problems was obtained by evaluating electronic databases of the orthodontic department and determining commonly faced problems. A thorough diagnosis is required with this method. The current status of each of the criteria listed in the table should be recorded in the “current status” column. After recognition of the major problems of each individual patient from a general problem list, the clinician attempts to prioritize them. To reduce the difference among examiners, we suggest following rules of thumb to be considered in prioritization:
Address the chief complaint of the patient as one of the first 2 priorities. For example, if the patient’s chief complaint is “just straighten my upper teeth,” the crowding/rotation may be the first priority. However, if the patient complains about “separated lips,” lip incompetency should be addressed as the first priority.
Address the criteria that are the most deviated from the norms as higher priorities.
Of the problems evident in the case, if there is a problem not listed in the table, the clinician can add it to the table on their own. Then the clinician records his/her commitment regarding each criterion. This would include maintaining the current status or improving it.
In the assessment of facial appearance, the vertical dimension contains 3 criteria: the lower anterior facial height (LAFH), the lip status during rest (competency/incompetency), and tooth display. In the antero-posterior dimension, it includes the nasolabial angle and skeletal relationships. We considered 90-100° nasolabial angle norms for male and 100-110° nasolabial angle norms for female patients. In the transverse dimension, it includes the buccal corridors and asymmetry. LAFH is measured considering the equation G-ANS/ANS-Me = 1, and evaluated on both photographs and lateral cephalogram. Lip separations of 4 mm or more at rest are considered abnormal.9 Over-closed lips, prominent lips, and retruded lips are also addressed in the list of problems. These conditions should be evaluated mainly on standard photographs, but lateral cephalograms can also be used. Anterior tooth display or tooth-show is addressed in the smile (at least a quarter of the crown should be visible in the smile, and 2 mm gingival show is generally the upper limit for an esthetically acceptable smile).10 The nasolabial angle (between 90° and 110°, with acute values acceptable for males and obtuse ones for females) is evaluated on photographs and cephalometry. Maxillary deficiency/excess is a complementary problem addressed as an upper lip problem and is evaluated on photographs and lateral cephalograms. The same is done for chin deficiency/excess. Buccal corridors are evaluated on frontal photographs. Asymmetry is measured on standard photographs, considering the relationship between the A-Pog line and the midsagittal line. If any evident asymmetry is evident and needs to be corrected by orthodontic treatment (like asymmetry due to a lateral functional shift), it would be scored 1, and if no visible asymmetry is present, it would score 0.
In the occlusion section, the evaluation is divided into transverse, antero-posterior, and vertical dimensions as well. Crowding/spacing, rotation, impaction, and missing teeth are also included in this part (Figure 1).
Midline and posterior crossbite are listed in the transverse section of the table. The midline for maxillary arch is recorded as the degree of millimetric deviation from the facial midline. The posterior crossbite is evaluated on dental casts and the teeth involved in this situation are mentioned in the table. Angle classification, overjet, and anterior crossbite are listed in the section regarding antero-posterior part. The Angle classification is recorded for canines and first molars. The overjet is recorded as the millimetric distance between the labial surfaces of incisors. The anterior crossbite is mentioned by the teeth involved in the situation. Overbite is listed in the vertical section, and is recorded by millimetric distance between the incisal edges of incisors.
Crowding is the most prevalent and usually the most important factor in the list of problems. It is measured by comparing the space available and space required, for simplicity. Having a commitment of “correcting the crowding” is not an accurate statement. The clinician should precisely determine the objective, for example “correcting the crowding to 0” or “accept 2-3 mm crowding in the lower incisor area,” in a Cl III compromised treatment case. As mentioned before, in some cases, the ideal occlusal relationship according to available occlusal indices cannot be achieved due to various obstacles. Rotation seems to be somehow overlapped by crowding; however, in some adjunctive orthodontic cases (e.g., pre-prosthetic orthodontic preparation), the problem is better addressed as a distinct one. Impaction and missing teeth are also mentioned in occlusion section by pointing the tooth/teeth involved in the situations.
The main problems are prioritized in the designed table (Figure 1), mostly up to a maximum of 5 main problems. This way, most important problem would have the first priority and the highest weightage of 5. The subsequent important problems would receive 2, 3, 4, and 5 priorities and weightages of 4, 3, 2, and 1 in the list, respectively
We developed a semi-quantitative scoring system. Determination of the score of each item depends on the treatment alterations that are intended for the course of treatment. After determining the score, it is written in the table; according to the priority of the commitment, the weighted score is then calculated: the scores of the first, second, third, fourth, and fifth priorities are multiplied by 5, 4, 3, 2, and 1 respectively.
We also considered some general criteria for each case which have neither weightages nor priorities and should be maintained/improved in all the cases (Figure 1):
Periodontal problems as a consequence of orthodontic treatment: the gingival condition is evaluated and photographed, and 1 or 2 points deducted for moderate and severe gingivitis, respectively. If new bone loss or exacerbation of previous periodontitis is evident, 3 points are deducted.
Root parallelism problems: scoring systems for these criteria are designed on a per-quadrant basis because any problematic mechanics in each quadrant can lead to these kinds of problems.
Root resorption remains a challenging issue in orthodontics.
Oral hygiene is evaluated based on progress notes; if more than 3 warning clinical notes are found, 1 point is deducted.
Decalcification is evaluated on photographs as stated in the CCA method.6
Finally, the “total weighted score” is calculated by the sum of total weighted scores and general criteria scores (Figure 1).
Test of the New Assessment Method
The new model was tested on 100 comprehensive orthodontically treated cases at the Orthodontic Department of Shahid Beheshti Medical University. These cases were randomly selected on each type of malocclusion including an equal 25 cases of Cl I, Cl II div 1, Cl II div 2, and Cl III routine malocclusions managed with fixed conventional edgewise appliances within the past 3 years, and had complete pre-treatment and post-treatment records. The sample records consisted of standardized initial and final study casts, panoramic and lateral cephalometric radiographs, and extra-oral and intra-oral standard photographs, in addition to the orthodontist’s notes of each treatment session. To limit the confounding variables, cases with any congenital or systemic disorders or cleft lip/palate were excluded. In addition, cases managed with concomitant orthognathic surgical or any interdisciplinary approach were not included. All the selected cases were treated by orthodontic postgraduate students under the supervision of the department’s instructing professors.
All the cases were scored using both CCA and PCCA methods with 2 calibrated examiners at 2 separate time intervals. Initially, 20 patients were selected to be scored by both examiners, due to intra-rater calibration procedure.
STATISTICAL ANALYSIS
To assess the intra-examiner repeatability, a subsample of 20 cases was selected and scored by examiners twice with a 4-week interval. The intraclass correlation Coefficient (ICC) was used to assess the intra-examiner repeatability of the total scores. All the cases were assessed by a postgraduate student (author AF) and an orthodontist (author FY) using CCA and PCCA scoring systems. The inter-examiner agreement was assessed also by ICC, the Bland–Altman test, and paired t-test for all the cases. Pearson’s product-moment correlation coefficients were computed to assess the linear relationships between CCA and PCCA scores. SPSS 18 software (SPSS Inc. Release 2009. PASW Statistics for Windows, Version 18.0. Chicago: SPSS Inc.) was used for statistical calculations.
Results
One hundred completed fixed orthodontic treatment cases were evaluated using 2 methods of CCA and PCCA by the 2 calibrated examiners at 2 different time intervals. To avoid any possible bias in interpretation of the data, there was a 4-week time interval between the 2 methods. The mean score of each case by each examiner, using either set up of quality assessment modalities, was calculated.
The linear correlation between the 2 methods was assessed by comparing the mean score of each case by the 2 examiners, and was considered significant at 0.01 (Pearson’s correlation = 0.752). Summarized data and descriptive statistics for the cases are presented in Table 1. ICC for intra-examiner and inter-examiner reliability and paired t-test values for inter-examiner reliability for the 2 methods are presented in Table 2. The Bland–Altman results are shown in Figure 2. The P-value by paired t-test between average CCA and average PCCA was .017. P-values < .05 were assumed significant.
Table 1.
Summarized data and descriptive statistics for the cases measured by authors FY and AF
| PCCA-FY (n = 100) | PCCA-AF (n = 100) | CCA-FY (n = 100) | CCA-AF (n = 100) | Average PCCA | Average CCA | |
|---|---|---|---|---|---|---|
| Mean | 1.11 | 1.26 | 0.74 | 0.66 | 1.18 | 0.70 |
| Minimum | 0 | 0 | 0 | 0 | 0 | 0 |
| Maximum | 15.00 | 15.00 | 7.00 | 8.00 | 15.00 | 7.50 |
| Range | 15.00 | 15.00 | 7.00 | 8.00 | 15.00 | 7.50 |
| Std. Deviation | 2.48 | 2.60 | 1.05 | 1.14 | 2.52 | 1.05 |
PCCA, prioritized commitment-based assessment; CCA, comprehensive clinical assessment; FY/AF, examiners’ initials.
Table 2.
Intraclass correlation coefficient (ICC) and paired t-test values for CCA and PCCA
| ICC | P (Paired t-Test) | |
|---|---|---|
| Intra-examiner reliability for PCCA (FY) | 0.94 | - |
| Intra-examiner reliability for PCCA (AF) | 0.95 | - |
| Intra-examiner reliability for CCA (FY) | 0.92 | - |
| Intra-examiner reliability for CCA (AF) | 0.94 | - |
| Inter-examiner reliability (PCCA) | 0.96 | .028 |
| Inter-examiner reliability (CCA) | 0.84 | .208 |
PCCA, prioritized commitment-based assessment; CCA, comprehensive clinical assessment; FY/AF, examiners’ initials.
Figure 2.
Bland–Altman test graphs. (A) CCA for evaluators AF and FY. (B) PCCA for evaluators AF and FY. (C) Average for CCA and PCCA.
Discussion
Setting objectives for orthodontic treatment can generally be based on 2 different presumptions: 1) convert all the observed malocclusions to an ideal occlusion. Therefore, for assessing the orthodontic practice and clinicians’ skills, one should set a gold standard with ideal occlusion, and any deviation from that would be considered a sort of inadequacy in practice. In this perspective, the main objective is to treat the malocclusion; the other factors important in providing medical services––patient concerns, treatment duration, costs versus benefits etc.––take the second place.
2) Clinicians should openly accept the fact that in nearly 70% of the cases, an ideal occlusion could not be achieved according to available gold standards.5 Therefore, they should try to focus on a method to assess the treatment outcomes and clinicians’ skills with a more realistic approach, rather than simply assign a majority of them to a so-called “failed” group. This can be done by considering the fulfillment of a prioritized commitment-based problem list at the end of treatment. Orthodontists should roughly classify their patients at the beginning of the service to 2 groups: 1) patients for whom the ideal occlusion can be achieved; 2) patients for whom the ideal occlusion is not achievable. For the first group, performing assessment according to an ideal gold standard seems logically acceptable. However, for the latter group, it is better to consider the proposed method of assessment of a prioritized commitment-based problem list as an adjunctive assessment tool. Therefore, in our opinion, the treatment outcome assessment in such cases should be based on 2 fundamental points in addition to well-designed routine standard assessment methods: (1) the clinician’s commitments, and (2) the fulfillment of the commitments according to their determined priority.
In the present study, we introduced a new assessment method for treatment outcomes based essentially on the clinician’s commitment at treatment initiation and their priorities in each individual case. According to the study design, that is, developing a new assessment tool and performing preliminary tests for evaluating the validity and reliability of the method, and also considering similar studies, we randomly selected 25 cases for each type of malocclusion (Cl I, Cl II div1, Cl III div2, and Cl III). Evaluating the reliability of a newly developed method needs several statistical assessments, since each statistical test may have its advantages and disadvantages. The preliminary test of the PCCA regarding its correlation with another comprehensive assessment method, the CCA, using ICC, was relatively good. Furthermore, the preliminary test of reproducibility showed excellent reliability. However, considering Bland–Altman and paired t-test, we did not have a perfect correlation between the 2 methods. This may refer to the fact that the CCA itself is not assumed as a perfect gold standard for assessing orthodontic treatment outcome.6 Regarding the results of the paired t-test considering the large number of cases, any little difference between assessment methods may induce a significant statistical difference; however, for clinical use, relying on the ICC test seems appropriate.
The first index with a comprehensive study on its validity and reliability was the PAR index.3 This index is totally based on assessing dental casts and includes no factor for assessing other aspects of orthodontic treatment. For the occlusion assessment, the index was not perfectly precise in discriminating between the minor malpositions of the teeth that are found in ABO case reports.4 Therefore, an ABO committee was formed to design a more precise method of objective quality assessment. As a result of 3 phases of examination, the ABO-OGS was introduced, and is now being used as the standard of board qualification in ABO.4
The ABO uses one of the most valid and reliable methods for assessing treatment quality.4,11 The ABO-OGS, by means of a specifically designed measuring gauge, objectively gives scores to treated cases. Final dental casts and panoramic radiographs are considered in the process of evaluation. This outcome assessment index can be assumed as a high standard occlusal index, considering its precise millimetric scores given to each of the 7 occlusal criteria.4 The board qualification process also includes a clinical management part, which considers skeletal, dental and facial analyses separately and is based on pre-treatment and post-treatment measurements and pre-treatment objectives. In the case of fulfillment of any objective, no score is deducted; otherwise, a point will be deducted. However, this section seems somehow brief compared to the occlusal assessment part. For example, it lacks an appropriate consideration of soft tissue changes as a consequence of orthodontic treatments, since only the E-line is being considered in the facial analysis. The case management section also uses a simple scoring method (0 or 1), lacking a weighting method to discriminate between higher treatment priorities from the lower ones. The PCCA concept may be especially used to improve this part of the board qualification process and help it achieve greater validity and reliability in assessing treatment outcomes.
The CCA was designed at Indiana University as a clinical complement to the ABO-OGS.6 It includes several factors and raises a more comprehensive point of view for outcome evaluation. However, this method lacks a condition-specific approach or a differential weighting scale applicable in clinical practice. For example, either a deterioration in the patient’s profile or the mistake of leaving bonding resin remnants on enamel surface lead to maximum of 2 points deducted from the total score of an orthodontic case. Who can claim that these faults are of the same importance?
On the other hand, some authors have questioned the rationale of determining an ideal occlusal situation, stating that different malocclusions should be assumed as normal variations rather than pathologic disorders, and that the orthodontists should improve some characteristics of such variations in terms of esthetics or function.12,13 Therefore, conducting the assessment is based essentially on an ideal standard occlusion without proper consideration of other of clinical expertise like ethics, life experience, patient satisfaction, work habits, and the ability to handle stressful situations, response to criticism, and ability to participate as part of health-care team.14 Therefore, it seems necessary to also properly include other aspects of an ideal treatment into outcome assessment methods, since the perfect occlusal outcome by itself cannot be an indicator of optimal treatment. We should assess the quality of treatment, or skills of the clinician in setting, and the prioritization of appropriate objectives for each patient individually and then the clinician’s ability to reach these commitments.
A compromised treatment option with less than ideal occlusal outcome may be preferred over an ideal plan with considerable treatment duration, costs, and risks.15 However, we do not claim that the idealistic occlusal indexes should not be used anymore or that they are of little value for assessing the outcomes. In fact, they should be applied to all cases, but one should also consider more comprehensive clinical management assessments.
The PCCA also uses a case-specific weighting system. A certain problem in different cases does not necessarily indicate the same level of importance. Therefore, the different weightages for each row of prioritized problems in the list provide a more accurate way for addressing problems and commitments in treatment outcome assessment. We suggest assessing the treatment outcome considering the first 5 priorities, based on the fact that after assessing the electronic database of the department, we faced a minimum of one problem (e.g., crowding in mild Cl II malocclusion) and a maximum of 7-8 problems; however it seems that the main problems in the most severe cases can be summarized to 5.
Another specific benefit of this quality assessment system includes the possibility for its use in the early mixed dentition for phase I treatment quality assessment. Since there are some specific goals in early phase of treatments that are not included in the routine overall goals for comprehensive treatments, most of the mentioned grading systems exclude the mixed dentition patients to restrict the confounding variables and increase the reliability of the method.3,4 However, with the ability to adjust the objectives of each phase in the PCCA method, it is possible to implement it for these interim treatment modalities as well.
It is clear that the presented method is not a perfect one; although the preliminary test of the method showed excellent results. As we go toward a more subjective assessment method, its reliability may decrease. The main problem with the PCCA is the possible differences between clinicians in determining treatment priorities in similar conditions. This problem was the case in our study, but interestingly, minor differences in setting treatment priorities between 2 examiners had a nonsignificant effect on the inter-examiner agreement. However, strict adherence to the soft tissue paradigm, the patient’s chief complaints, and the most deviant aspects of each case of malocclusion might decrease the level of possible heterogeneity in this regard. We also found a good linear correlation between PCCA and CCA (but not an excellent one). We believe this finding only shows different points of view of these methods in assessing the cases.
Future efforts should be made to find a solution to increase the clinicians’ agreement on developing prioritized problem lists, identifying patients’ needs, and providing the best approach to address them. This may need several well-conducted clinical trials and meta-analysis studies to develop specific guidelines considering the costs versus the benefits of any treatment modality for various clinical conditions.
As final words, we look at the PCCA as an adjunctive tool for assessing orthodontic treatment outcome, but not as the only one. The patients may be satisfied when the chief complaint is treated, but this does not necessarily mean that the patient is free of further possibly progressive risks of poor oral health.
Conclusion
The preliminary test on a new PCCA index presented good inter- and intra-observer agreements in comparison with the currently available comprehensive clinical outcome assessment method.
Ethical Committee Approval:
Ethical committee approval was received from the Ethics Committee of Shahid Beheshti University of Medical Sciences, (Approval No: EC1392-117).
Informed Consent:
Written informed consent was obtained from all participants who participated in this study.
Peer-review:
Externally peer-reviewed.
Author Contributions:
Concept - S.M.S., A.F.; Design- S.M.S., A.F., F.Y.; Data Collection and/or Processing - A.F.; Analysis and/or Interpretation - A.A.B.; Writing - A.F., F.Y.
Conflict of Interest:
The authors have no conflict of interest to declare.
Financial Disclosure:
The authors declared that this study has received no financial support.
References
- 1. Sarver DM, Ackerman JL. Orthodontics about face: the re-emergence of the esthetic paradigm. Am J Orthod Dentofacial Orthop. 2000;117(5):575–576.. 10.1016/s0889-5406(00)70204-6) [DOI] [PubMed] [Google Scholar]
- 2. Summers CJ. The occlusal index: a system for identifying and scoring occlusal disorders. Am J Orthod. 1971;59(6):552–567.. 10.1016/0002-9416(71)90002-9) [DOI] [PubMed] [Google Scholar]
- 3. Richmond S, Shaw WC, O'Brien KD. et al. The development of the PAR Index (Peer Assessment Rating): reliability and validity. Eur J Orthod. 1992;14(2):125–139.. 10.1093/ejo/14.2.125) [DOI] [PubMed] [Google Scholar]
- 4. Casko JS, Vaden JL, Kokich VG. et al. Objective grading system for dental casts and panoramic radiographs. American Board of Orthodontics. Am J Orthod Dentofacial Orthop. 1998;114(5):589–599.. 10.1016/s0889-5406(98)70179-9) [DOI] [PubMed] [Google Scholar]
- 5. Abei Y, Nelson S, Amberman BD, Hans MG. Comparing orthodontic treatment outcome between orthodontists and general dentists with the ABO index. Am J Orthod Dentofacial Orthop. 2004;126(5):544–548.. 10.1016/j.ajodo.2003.11.020) [DOI] [PubMed] [Google Scholar]
- 6. Pinskaya YB, Hsieh TJ, Roberts WE, Hartsfield JK. Comprehensive clinical evaluation as an outcome assessment for a graduate orthodontics program. Am J Orthod Dentofacial Orthop. 2004;126(5):533–543.. 10.1016/j.ajodo.2004.05.017) [DOI] [PubMed] [Google Scholar]
- 7. Ackerman MB, Rinchuse DJ, Rinchuse DJ. ABO certification in the age of evidence and enhancement. Am J Orthod Dentofacial Orthop. 2006;130(2):133–140.. 10.1016/j.ajodo.2005.07.012) [DOI] [PubMed] [Google Scholar]
- 8. Sachdeva RC. SureSmile technology in a patient-centered orthodontic practice. J Clin Orthod. 2001;35(4):245–253.. [PubMed] [Google Scholar]
- 9. Nanda R. Biomechanics and Esthetic Strategies in Clinical Orthodontics. Amsterdam: Elsevier Health Sciences; 2005. [Google Scholar]
- 10. Graber LW, Vanarsdall RL, Vig KW. Orthodontics: Current Principles and Techniques. Amsterdam: Elsevier Health Sciences; 2005. [Google Scholar]
- 11. Lieber WS, Carlson SK, Baumrind S, Poulton DR. Clinical use of the ABO-Scoring Index: reliability and subtraction frequency. Angle Orthod. 2003;73(5):556–564.. 10.1043/0003-3219(2003)073<0556:CUOTAI>2.0.CO;2) [DOI] [PubMed] [Google Scholar]
- 12. Ackerman MB. Six keys for making orthodontics a sustainable dental specialty. Angle Orthod. 2013;83(6):1102–1103.. 10.2319/0003-3219-83.6.1102) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Ackerman MB. Selling orthodontic need: innocent business decision or guilty pleasure? J Med Ethics. 2010;36(5):275–278.. 10.1136/jme.2009.033761) [DOI] [PubMed] [Google Scholar]
- 14. McCartney RD. Assessing a physician’s true worth. Internist. 1995;36(4):8-10. [PubMed] [Google Scholar]
- 15. Safavi S, Namazi A. Evaluation of mandibular incisor extraction treatment outcome in patients with Bolton discrepancy using peer assessment rating index. J Dent. 2012;9(1):27-34. [PMC free article] [PubMed] [Google Scholar]