Abstract
Purpose
Accurate and exact measurement is an important factor for generating meaningful results in any properly designed study. If all the participating examiners are able to yield similar results, it will be possible to evaluate the objective results of the study more easily and quickly. The purpose of this study was to evaluate the intra- and inter-examiner reproducibility of histometric measurements in the intrabony periodontal defect model.
Methods
One wall intrabony defects were surgically created at the distal aspect of the second and the medial aspect of the fourth mandibular premolars in the right and left jaw quadrants in twenty beagle dogs and the defect sites received the following β-tri calcium phosphate, growth differentiation factor-0, growth differentiation factor-100 and sham surgery. Histometric analysis was performed after 8 weeks. Histometric parameters were recorded and repeated at three months interval by three examiners. Intra- and inter-examiner reproducibility was assessed.
Results
Most parameters of all the groups showed high intra- and inter-examiner reproducibility. Parameters including defect height, bone regeneration height, cementum regeneration height, and formation of junctional epithelium yielded inter-examiner correlation ≥ 0.9. The intra-examiner reproducibility showed a high result, over 0.9.
Conclusions
Histometric evaluation of the one-wall intra-alveolar periodontal defect model showed high reproducibility not only for a single given examiner but also among the three examiners.
Keywords: Alveolar bone loss, Histology, Reproducibility of result
INTRODUCTION
Reliable and accurate measurements serve as the basis for evaluation in many scientific disciplines. Clinical measurements serve as a basis for recording variable parameters such as probing depth, clinical attachment level, comparisons of radiographic registration pre- to post-therapy, and surgical procedures to assess changes in alveolar morphology [1]. Any of these methods may not be reliable as evidence of regeneration of the periodontal attachment. The inability of periodontal probing to assess the coronal level of connective tissue attachment has been amply demonstrated [2-4]. In addition, radiographic analysis may not clearly detect cementum or the periodontal ligament. However, histologic evaluations may disclose the genuine nature of healing following periodontal regenerative procedures. Histometric assessments are used as part of a overall histopathologic evaluation. For a periodontal regeneration protocol, the histometric evaluation is used to quantify the amount of regeneration of alveolar bone, cementum, a functionally-oriented periodontal ligament, formation of a junctional epithelium, and the position of any devices or biomaterial implanted in conjunction with the surgical procedure [5]. For successful results, establishment of the purpose of the study, proper study design, concrete surgical protocol and procedure, and precise histometric analysis are very important factors; in particular, the histometric analysis of the slides acts as the main barometer in evaluating the results. However, to obtain an expected study result, most examiners need to spend immeasurable time and effort. Therefore, if all the participating examiners are able to yield a similar result, it will be possible to evaluate the objective results of the study more quickly and easily.
Often, a reliability or validity study involving multiple observers is conducted in a clinical or experimental setting. In method comparison and reliability studies, it is often important to assess agreement among multiple measurements made by different methods, devices, laboratories, observers, or instruments. For continuous data, the concordance correlation coefficient (CCC) is a popular index for accessing agreement among multiple methods on the same subject [6]. Lin stated that the appropriate index for measuring agreement between two observers is the CCC and argued that, even though the agreement is often evaluated by using the Pearson correlation coefficient, the paired t-test, the least square analysis of slope (= 1) and intercept (= 0), the coefficient of variation, or the intra-class correlation coefficient (ICC), none of these can fully assess the desired reproducibility characteristics. Use of the CCC as measure of reproducibility has gained popularity [7-9]. However this agreement index is defined in the context of comparing two fixed observers. Because the reliability and validity studies often involve more than two observers, there is a need to achieve agreement among multiple observers [10]. The purpose of this study was to evaluate the intra-examiner and inter-examiner reliability and/or reproducibility of histologic and histometric measurements by three examiners of one-wall intrabony periodontal defects.
MATERIALS AND METHODS
Animals
Twenty young adult (age, 15 months; body weight, 12-15 kg) beagle dogs were used.
The animals had intact dentition with healthy periodontium. Animal selection, management, surgery protocol, and periodontal defect preparation followed routines approved by the local Institutional Animal Care and Use Committee, Yonsei Medical Center, Seoul, Korea. The animals were fed soft diets throughout the study to reduce the chance of mechanical interference with healing during food intake.
Surgical protocol
Surgical procedures were performed under the general anesthesia induced by IV injection of atropine (0.04 mg/kg, Kwangmyung Pharm., Seoul, Korea) and I.M injection of combination of xyline (Rompun, Bayer Korea Co., Seoul, Korea) and ketamine (Ketara, Yuhan Co., Seoul, Korea) followed by inhalation anesthesia (Gerolan, Choongwae Pharm., Seoul, Korea). Routine dental infiltration anesthesia was used at the surgical sites. During surgery, the animals received lactated Ringer's solution (300-500 mL, IV).
The mandibular first and third premolar teeth were surgically extracted prior to the experimental surgery in each animal [11]. The extraction sockets were allowed to heal for two months. The residual dentition received oral prophylaxis in conjunction with the extractions.
Experimental protocol
The experimental surgery included elevation of buccal and lingual mucoperiosteal flaps to surgically create "box-type" one-wall intrabony defects of critical size (4 × 4 × 4 mm) at the distal aspect of the second and the mesial aspect of the fourth mandibular premolars in the right and left jaw quadrants in each animal [12] (Fig. 1). Following root planing, a reference notch was made with a round bur on the root surface at the base of the defect [13]. Using a split design, the unilateral defects in five animals were implanted with β-tri calcium phosphate (TCP), another five animals received growth differentiation factor-0 (GDF-0), five more were treated with growth differentiation factor-100 (GDF-100), and the last five underwent sham surgery. We also implanted at contralateral sides with same materials but with difference in same animal. The mucoperiosteal flaps were advanced, adapted, and sutured using a resorbable suture material (Vicryl 5.0 polyglactin 910, Ethicon, Johnson & Johnson, New Brunswick, NJ, USA).
Figure 1.
Surgically-created critical size one-wall intrabony periodontal defect at the distal aspect of the mandibular second and mesial aspect of the mandibular 4th premolar teeth. Clinical observation of surgical sites 8 weeks after surgery. All of surgical sites appeared to have healthy gingival conditions.
Radiographs of the defect sites were taken at presurgery, immediately postsurgery, and on the day of euthanasia.
Postsurgery management
Postsurgery care included intramuscular injection of antibiotics (cefazoline sodium 20 mg/kg, Yuhan Co., Seoul, Korea) and daily topical application of a 0.2% chlorhexidine solution (Hexamedine®, Bukwang Pharm., Seoul, Korea) for infection control [14]. Observations of the experimental sites with regard to gingival health, maintenance of the suture line closure, edema, and evidence of tissue necrosis or infection were made daily until suture removal, and at a least twice weekly thereafter.
Histological procedures
The animals were euthanized at 8 weeks postsurgery using an overdose of pentobarbital (90-120 mg/kg: IV). Block sections including the defect sites along with the surrounding alveolar bone and mucosal tissues were collected. The block specimens were rinsed in sterile saline and were immersed in 10% neutral buffered formalin at a volume 10 times that of the block section for 10 days. After rinsing in sterile water, the sections were decalcified in 5% formic acid for 14 days, trimmed, dehydrated in a graded ethanol series, and embedded in paraffin. Step-serial sections 5 µm thick were cut in a mesial-distal vertical plane at 80 µm intervals [15]. The sections were stained using hematoxyline/eosine stains. The three most central sections of each defect site selected based on the width of the root canal were used for the histological and histometric analysis.
Histometric measurement protocols for intra- and inter-examiner reproducibility
Selection of examiners
Three periodontists with more than three years of work experience and more than one year of experience with histometric analysis were chosen.
Training of examiners
The examiners were trained by a professional histologist for about three months to assess four parameters of surgically created defects. Also, these examiners performed two separate repeated histometric evaluations three months apart.
Histometric analysis
The three examiners (KSK, KTG, LJS) independently performed histometric analyses using incandescent microscopy (BX 60, Olympus America Inc., Melville, NY, USA), a microscope digital camera system (DP 10, Olympus America Inc.), and a PC-based image analysis system (Image-Pro Plus™, Media Cybernetics, Silver Spring, MD, USA) customized for intra-alveolar periodontal defect models [16].
The following parameters were recorded for the buccal and lingual tooth surfaces for each section (Fig. 2):
Figure 2.
Parameters used in the histometric analysis. The new cementum appeared thicker at the notch area, becoming thinner coronally. The new bone (NB) appeared the apical extension of the root surface notch to the coronal extension of along the root surface.
CEJ: cement-enamel junction, JE: junctional epithelium, NC: new cementum, BN: base of notch.
Defect height: distance from the apical extension of the root surface notch to the cemento-enamel junction (CEJ).
Bone regeneration (height): distance from the apical extension of the root surface notch to the coronal extension of newly formed bone along the root surface.
Cementum regeneration (height): distance from the apical extension of the root surface notch to the coronal extension of newly formed cementum or a cemetum-like substance on the root surface.
Epithelial attachment: distance from the CEJ to the apical extension of an epithelial attachment on the root surface. This parameter includes gingival recession.
The measurement of inter-examiner reproducibility
The measurement values of histometric parameters were recorded and repeated within three-month intervals by the three examiners. The blind data of the three examiners were collected and the intra-examiner reproducibility and inter-examiner reproducibility were assessed [10].
Statistical analysis
Blind data was collected from the three examiners, and the data was taken into consideration in the analysis. Standard errors of mean were adjusted for the correlation. SAS ver. 9.1 (SAS Inc., Cary, NC, USA) was used for the statistical analysis for differences among experimental conditions using one-way analysis of variance and the Pearson correlation coefficient. The statistical method was used for ICC to identify intra-examiner reproducibility [17,18]. Two examiners were then paired and the inter-class correlation coefficient was found to evaluate the inter-examiner reproducibility among the three examiners.
Intra-class correlation coefficient
An index to observe the reproducibility of data which an examiner had measured several times. It can be expressed as the coefficient value.
Inter-class correlation coefficient
An index to observe the reproducibility among several examiners. The Pearson correlation coefficient can be used so that it is expressed as the value of two paired examiners.
RESULTS
Clinical observations
All of the surgical sites appeared to have healthy gingival conditions except for one coronal site exhibiting gingival inflammation. There were no specific clinical characteristics that differed from the non-surgical sites.
Histological observations
The defect sites were available for analysis with the exception of one root in a control site that was lost in the histotechnical preparation. Generally, the barrier device was located near the CEJ and the epithelium arrested at the CEJ. Two animals exhibited an inflammatory infiltrate, partially involved in the defect site, localized in the buccal and/or distal root of the premolar teeth in sites receiving sham surgery. These animals also exhibited sites without an inflammatory infiltrate. The histologic evaluation revealed limited inflammatory cell infiltration in the defect sites. The junctional epithelium appeared to extend more apically in the treated sites compared to the control site. New bone was observed in the treated sites. All defect sites maintained a periodontal ligament space. The new cementum appeared thicker at the notch area, becoming thinner coronally. None of the specimens exhibited ankylosis. Limited root resorption was observed in most of the defect sites (Fig. 3).
Figure 3.
Photomicrographs showing the critical size intrabony periodontal defects at 8 weeks postsurgery. They display the sites from the apical extension of the root planing along the root surface to the coronal extension of the newly formed bone and cementum and the apical extension of an epithelial attachment (H&E stain).
Histometric analysis
The intra-examiner histometric records showed high reproducibility for most parameters.
There were no statistically significant differences in bone formation among the treatments. Measurement of defect height, bone regeneration height, cementum regeneration height, and epithithelial attachment height expressed ICCs. The intra-examiner reproducibility turned out to be very high. ICCs were greater than or equal to 0.99 for assessment of defect height, bone height, cementum, and epithelium. Mean histometric recordings for individual measurement score and overall group means (± SD) are shown in Tables 1 and 2. The reproducibility of the three examiners is expressed as different measurement values for each group (β-TCP, GDF-0, GDF-100, sham), but most of the measurement values showed high reproducibility. P-values are over 0.05 and inter-examiner correlation coefficients are greater than or equal to 0.95. The reproducibility value of histometric measurement for the three examiners and overall group means (± SD) are shown in Tables 2 and 3. Upon initial review, it was decided that the histometric evaluation should only include the three most central sections since histologic observations showed that the healing events were closely repeated among the four central sections. This is not expected to alter the outcomes of the histometric evaluation. As for the qualitative analysis, a few experimental sites showing pulp exposure or an unresorbed suture knot with minor inflammation were excluded from the histometric analysis. Similarly, a few sites exhibiting residual old cementum were excluded [1].
Table 1.
Intra-examiner reproducibility: Intra-class correlation coefficient.
CEJ: cemento-enamel junction, I: β-tri calcium phosphate (β-TCP), II: growth differentiation factor-0 (GDF-0), III: growth differentiation factor-100 (GDF-100), IV: sham surgery.
Table 2.
Reproducibility in histometric measurement among the three examiners.
CEJ: cemento-enamel junction, β-TCP: β-tri calcium phosphate, GDF-0: growth differentiation factor-0, GDF-100: growth differentiation factor-100.
Table 3.
Inter-examiner reproducibility in histometric measurement between each two examiners.
Each examiner expressed by A, B, C.
CEJ: cemento-enamel junction, β-TCP: β-tri calcium phosphate, GDF-0: growth differentiation factor-0, GDF-100: growth differentiation factor-100.
DISCUSSION
The objective of this study was to evaluate the intra-examiner and inter-examiner reliability and/or reproducibility of histologic and histometric measurements among three examiners in one-wall intrabony periodontal defects.
For the periodontal regeneration protocol, the histometric evaluation was used to quantify the amount of regeneration of alveolar bone, cementum, a functionally-oriented periodontal ligament, formation of a junctional epithelium, and the position of any devices and biomaterial implanted in conjunction with the surgical procedure [5]. Thus, the assessment of reproducibility is an essential element for histometric measurement of any study evaluating the bone regenerative effect of recombinant human bone. It is well accepted that repeated measures could increase the reliability and reduce the examiner or technique error.
Before using the measurement scale in practice, one often needs to assess the agreement of multiple observers using several methods. If there is an observed disagreement, one often wants to know whether the disagreement is due to random error within a method or due to true differences attributed to the different methods. If the disagreement is due to random error within a particular method, this method should not be used in practice. If disagreement is due to a true difference among the methods, the methods will need to be modified for improvement. Therefore, assessing agreement often leads to assessing both intra-method agreement and inter-method agreement, where intra-method agreement measures consistency of readings taken by the same method and the inter-method agreement measures consistency of true readings attributed by the methods [19].
Issues related to reliable and accurate measurement have evolved over many decades, dating back to the pioneering work of the nineteenth century [20]. The popular agreement indices based on an observed reading such as the CCC or different versions of the ICC are, in fact, the measure of total agreement although they are often reported as inter-method agreement [19,21]. The CCC is more appropriate than other indices for measuring agreement when the variable of interest is consecutive [9,22]. However, this agreement index is defined in the context of comparing two fixed observers [23]. In order to use multiple observers in a study involving large numbers of subjects, there is a need to assess agreement among these multiple observers. In this study of repeated measurements, the class variable was the examiner. Multiple examiners can be trained at different times to measure histometric characteristics with good to excellent intra-examiner reliability [24]. The measurement errors of histometric assessments made by one or two examiners of a critical size intra-alveolar periodontal defect model was evaluated. For repeated measurements made by the same examiner, histometric parameters yielded ICCs greater than or equal to 0.97. This suggests that a single given examiner had excellent reproducibility in assessing most of the histometric parameters evaluated.
When measurements made by three independent examiners were evaluated, all the parameters assessed yielded inter-class correlation coefficients greater than or equal to 0.95, also suggesting excellent consistency. The correlations were somewhat higher in repeated measurements made by one examiner compared to the measurements made by three examiners, and this was consistent for all the parameters studied. This finding is in accordance with other studies showing higher intra-examiner than inter-examiner reproducibility of repeated measurements of clinical periodontal parameters [25].
The goal of periodontal treatment is the regeneration of periodontal attachments including the formation of new cementum, a functionally-oriented periodontal ligament, and alveolar bone on the root surface previously exposed to periodontitis. To achieve this goal, various protocols have been proposed. Treatment outcomes for periodontal defects were influenced by several factors: bacterial contamination, innate wound healing potential, site characteristics, and surgical procedure. Among these factors, evidence suggests that the healing potential of intrabony lesions is primarily dependent on the defect morphology as expressed by the number of associated bone walls. Therefore, we used a surgically created intrabony box-type (4 × 4 × 4 mm) periodontal defect model [12]. Many of these outcomes are continuous scale variables and the more important of these include defect height and area, and newly formed alveolar bone, height and area. Histometric analyses of the latter variables are appeared regularly in studies of novel periodontal and implant therapies. This study showed that, when a single examiner performed the measurement of these parameters, 1-6% of the total variation was due to the examiners' random measurement error. Furthermore, a measurement error component of about 1-17% of the total variation was due to differences among the examiners. This suggests that assessments made by one examiner are quite adequate for analyzing data. Other study outcomes examined in this model demonstrated a range of measurement reproducibilities [22], some variables showing higher measurement consistency than others. Thus, measurement of newly regenerated bone and density of biomaterial exhibited the highest level of measurement errors. The estimated error component was 23-34% for a single examiner and 27-40% for two examiners. On the other hand, measurement of junctional epithelium and ankylosis showed low intra-examiner measurement errors, but a higher inter-examiner error component ranging from 23-27%. For some of the histometric variables, the measurement errors may have been attenuated by the skewness of the variables, since some specimens did not reveal the presence of the outcome variable being assessed. This was particularly evident for the measurement of cementum regeneration, which was scored as absent in a high percentage of the specimens. For this reason, the consistency of measuring cementum regeneration in this study was not assessable.
Present study has determined that histometric evaluations of one-wall intra-alveolar defects yielded highly reproducible inter-examiner but also intra-examiner results. Even though the study was properly designed, with a concrete surgical protocol and procedure, a precise histometric analysis was beneficial for evaluating the results of the objective study more quickly and easily. In the case of a single given examiner performing the histometric measurements, this applied even when the examiner had received limited training. What is more, the use of analysis rules may help to increase reproducibility among and within examiner groups. Consequently, if the participating examiners are able to yield a similar result, it will be possible to evaluate the objective results of the study more efficiently.
ACKNOWLEDGMENTS
This study was supported by a faculty research grant from Yonsei University College of Dentistry (6-2009-0039).
Footnotes
No potential conflict of interest relevant to this article was reported.
References
- 1.Koo KT, Polimeni G, Albandar JM, Wikesjo UM. Periodontal repair in dogs: examiner reproducibility in the supraalveolar periodontal defect model. J Clin Periodontol. 2004;31:439–442. doi: 10.1111/j.1600-051X.2004.00508.x. [DOI] [PubMed] [Google Scholar]
- 2.Hatakeyama Y, Uzel MI, Santana RB, Ruben MP. Relationship between position of probe tip and periodontal tissues after periodontal surgery in dogs. Int J Periodontics Restorative Dent. 2005;25:247–255. [PubMed] [Google Scholar]
- 3.Armitage GC, Svanberg GK, Loe H. Microscopic evaluation of clinical measurements of connective tissue attachment levels. J Clin Periodontol. 1977;4:173–190. doi: 10.1111/j.1600-051x.1977.tb02271.x. [DOI] [PubMed] [Google Scholar]
- 4.van der Velden U, de Vries JH. Introduction of a new periodontal probe: the pressure probe. J Clin Periodontol. 1978;5:188–197. doi: 10.1111/j.1600-051x.1978.tb02279.x. [DOI] [PubMed] [Google Scholar]
- 5.Koo KT, Polimeni G, Qahash M, Kim CK, Wikesjo UM. Periodontal repair in dogs: guided tissue regeneration enhances bone formation in sites implanted with a coral-derived calcium carbonate biomaterial. J Clin Periodontol. 2005;32:104–110. doi: 10.1111/j.1600-051X.2004.00632.x. [DOI] [PubMed] [Google Scholar]
- 6.Barnhart HX, Lokhnygina Y, Kosinski AS, Haber M. Comparison of concordance correlation coefficient and coefficient of individual agreement in assessing agreement. J Biopharm Stat. 2007;17:721–738. doi: 10.1080/10543400701329497. [DOI] [PubMed] [Google Scholar]
- 7.Quinn C, Haber MJ, Pan Y. Use of the concordance correlation coefficient when examining agreement in dyadic research. Nurs Res. 2009;58:368–373. doi: 10.1097/NNR.0b013e3181b4b93d. [DOI] [PubMed] [Google Scholar]
- 8.Lee J, Tran Q, Seeba G, Wikesjo UM, Susin C. The criticalsize supraalveolar peri-implant defect model: reproducibility in histometric data acquisition of alveolar bone formation and osseointegration. J Clin Periodontol. 2009;36:1067–1074. doi: 10.1111/j.1600-051X.2009.01487.x. [DOI] [PubMed] [Google Scholar]
- 9.Liao JJ, Lewis JW. A note on concordance correlation coefficient. PDA J Pharm Sci Technol. 2000;54:23–26. [PubMed] [Google Scholar]
- 10.Barnhart HX, Haber M, Song J. Overall concordance correlation coefficient for evaluating agreement among multiple observers. Biometrics. 2002;58:1020–1027. doi: 10.1111/j.0006-341x.2002.01020.x. [DOI] [PubMed] [Google Scholar]
- 11.Sculean A, Nikolidakis D, Schwarz F. Regeneration of periodontal tissues: combinations of barrier membranes and grafting materials-biological foundation and preclinical evidence: a systematic review. J Clin Periodontol. 2008;35:106–116. doi: 10.1111/j.1600-051X.2008.01263.x. [DOI] [PubMed] [Google Scholar]
- 12.Kim CS, Choi SH, Chai JK, Cho KS, Moon IS, Wikesjo UM, et al. Periodontal repair in surgically created intrabony defects in dogs: influence of the number of bone walls on healing response. J Periodontol. 2004;75:229–235. doi: 10.1902/jop.2004.75.2.229. [DOI] [PubMed] [Google Scholar]
- 13.Kim HY, Kim CS, Jhon GJ, Moon IS, Choi SH, Cho KS, et al. The effect of safflower seed extract on periodontal healing of 1-wall intrabony defects in beagle dogs. J Periodontol. 2002;73:1457–1466. doi: 10.1902/jop.2002.73.12.1457. [DOI] [PubMed] [Google Scholar]
- 14.Wikesjo UM, Hagen K, Nielsen DD. Periodontal repair in dogs: effect of saliva contamination of the root surface. J Periodontol. 1990;61:559–563. doi: 10.1902/jop.1990.61.9.559. [DOI] [PubMed] [Google Scholar]
- 15.Wikesjo UM, Lim WH, Razi SS, Sigurdsson TJ, Lee MB, Tatakis DN, et al. Periodontal repair in dogs: a bioabsorbable calcium carbonate coral implant enhances space provision for alveolar bone regeneration in conjunction with guided tissue regeneration. J Periodontol. 2003;74:957–964. doi: 10.1902/jop.2003.74.7.957. [DOI] [PubMed] [Google Scholar]
- 16.Wikesjo UM, Nilveus R. Periodontal repair in dogs: effect of wound stabilization on healing. J Periodontol. 1990;61:719–724. doi: 10.1902/jop.1990.61.12.719. [DOI] [PubMed] [Google Scholar]
- 17.Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–428. doi: 10.1037//0033-2909.86.2.420. [DOI] [PubMed] [Google Scholar]
- 18.Jomha NM, Anoop PC, Elliott JA, Bagnall K, McGann LE. Validation and reproducibility of computerised cell-viability analysis of tissue slices. BMC Musculoskelet Disord. 2003;4:5. doi: 10.1186/1471-2474-4-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Barnhart HX, Song J, Haber MJ. Assessing intra, inter and total agreement with replicated readings. Stat Med. 2005;24:1371–1384. doi: 10.1002/sim.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Barnhart HX, Haber MJ, Lin LI. An overview on assessing agreement with continuous measurements. J Biopharm Stat. 2007;17:529–569. doi: 10.1080/10543400701376480. [DOI] [PubMed] [Google Scholar]
- 21.Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45:255–268. [PubMed] [Google Scholar]
- 22.Wang SF, Leknes KN, Zimmerman GJ, Sigurdsson TJ, Wikesjo UM, Selvig KA. Intra- and inter-examiner reproducibility in constant force probing. J Clin Periodontol. 1995;22:918–922. doi: 10.1111/j.1600-051x.1995.tb01795.x. [DOI] [PubMed] [Google Scholar]
- 23.Shaoi J, Zhong B. Assessing the agreement between two quantitative assays with repeated measurements. J Biopharm Stat. 2004;14:201–212. doi: 10.1081/BIP-120028515. [DOI] [PubMed] [Google Scholar]
- 24.Shultz SJ, Nguyen AD, Windley TC, Kulas AS, Botic TL, Beynnon BD. Intratester and intertester reliability of clinical measures of lower extremity anatomic characteristics: implications for multicenter studies. Clin J Sport Med. 2006;16:155–161. doi: 10.1097/00042752-200603000-00012. [DOI] [PubMed] [Google Scholar]
- 25.Kingman A, Albandar JM. Methodological aspects of epidemiological studies of periodontal diseases. Periodontol 2000. 2002;29:11–30. doi: 10.1034/j.1600-0757.2002.290102.x. [DOI] [PubMed] [Google Scholar]