Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Nov 1.
Published in final edited form as: J Craniofac Surg. 2019 Nov-Dec;30(8):2308–2312. doi: 10.1097/SCS.0000000000005679

Brazilian-Portuguese Linguistic Validation of the Velopharyngeal Insufficiency Effects on Life Outcome Instrument

Rafael Denadai *, Cassio Eduardo Raposo-Amaral *, Anelise Sabbag *, Rafael Andrade Ribeiro *, Celso Luiz Buzzo *, Cesar Augusto Raposo-Amaral *, Man Hung †, Jonathan R Skirko ‡
PMCID: PMC6834880  NIHMSID: NIHMS1528397  PMID: 31233001

Abstract

BACKGROUND:

The purpose of this study was to conduct a linguistic validation of the velopharyngeal insufficiency (VPI) Effects on Life Outcome (VELO) instrument for use in Brazilian-Portuguese patients with VPI.

METHODS:

The original English version of the VELO instrument was translated into Brazilian-Portuguese, back-translated, and adapted among the Brazilian patients (n=21) with VPI and their parents, based on the standardized guidelines for the cross-culture adaption process. Discrepancies in the forward and backward translation steps were computed. Comprehension rates were captured for each debriefing interview. The content validity index (CVI) per item (I-CVI) and of the scale (S-CVI universal agreement [S-CVI/UA] and averaging [S-CVI/Ave]) were calculated.

RESULTS:

Reconciliation of the two forward translations and the comparison between the back translation and the original VELO version resulted in some item wordings with discrepancies which were reviewed by the research team (translators, expert committee, and original developers of instrument). Three rounds of cognitive interviews also led to some revisions of wording. Comprehension rates of patients and their parents were 60-100%, 80-100%, and 100% in the first, second, and third rounds of cognitive interviews, respectively. The I-CVI, S-CVI/AU, and SCI/Ave for the Brazilian-Portuguese VELO version were 0.83 (or higher), 0.83 (or higher), and 0.97, respectively.

CONCLUSIONS:

The linguistic validation process of the VELO instrument created a cross-culturally equivalent Brazilian-Portuguese version for use in Brazilian-Portuguese speaking patients with VPI.

Keywords: Brazilian-Portuguese language, Linguistic validation, Quality of Life, Velopharyngeal insufficiency, VELO instrument

INTRODUCTION

Velopharyngeal insufficiency (VPI), a disorder of speech production and swallowing due to inadequate velopharyngeal closure,1,2 can severely and significantly affects many aspects of health-related quality of life (HRQOL).3,4 However, most of the existing literature have been utilized generic HRQOL measures, which are broad-based tools that typically lack sensitivity for specific conditions within unique patient populations and may overlook pertinent issues.3,4

In this context, the VPI Effects on Life Outcome (VELO) instrument, a VPI-specific patient-reported outcome (PRO) and proxy-reported outcome (ProxRO), was developed by Skirko et al.5 using focus groups (which provide face validity) to measure concepts that matter to patients with VPI and their parents, the effects of how VPI affects patients and parents, and the changes after therapeutic interventions.5-7 Its precursor VPI Quality-of-Life (VPIQL) instrument is a 45-item questionnaire originally developed from focus groups which also provides face validity.4 The VPIQL was modified to reduce burden and refine questions, resulting in the VELO instrument. 5

The original English-language version of the VELO instrument is a psychometrically validated and effective PRO/ProxRO measure to assess VPI-specific HRQOL.5-8 However, it cannot be used in non-English speaking populations as a careful process of cultural adaptation and linguistic validation in the specific cultural context is required to achieve maximum semantic, idiomatic, experiential, and conceptual equivalence between the source and target instruments.9,10. The VELO instrument was recently translated for Chinese-speaking, Spanish-speaking, Dutch-speaking populations,11-13 but it is not currently translated to use in the Brazilian-Portuguese-speaking population.

The purpose of this study was to conduct a linguistic validation of the VELO instrument to Brazilian-Portuguese for VPI-specific HRQOL measuring in patients with VPI.

METHODS

This linguistic validation study was carried out at the SOBRAPAR Hospital after approval by the Ethics Committee of the Institution, and in accordance with the Helsinki Declaration of 1975, as amended in 1983.

VELO Instrument

The VELO instrument includes a 26-item parent version (VELO-Parent) and a 23-item youth version (completed by patients 8 years and older; VELO-Youth). The response format is a 5-point Likert-type scale ranging from never (0) to almost always (4). The total score (VELO-Parent Total or VELO-Youth Total) ranges from 0 to 100 with 100 representing the highest QOL. Subscales are scored similarly and include Speech, Swallow, Situational Difficulty, Perception, and Emotional for both the VELO-Parent and VELO-Youth, and also includes Caregiver Impact for the VELO-Parent.5-7

Cross-cultural Adaptation Process

To obtain a scientifically accurate translation and cross-culturally adaptation of the original English VELO version into Brazilian-Portuguese language, we followed linguistic validation principles detailed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and World Health Organization (WHO).10,141. Permission was obtained from the authors of the original VELO instrument. An expert committee was assembled, composed by members of a multidisciplinary cleft team (2 speech language therapists, 2 psychologists, and 2 plastic surgeons) with experience in management of VPI, knowledge of Brazilian-Portuguese and English languages, and on the research methodology. A member (R.D.) of the expert committee team was assigned as the project manager. 2. To identify the concepts to be measured, the committee reviewed and discussed the original publications involving the VELO instrument.5-7 3. Two forward translations were performed by two independent translators (Brazilian-Portuguese native speakers with proficiency in English) using simple terminology, rather than literal translations. 4. A consensus harmonized version was constructed by the expert committee and translators. The suggestions were analyzed and justified, in order to achieve the better understandability of instrument and the equivalence with Brazilian culture. 5. The consensus harmonized version was back-translated into English by an independent translator (English native speaker and fluent in Brazilian-Portuguese language). 6. The back-translated version was reviewed by the authors of the original instrument, in order to ensure semantic and idiomatic equivalences between the original and translated versions. Items whose meaning was not maintained were re-translated in an iterative manner until an acceptable result was achieved, leading to the approved harmonized version. 7. The expert committee meeting evaluated possible changes to ensure the Brazilian-Portuguese version was understandable and measured all clinically relevant issues, leading to consensus on the pre-interview version. 8. Cognitive debriefing interviews and post-cognitive interview review were performed. 9. The final Brazilian-Portuguese version (linguistically validated translation) of the VELO instrument constructed by consensus/proofreading was then ready for psychometric validation (Fig. 1).

FIGURE 1.

FIGURE 1.

Open in a new tab

Step sequence for the linguistic validation process of Brazilian-Portuguese version of the VELO instrument.

Cognitive Debriefing Interviews

A cognitive debriefing interview was performed with a group of 7 VPI patients and their parents in interview to ensure that the meaning of the instruction, recall period, items, and response options are the same, and that the wording is appropriate.15,16 We conducted a face-to-face interview with each group by reading the translated instrument and the script form provided by the original authors of the VELO instrument.5 Participants were encouraged to make comments, express their difficulties, and give their opinion on possible changes that they thought would enable comprehension. They were also requested to paraphrase each sentence in the instrument and explain why any words were difficult to understand. The interviewer judged whether items were correctly paraphrased and recorded any comprehension problems or proposed changes to the wording. Items deemed problematic by two or more VPI patients or their parents (VELO-Youth and VELO-Parent, respectively) were revised for clarity.17 If alterations were undertaken, an additional round of cognitive interview was held to evaluate the modified instrument. Comprehension rates (defined as total percentage of subjects who were able to successfully paraphrase the items within this instrument) were captured for each debriefing interview.18 Interviews were conducted until a pre-established percentage (≥ 90%) of understanding was achieved for all items.19

Subjects

Patients with VPI and their parents were recruited for the cognitive interviews during the cross-cultural adaptation phase of the VELO. Patients diagnosed with VPI by speech pathologist team using perceptual speech and nasal endoscopy evaluations20-22 were enrolled randomly from the VPI clinics. Exclusion criteria included non–native Brazilian Portuguese speaker or patients with severe cognitive delay status.

Content Validity

The expert committee assessed content validity of the Brazilian-Portuguese VELO version. Content validity was measured by calculating the content validity index (CVI), using ratings of item relevance by experts.23-25 Experts were asked to rate each item in terms of its relevance to the underlying construct on a 4-point ordinal scale (1 [not relevant], 2 [somewhat relevant], 3 [quite relevant], 4 [highly relevant]). We calculated the CVI per item (I-CVI) and of the scale (S-CVI). The I-CVI was computed as the number of experts giving a rating of either 3 or 4, divided by the total number of experts. The S-CVI/UA (universal agreement) was computed as the proportion of items on an instrument that achieved a rating of 3 or 4 by all the experts. The S-CVI/Ave (averaging) was computed as the average of the I-CVIs by summing them and dividing by the number of items. It was recommended that an instrument with excellent content validity should be composed of I-CVIs, S-CVI/UA, and S-CVI/Ave of 0.78, 0.80, and 0.90 or higher, respectively.23-25

RESULTS

Forward Translations

Reconciliation of the two independent forward translations revealed some discrepancies related to the wording of the items. Instructions, recall period, and response options were translated without discrepancies. A total of 49 items (23 and 26 items for VELO-Youth and VELO-Parent version, respectively) were reviewed. Most items presented discrepancies as some similar words (e.g., “talk” and “speech”) were repeated in more than one item. Further examples were “understood” and “understand” which appeared in 8 (16.3%) and 7 (14.3%) items, respectively. Considering these repeated words only once, there were 5 (10.2%) items with discrepancies. These discrepancies were properly resolved after a consensus meeting held between the two forward translators and the expert committee (Table 3-SDC).

Back Translation

The backward translation was compared to the original English VELO, with no discrepancies in the instructions, recall period, and response options. It revealed some discrepancies in specific wordings of four items (8.2%; without considering repeated words between items) (Table 4-SDC). These translated words were considered to have a different meaning than the original version and required revision (re-translation until a satisfactory result was achieved), such as the replacement of the word “unintelligent” by “not smart” (i.e., “He/She is treated as unintelligent due to his/her speech” was replaced by “He/She is treated as not smart due to his/her speech”). Additional explanations of the exact context provided by the instrument developers were crucial to maintaining the content, such as “spends a lot of time alone” for the context of item 19 (“He/She is shy or isolated because of his/her speech” was replaced by “He/She is shy or withdraw isolated because of his/her speech”).

Cognitive Debriefing Interviews

Patients with VPI and their parents (Table 5-SDC) were encouraged to make comments, express their difficulties, and give their opinion on possible changes that they thought would enable comprehension. Overall, the participants had a good understanding of the form filling, titles, instructions, recall period, and response items. Only specific words were difficult to understand or interpret. These words were addressed by the expert committee and were revised mainly to clarify their meaning or to add an example. In the first round of cognitive interview, all of items (38, 77.6%) remained the same, except 11 (22.4%) items were revised. In the second round, 1 (2%) item was revised. In the third round, all participants paraphrased the items with ease and reported no difficulty in comprehending the items (Table 6-SDC). Since no difficulty was expressed for any item/word and comprehension rates for VELO-Youth (23 items) and VELO-Parent (26 items) were 100%, no additional changes or round of cognitive interview were implemented (See Table, Supplemental Digital Content 1).

Content Validity

The 49 items of the translated VELO instrument (23 and 26 items of the VELO Youth and VELO Parent, respectively) were assessed by the expert committee, for a total of 294 evaluations, of which 286 (97.3%) were rated as 3 or 4 (“quite relevant” or “highly relevant”, respectively). The I-CVI, S-CVI/AU, and SCI/Ave for the Brazilian-Portuguese VELO version were 0.83 (or higher), 0.83 (or higher), and 0.97, respectively (See Table, Supplemental Digital Content 2).

Final Version

The final cross-culturally adapted, Brazilian-Portuguese version of the VELO preserved the main features of the original English instrument. Minor changes to spelling and punctuation were required upon final proofreading of the Brazilian-Portuguese version of the VELO instrument. The standardization of format and layout (uppercase and lowercase, and bold) was thoroughly addressed, as in the original instrument.

DISCUSSION

Simply translating and adopting an existing PRO/ProxRO instrument in another linguistic or cultural context will lead to unknown psychometric functioning and validity of the instrument in the target language and may even hamper research outcomes, and may lead to erroneous conclusions.9,10,26-28 Health care professionals seeking to adopt these PRO/ProxRO questionnaires will best serve their patients by following the linguistic validation process including translation, cross-cultural adaptation, and cognitive interviewing of the resulting translations in the target population.9,10,26-28

For this linguistic validation study, we adopted the ISPOR and WHO recommendations.10,14 The developers of the original instrument checked in detail all steps of this linguistic validation process; from the initial planning to the final VELO version, the constant discussion between the project manager and the developer of original VELO instrument allowed the longitudinal evolution of the whole process. Discrepancies were mainly related to choice or meaning of specific words, which were modified generate the pre-interview Brazilian-Portuguese version.

In the linguistic validation process, the composition of subjects in the cognitive interviews helps ensure the translation is linguistically valid in the target population.10,15,16 As in the original VELO report,5 subjects were selected to represent, as well as possible, the population that the VELO instrument will target, namely VPI patients of different ages, with or without associated syndrome or previous VPI surgery, and with no restriction related to gender, ethnicity, educational level, or socioeconomic status. All the parents of these patients participated actively in the cognitive interviews. Patients aged 8 years or greater were part of the interviews, as the age limit of 8 years is a key cutoff for self-report studies.29,30 Patients under age 8 years old (n=6; 2 in each round of cognitive interview) did not actively respond to the interviews, as defined in the VELO instructions.5-7

The core goal of the cognitive interviews was to optimize the language until reaching a point where there was sufficient evidence of no remaining problems with patient/parent comprehension of the draft items.16 In this study, the comprehension rates of both patients and their parents were 60–100%, 80–100%, and 100% in the first, second, and third rounds of cognitive interviews, respectively. As there is no consensus on the number of rounds of cognitive interviews required,15-17 modifications were made to the Brazilian-Portuguese version until a comprehension rate of at least 90 percent was achieved for all items.19

The content validity of the VELO instrument was achieved by the use of focus groups in the original development of the English version.5 Although previous validation studies of the VELO instrument did not include content validity assessment,11-13 the Brazilian-Portuguese VELO version was tested for content validity by a panel of experts that focused on the relevance of the items. The experts were not in total agreement on the content validity, particularly in the VELO subscale Swallowing Problems. This was probably because the most striking aspects of VPI are related to functional problems with speech as highlighted in surgical and nonsugical VPI reports.1,2 It may also have been influenced, at least partially, by the training background and the focus in the practice of each professional, as part of the experts has focused efforts on the speech outcomes of patients with VPI.20-22 However, it is important to emphasize that the VELO instrument covers a wide spectrum of functional and psychosocial nuances potentially involved in patients with VPI, and all CVI values (I-CVIs, S-CVI/UA, and S-CVI/Ave) were considered excellent for all items and the overall instrument. We adopted the CVI as it is the most widely utilized method to quantify content validity for an instrument.23-25,32 In addition, it is simple and easy to compute, easily understood and interpreted, and provides content validity of each item and the instrument as a whole.23,25,31 CVI has been criticized because its calculation process collapses a 4-point ordinal scaling response (1 to 4) into a two-point dichotomous nominal scale (1 and 2 versus 3 and 4); this diminishes its ability to discriminate varying levels of agreement between experts and to adjust for chance.32-34 We have not included further inter-rater agreement tests (e.g., multi-rater kappa or intraclass correlation coefficient) as these statistical indexes were designed and developed to evaluate the general inter-rater agreement but not for the purpose of quantifying the content validity.34-36 Although these indexes adjust the risk of the chance agreement, they consider the full inter-rater agreement regardless the type; if the raters all disagree on an item, the index will capture and count this as an agreement. 34-36

Potential caveats of our study should be addressed. In this linguistic validation process, the words “youth”, “child”, and “kids” were questioned by older patients with VPI (“I am not a child”; “I am not youth”), their parents (“My son is not a child”), and members of the expert team (“There are patients who are not pediatric”). For example, in Brazil, “child” is adopted essentially for the pediatric population as well as possessing a pejorative meaning (“a little serious person and of little judgment”) in some contexts. More generic and broad alternatives have been suggested (e.g., replace “youth” by “patient” or “son/daughter”). However, the terms were maintained as in the original English version as there were no translation discrepancies and/or comprehension problems. Because there may be centers with older patients who are being treated for VPI, a specifically designed and tested VELO-Adult version is in progress.

For the formation of the committee of experts, we did not include all the professionals potentially involved in the VPI management. Although we have followed recommendations for the selection of these experts (e.g., knowledge on the subject, clinical experience, published in the area),24,37 the involvement of other disciplines (e.g., social workers, orthodontists, anesthesiologists, otolaryngologist or head and neck surgeon) might influence the CVI evaluation. The linguistic validation process provided evidence of transferability of the VELO instrument into Brazilian-Portuguese language, but additional tests should be conducted on the psychometric properties of the translated instrument. Further linguistic validation studies in other languages have also been initiated, and a multi-institutional data bank is planned to be established for VPI-specific HRQOL measuring in patients with VPI.

CONCLUSION

The linguistic validation process of the VELO instrument created a cross-culturally equivalent Brazilian-Portuguese version for use in Brazilian-Portuguese speaking patients with VPI.

Supplementary Material

Supp. Table

TABLE 1, SUPPLEMENTAL DIGITAL CONTENT 1, which displays comprehension rates for VELO-Youth and VELO-Parent according to three rounds of cognitive debriefing interviews.

TABLE 2, SUPPLEMENTAL DIGITAL CONTENT 2, which displays ratings on a 23-item VELO Youth and 26-item VELO Parent by six experts: items rated 3 or 4 (“quite relevant” or “highly relevant”, respectively) on a 4-point relevance scale.

Acknowledgments

Funding: The authors received no financial support for the research, authorship, and/or publication of this article.

Footnotes

Declaration of Conflicting Interests: The authors report no conflicts of interest.

REFERENCES

  • 1.Kummer AW. A Pediatrician's Guide to Communication Disorders Secondary to Cleft Lip/Palate. Pediatr Clin North Am 2018;65:31–46 [DOI] [PubMed] [Google Scholar]
  • 2.Rudnick EF, Sie KC. Velopharyngeal insufficiency: current concepts in diagnosis and management. Curr Opin Otolaryngol Head Neck Surg 2008;16:530–535 [DOI] [PubMed] [Google Scholar]
  • 3.Gomersall T, Spencer S, Basarir H, Tsuchiya A, Clegg J, Sutton A, Dickinson K. Measuring quality of life in children with speech and language difficulties: a systematic review of existing approaches. Int J Lang Commun Disord 2015;50:416–435 [DOI] [PubMed] [Google Scholar]
  • 4.Barr L, Thibeault SL, Muntz H, de Serres L. Quality of life in children with velopharyngeal insufficiency. Arch Otolaryngol Head Neck Surg 2007;133:224–229 [DOI] [PubMed] [Google Scholar]
  • 5.Skirko JR, Weaver EM, Perkins J, Kinter S, Sie KC. Modification and evaluation of a Velopharyngeal Insufficiency Quality-of-Life instrument. Arch Otolaryngol Head Neck Surg 2012;138:929–935 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Skirko JR, Weaver EM, Perkins JA, et al. Validity and responsiveness of VELO: a velopharyngeal insufficiency quality of life measure. Otolaryngol Head Neck Surg 2013;149:304–311 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Skirko JR, Weaver EM, Perkins JA, Kinter S, Eblen L, Martina J, Sie KC. Change in Quality of Life with Velopharyngeal Insufficiency Surgery. Otolaryngol Head Neck Surg 2015;153:857–864 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Bhuskute A, Skirko JR, Roth C, Bayoumi A, Durbin-Johnson B, Tollefson TT. Association of Velopharyngeal Insufficiency With Quality of Life and Patient-Reported Outcomes After Speech Surgery. JAMA Facial Plast Surg 2017;19:406–412 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: Literature review and proposed guidelines. J Clin Epidemiol 1993;46:1417–1432 [DOI] [PubMed] [Google Scholar]
  • 10.Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, Erikson P; ISPOR Task Force for Translation and Cultural Adaptation. Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Health 2005;8:94–104 [DOI] [PubMed] [Google Scholar]
  • 11.Huang H, Chen N, Yin H, Skirko JR, Guo C, Ha P, Li J, Shi B. Validation of the Chinese Velopharyngeal Insufficiency Effects on Life Outcomes Instrument. Laryngoscope. 2019. doi: 10.1002/lary.27792. [Epub ahead of print] [DOI] [PubMed] [Google Scholar]
  • 12.Santillana R, Skirko J, Roth C, Tollefson TT. Spanish Linguistic Validation for the Velopharyngeal Insufficiency Effects on Life Outcomes. JAMA Facial Plast Surg 2018;20:331–332 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Bruneel L, Van Lierde K, Bettens K, Corthals P, Van Poel E, De Groote E, Keppler H. Health-related quality of life in patients with cleft palate: Validity and reliability of the VPI Effects on Life Outcomes (VELO) questionnaire translated to Dutch. Int J Pediatr Otorhinolaryngol 2017;98:91–96 [DOI] [PubMed] [Google Scholar]
  • 14.WHO (World Health Organization). Process of translation and adaptation of instruments. http://www.who.int/substance_abuse/research_tools/translation/en/. Accessed 02 Feb 2019
  • 15.Willis G Cognitive Interviewing. Sage Publications, Inc., Thousand Oaks, CA; 2005. [Google Scholar]
  • 16.Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, Ring L. Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2--assessing respondent understanding. Value Health 2011;14:978–988 [DOI] [PubMed] [Google Scholar]
  • 17.Leidy NK, Vernon M. Perspectives on patient-reported outcomes: content validity and qualitative research in a changing clinical trial environment. Pharmacoeconomics 2008;26:363–370 [DOI] [PubMed] [Google Scholar]
  • 18.Piault E, Doshi S, Brandt BA, Angün Ç, Evans CJ, Bergqvist A, Trocio J. Linguistic validation of translation of the Self-Assessment Goal Achievement (SAGA) questionnaire from English. Health Qual Life Outcomes 2012;10:40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Reichenheim ME, Moraes CL. Operationalizing the cross-cultural adaptation of epidemiological measurement instruments. Rev Saude Publica 2007;41:665–673 [DOI] [PubMed] [Google Scholar]
  • 20.Denadai R, Sabbag A, Raposo-Amaral CE, Filho JCP, Nagae MH, Raposo-Amaral CA. Bilateral buccinator myomucosal flap outcomes in nonsyndromic patients with repaired cleft palate and velopharyngeal insufficiency. J Plast Reconstr Aesthet Surg 2017;70:1598–1607 [DOI] [PubMed] [Google Scholar]
  • 21.Mangilli LD, Sabbag A. Speech therapy in cleft patients In: Alonso N, Raposo-Amaral CE, eds. Cleft Lip and Palate Treatment: A Comprehensive Guide. 1st ed. Springer, Cham; 2018:215–223 [Google Scholar]
  • 22.Denadai R, Raposo-Amaral CE, Sabbag A, Raposo-Amaral CA. Surgical Management of Velopharyngeal Insufficiency: The SOBRAPAR Hospital Algorithm In: Alonso N, Raposo-Amaral CE, eds. Cleft Lip and Palate Treatment: A Comprehensive Guide. 1st ed. Springer, Cham; 2018:194–204 [Google Scholar]
  • 23.Lynn MR. Determination and quantification of content validity. Nurs Res 1986;35:382–385 [PubMed] [Google Scholar]
  • 24.Davis LL. Instrument review: Getting the most from a panel of experts. Appl Nurs Res 1992;5:194–197 [Google Scholar]
  • 25.Polit DF, Beck CT. The content validity index: are you sure you know what's being reported? Critique and recommendations. Res Nurs Health 2006;29:489–497 [DOI] [PubMed] [Google Scholar]
  • 26.Breugelmans R Dangers in using translated medical questionnaires. The importance of conceptual equivalence across languages and cultures in patient-reported outcome measures. Chest 2009;136:1175–1177 [DOI] [PubMed] [Google Scholar]
  • 27.Eremenco S, Pease S, Mann S, Berry P; PRO Consortium’s Process Subcommittee. Patient-Reported Outcome (PRO) Consortium translation process: consensus development of updated best practices. J Patient Rep Outcomes 2017;2:12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Acquadro C, Conway K, Hareendran A, Aaronson N, European Regulatory Issues and Quality of Life Assessment (ERIQA) Group Literature review of methods to translate health-related quality of life questionnaires for use in multinational clinical trials. Value Health 2008;11:509–5121 [DOI] [PubMed] [Google Scholar]
  • 29.Bevans KB, Riley AW, Moon J, Forrest CB. Conceptual and methodological advances in child-reported outcomes measurement. Expert Rev Pharmacoecon Outcomes Res 2010;10:385–96 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Matza LS, Patrick DL, Riley AW, Alexander JJ, Rajmil L, Pleil AM, Bullinger M. Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force. Value Health 2013;16:461–479 [DOI] [PubMed] [Google Scholar]
  • 31.Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007;30:459–467 [DOI] [PubMed] [Google Scholar]
  • 32.Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res 2003;25:508–518 [DOI] [PubMed] [Google Scholar]
  • 33.Beckstead JW. Content validity is naught. Int J Nurs Stud 2009;46:1274–83 [DOI] [PubMed] [Google Scholar]
  • 34.Banerjee M, Capozzoli M, McSweeney L, Sinha D. Beyond kappa: A review of interrater agreement measures. Can J Stat 1999;27:3–23 [Google Scholar]
  • 35.O’Neill TA. An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners. Front Psychol 2017;8:777. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Jang JH, Manatunga AK, Taylor AT, Long Q. Overall indices for assessing agreement among multiple raters. Stat Med 201837:4200–4215 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Grant JS, Davis LL. Selection and use of content experts for instrument development. Res Nurs Health 1997;20:269–274 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supp. Table

TABLE 1, SUPPLEMENTAL DIGITAL CONTENT 1, which displays comprehension rates for VELO-Youth and VELO-Parent according to three rounds of cognitive debriefing interviews.

TABLE 2, SUPPLEMENTAL DIGITAL CONTENT 2, which displays ratings on a 23-item VELO Youth and 26-item VELO Parent by six experts: items rated 3 or 4 (“quite relevant” or “highly relevant”, respectively) on a 4-point relevance scale.

NIHMS1528397-supplement-Supp__Table.pdf (210.5KB, pdf)

RESOURCES