Skip to main content
NCBI home page
Saudi Journal of Anaesthesia logoLink to Saudi Journal of Anaesthesia
. 2026 Jan 2;20(1):27–34. doi: 10.4103/sja.sja_407_25

Development and validation of a questionnaire for feedback responses regarding the usefulness of the anatomical model to teach lung isolation skills

Swathi Nagaraja 1,✉, Mamatha Chikkanarasimha 1, Bhargavi Sanket 1, Sudheesh Kannan 1, Nethra Surhonne 1, Sandhya Kalappa 1
PMCID: PMC12912470  PMID: 41710629

Abstract

Background and Aims:

An anatomical model was developed in-house to teach novices lung isolation skills bedside. To know the utility of the model, we planned a survey following a structured teaching program. The questionnaire used for feedback purpose was tested for content validity and has been detailed.

Materials and Methods:

Eleven questions were included in the survey. A Google form was prepared and mailed to 12 content experts to grade each question. In the Microsoft excel sheet, item-wise content validity index (I-CVI), scale-wise content validity index (S-CVI) and modified kappa statistic (κ) were calculated. The I-CVI of 0.78 and the S-CVI/average of 0.90 were considered acceptable.

Results:

Eleven out of 12 content experts responded. SCVI and S-CVI/Ave were uniformly 0.99 and 0.91, respectively, for relevance, simplicity, clarity, and ambiguity for all 11 questions.

Conclusion:

The questionnaire designed to obtain feedback regarding the anatomical model to teach lung isolation techniques met the content validity in both qualitative and quantitative terms.

Keywords: Anatomical model, feedback, questionnaire, survey, reliability and validity, program evaluation

Introduction

The simulation technology improves cognitive and psychomotor skills of basic and advanced airway skills.[1,2,3]

We devised a simple anatomical simulator to teach advanced lung isolation skills to novices. To assess the usefulness of the model, we planned to conduct a survey on postgraduate residents. Postsurvey, we intended to distribute a questionnaire to them to get feedback regarding the utility of the model. The reliability and validation of the questionnaire tool had to be tested by quantification of content validity.[4]

We describe the content validity of the questionnaire in two steps. Content validity index (CVI) and modified kappa statistic (κ) using the probability of chance agreement (Pc) were used to estimate the content validity of the questionnaire developed to know the usefulness of the model in both qualitative and quantitative terms.

Materials and Methods

After obtaining institutional ethics committee clearance with REF no: BMCRI/EC/15/23-24 dated 16.09.2023 and CTRI registration number: CTRI/2023/10/059129 dated 03.11.2023, we planned a survey among anesthesia residents of our institution who lacked knowledge about lung isolation to assess the effectiveness and utility of the model in training lung isolation techniques. In the anatomical model, materials from condemned circuits of the anesthesia workstation have been used to mimic the structures of the trachea-bronchial tree. In order to objectively assess the utility of the model, we in our structured teaching program consisting of video-didactic lectures regarding lung isolation techniques following which we would ask the residents to perform lung isolation techniques on the model. After this teaching program, we would distribute the prepared validated questionnaire to them to obtain their feedback responses regarding the model. We describe the validation process of the questionnaire as follows.

A questionnaire related to feedback responses from residents participating in the survey, which would be done to know the utility of the in-house made anatomical model in learning lung isolation techniques, was prepared. It consisted of 11 questions prepared by three anesthetists based on literature review and their experience with lung isolation techniques, which was discussed with senior anesthetists [Annexure 1].

A Google form was prepared to grade each question based on clarity, simplicity, ambiguity, and relevance on a 4-point Likert scale [Annexure 2]. In order to validate the questionnaire, it was mailed to 12 Anaesthesia postgraduates of other medical institutes of the contemporary batch who acted as content experts. They were expected to revert within 15 days; the expected response rate was 50%. Their response was related to clarity, simplicity, ambiguity, and relevance (Grading 1-4) adopted from Yaghmaie[5] The proportion of experts who agreed about item relevance provides the quantitative measure of CVI. Then, the CVI value for each item on the scale (I-CVI), overall scale (S-CVI), and S-CVI/Average was computed using formulae as given by Polit et al.[6,7]

Item-wise CVI was calculated for each question using the formula:

graphic file with name SJA-20-27-g001.jpg

Scale content validity index (S-CVI) is the proportion of items given a rating of 3 or 4 by all content experts ratings involved. The average proportion of items rated as 3 or 4 across various content experts is S-CVI/Ave (Average), which was computed using the formula:

graphic file with name SJA-20-27-g002.jpg

If I-CVI is less than 0.78 and S-CVI/Average <0.90, the question would be changed as per suggestion by the reviewing content experts. For better quantification, the kappa coefficients, representing the proportion of agreement after the chance agreement is removed (κ), were calculated, and the strength of agreement, which indicated data variability (Pc), was also reported. MKS, which was used to evaluate the agreement between the experts for every question in each domain, was calculated using the formula:

graphic file with name SJA-20-27-g003.jpg

where Pc is probability of chance agreement calculated by the formula:

graphic file with name SJA-20-27-g004.jpg

where N = Number of experts; A = Number of experts in agreement on relevance.

Modified kappa statistic and strength of agreement were graded as <0.40 = poor, 0.40–0.59 = fair, 0.60–0.74 = good, and 0.75–1.00 = excellent as recommended by Cinchetti and Fleiss.[8] After modification and validation, the final survey questions were planned to be given to the survey participants following the teaching program.

Results

Response was obtained from 11 of 12 experts.

The experts who gave a score lesser than or equal to 2 are depicted in Tables 1-4. None of them gave any additional remarks for change or modification. All 11 questions received SCVI and S-CVI/Ave uniformly 0.99 and 0.91, respectively, for relevance, simplicity, clarity, and ambiguity. Hence, all questions were acceptable as they were. The probability of chance agreement and modified kappa statistic was graded “Excellent” for all questions in all domains as they were between the range of 0.75–1.00.

Table 1.

Content validity for relevance

Question number in questionnaire Number of experts in agreement I-CVI Probability of chance agreement Modified kappa statistic Interpretation
1 10 (Expert 2 gave a score of 2) 0.91 0.005 0.895 Excellent
2 11 1 0.0004 0.996 Excellent
3 11 1 0.0004 0.996 Excellent
4 11 1 0.0004 0.996 Excellent
5 11 1 0.0004 0.996 Excellent
6 11 1 0.0004 0.996 Excellent
7 11 1 0.0004 0.996 Excellent
8 11 1 0.0004 0.996 Excellent
9 11 1 0.0004 0.996 Excellent
10 11 1 0.0004 0.996 Excellent
11 11 1 0.0004 0.996 Excellent
Open in a new tab

Mean S-CVI=0.99; S-CVI/Average=0.91as per formulae by Polit et al.[6,7] *CVI: Content Validity Index; I-CVI: Item-wise CVI; S-CVI: Scale-wise CVI

Table 4.

Content validity for ambiguity

Question number in questionnaire Number of experts in agreement I-CVI Probability of chance agreement Modified kappa statistic Interpretation
1 10 (Expert 9 gave a score of 2) 0.91 0.005 0.895 Excellent
2 11 1 0.0004 0.996 Excellent
3 11 1 0.0004 0.996 Excellent
4 11 1 0.0004 0.996 Excellent
5 11 1 0.0004 0.996 Excellent
6 11 1 0.0004 0.996 Excellent
7 11 1 0.0004 0.996 Excellent
8 11 1 0.0004 0.996 Excellent
9 11 1 0.0004 0.996 Excellent
10 11 1 0.0004 0.996 Excellent
11 11 1 0.0004 0.996 Excellent
Open in a new tab

Mean S-CVI=0.99; S-CVI/Average=0.91as per formulae by Polit et al.[6,7] *CVI: Content Validity Index; I-CVI: Item-wise CVI; S-CVI: Scale-wise CVI

Table 2.

Content validity for clarity

Question number in questionnaire Number of experts in agreement I-CVI Probability of chance agreement Modified kappa statistic Interpretation
1 10 1 0.0004 0.996 Excellent
2 11 1 0.0004 0.996 Excellent
3 11 1 0.0004 0.996 Excellent
4 11 1 0.0004 0.996 Excellent
5 10 (Expert 6 gave a score of 1) 0.91 0.005 0.895 Excellent
6 11 1 0.0004 0.996 Excellent
7 11 1 0.0004 0.996 Excellent
8 11 1 0.0004 0.996 Excellent
9 11 1 0.0004 0.996 Excellent
10 11 1 0.0004 0.996 Excellent
11 11 1 0.0004 0.996 Excellent
Open in a new tab

Mean S-CVI=0.99; S-CVI/Average=0.91as per formulae by Polit et al.[6,7] *CVI: Content Validity Index; I-CVI: Item-wise CVI; S-CVI: Scale-wise CVI

Table 3.

Content validity for simplicity

Question number in questionnaire Number of experts in agreement I-CVI Probability of chance agreement Modified kappa statistic Interpretation
1 10 1 0.0004 0.996 Excellent
2 11 1 0.0004 0.996 Excellent
3 11 1 0.0004 0.996 Excellent
4 11 1 0.0004 0.996 Excellent
5 10 (Expert 6 gave a score of 1) 0.91 0.005 0.895 Excellent
6 11 1 0.0004 0.996 Excellent
7 11 1 0.0004 0.996 Excellent
8 11 1 0.0004 0.996 Excellent
9 11 1 0.0004 0.996 Excellent
10 11 1 0.0004 0.996 Excellent
11 11 1 0.0004 0.996 Excellent
Open in a new tab

Mean S-CVI=0.99; S-CVI/Average=0.91as per formulae by Polit et al.[6,7] *CVI: Content Validity Index; I-CVI: Item-wise CVI; S-CVI: Scale-wise CVI

Discussion

The survey questions met the content validity criteria through both qualitative and quantitative analyses. Hence, no modifications were required in the initial draft of the questionnaire.

We followed the two-step method by Wynd et al.[9] Domain identification was done through a thorough review of the literature and the authors’ judgement, logic, and reasoning in the first stage. Later in the second stage, content expert reviewers who were contemporary batch postgraduates were expected to have the same level of knowledge as the survey participants who reviewed the items. They served as representatives of the population on whom the survey was planned. They were expected to evaluate the questions critically and rate the items relevant to the domain of content in terms of relevance, simplicity, clarity, and ambiguity. Lynn[10] advised a minimum of 3 experts. However, we chose to include at least six experts as it is expected to yield greater absolute agreement, but greater than 10 experts can increase the risk of chance agreement. So, with 11 experts, there could be two not relevant ratings acceptable with I-CVI = 0.78.

Content experts graded using a 4-point ordinal Likert-type rating scale [Appendix 2]. However, the four ordinal response rankings are collapsed into two dichotomous categories of responses as “content valid” and “content invalid” so that I-CVI becomes a two-category nominal scale.[10]

S-CVI is the proportion of items given a rating of 3 or 4 by all content experts ratings involved. S-CVI of 0.80 or higher is considered acceptable. A higher S-CVI greater than 0.90 is more acceptable.[11] Since our values were more than acceptable, there was no requirement to change or modify any question. The average proportion of items rated as 3 or 4 across various content experts is S-CVI/Ave (Average). This accounts for the performance of each item through the averaging feature.

When dichotomous values of CVI are used for evaluation, there can be concern about chance agreement.[8] To re-examine the inter-rater agreement, a second analysis was performed to eliminate this risk, and a κ value was calculated. It captures agreement of all types. The low κ items in our analysis were consistent with low CVI ratings. The I-CVI was considered excellent when there was perfect agreement among the experts.

The limitation of the questionnaire would be performance bias or also inflated answers by the novice anesthesia residents to whom it would be distributed.

Conclusion

With excellent content validity, the questionnaire to obtain feedback regarding the DIY anatomical model post survey could be validated in terms of covering the content and the domains expected to be measured by both qualitative and quantitative analyses.

Author contributions

SN: Concepts, design, definition of intellectual content, literature search, data acquisition, statistical analysis, manuscript preparation, manuscript editing, guarantor. MC, BS: Concepts, design, definition of intellectual content, literature search, data acquisition, manuscript preparation, guarantor. SK, NS, SK: Concepts, design, definition of intellectual content, manuscript editing, manuscript review, guarantor.

Key message

Validation of questionnaire is an important requirement to standardize the questions which get missed out by researchers. Here, we describe the process in detail and our questionnaire has excellent content validity.

Standardized study reporting requirements

Not applicable since validation of the questionnaire is described.

Conflicts of interest

There are no conflicts of interest.

Data availability statement

Nil.

Clearances

The validation was a part of methodology of survey for which appropriate clearances have been obtained.

Ethical approval

Ethics Committee of Bangalore Medical College and Research Institute; BMCRI/EC/15/23-24; 16.09.2023.

Trial registration

Clinical Trials Registry of India; ctri.nic.in; CTRI/2023/10/059129; 3.11.2023.

Acknowledgement

  1. Dr. Ramachandraiah R. - Professor, Department of Anaesthesia, BMCRI for extending his full support in the conduct of the study.

  2. Mr. Praveen Deepak - For the technical help in developing the in-house model.

  3. All the content experts from ESIC Medical College (Bangalore), Kidwai Memorial Institute of Oncology (Bangalore) and KS Hedge Medical Academy (Mangalore) who validated the questionnaire and respective Head of Departments of Anaesthesia for permitting them.

ANNEXURE 1

  1. Anaesthesia experience as a resident

    1. 1st-year

    2. 2nd-year

    3. 3rd-year

  2. Participated in lung isolation workshop/Continuing Medical Education in last 5 years:

    1. Yes

    2. No

  3. Approximate number of lung isolations performed during residency ……………………

  4. If performed,

    1. Observed

    2. Assisted

    3. Performed independently under supervision

  5. Which lung-isolating device have you used?

    1. Double lumen tube (DLT)

    2. Bronchial blocker (BB)

  6. Were you familiar with the exact steps of lung isolation using a Double lumen tube before this teaching programme?

    1. Very unfamiliar

    2. Unfamiliar

    3. Somewhat familiar

    4. Familiar

    5. Very familiar

  7. Were you familiar with the exact steps of lung isolation using a bronchial blocker before this teaching programme?

    1. Very unfamiliar

    2. Unfamiliar

    3. Somewhat familiar

    4. Familiar

    5. Very familiar

  8. Do you feel confident in lung isolation using Double Lumen Tube after attending this teaching programme?

    1. Not confident

    2. Slightly confident

    3. Somewhat confident

    4. Fairly confident

    5. Completely confident

  9. Do you feel confident in lung isolation using Bronchial blocker after attending this teaching programme?

    1. Not confident

    2. Slightly confident

    3. Somewhat confident

    4. Fairly confident

    5. Completely confident

  10. Which step in lung isolation using Double Lumen Tube/Bronchial Blocker do you feel is confusing and needs more emphasis?

    ………………………………………………………………………………………

    ………………………………………………………………………………………….

    ………………………………………………………………………………….

  11. Your review regarding the DIY anatomical model for learning lung isolation techniques

    ……………………………………………………………………………………………

    ……………………………………………………………………………………………

    …………………………………………………………………

ANNEXURE 2

We have developed an anatomical model of tracheo-bronchial tree to train postgraduate students the techniques of lung isolation. To assess the utility of the model, we have planned a survey wherein the participants (concurrent batch postgraduate students of our institute) will perform the technique on the model once before a video demonstration and later again the after the video demonstration.

A survey has been prepared for them and we require your assistance to validate the survey questions.

Kindly note:

  1. Find attached Word document and the google form.

  2. Appendix 1 titled is the actual survey draft.

STEPS FOR VALIDATION IN THE GOOGLE FORM

  1. There are overall 11 questions.

  2. Each of the question has to be graded in terms of Relevance, Clarity, Simplicity and Ambiguity.

  3. In case you choose the 2nd or 3rd option, you will be redirected to suggest modifications of the survey question.

  4. Based on your valuable input regarding the survey questions, they will be suitably modified for the final draft of the questionnaire.

  5. Kindly fill in the Google form within 15 days of receipt of mail.

GRADING (in google form)

Relevance

1 = not relevant

2 = item needs some revision

3 = relevant but need minor revision

4 = very relevant.

Clarity

1 = not clear

2 = item needs some revision

3 = clear but needs minor revision

4 = very clear.

Simplicity

1 = not simple

2 = item need some revision

3 = simple but need minor revision

4 = very simple.

Ambiguity

1 = doubtful

2 = needs revision

3 = no doubt but need minor revision

4 = meaning is clear.

Funding Statement

Nil

References

  • 1.Campos JH. Cohen’s Comprehensive Thoracic Anesthesia. Elsevier; 2022. Separation of the lung: Double-lumen endotracheal tubes and endobronchial blockers; pp. 213–39. [Google Scholar]
  • 2.Latif RK, VanHorne EM, Kandadai SK, Bautista AF, Neamtu A, Wadhwa A, et al. Teaching basic lung isolation skills on human anatomy simulator: Attainment and retention of lung isolation skills. BMC Anesthesiol. 2016;16:7. doi: 10.1186/s12871-015-0169-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Maurya I, Ahmed SM, Garg R. Simulation in airway management teaching and training. Indian J Anaesth. 2024;68:52–7. doi: 10.4103/ija.ija_1234_23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Dongare PA, Bhaskar SB, Harsoor SS, Kalaivani M, Garg R, Sudheesh K, et al. Development and validation of a questionnaire for a survey on perioperative fasting practices in India. Indian J Anaesth. 2019;63:394–9. doi: 10.4103/ija.IJA_118_19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Yaghmaie F. Content validity and its estimation. J Med Edu. 2003;3:e105015. [Google Scholar]
  • 6.Polit DF, Beck CT. The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006;29:489–97. doi: 10.1002/nur.20147. [DOI] [PubMed] [Google Scholar]
  • 7.Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30:459–67. doi: 10.1002/nur.20199. [DOI] [PubMed] [Google Scholar]
  • 8.Cicchetti DV, Fleiss JL. Comparison of the null distributions of weighted kappa and the C ordinal statistic. Appl Psychol Meas. 1977;1:195–201. [Google Scholar]
  • 9.Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res. 2003;25:508–18. doi: 10.1177/0193945903252998. [DOI] [PubMed] [Google Scholar]
  • 10.Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35:382–5. [PubMed] [Google Scholar]
  • 11.Davis LL. Instrument review: Getting the most from a panel of experts. Appl Nurs Res. 1992;5:194–7. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Nil.

Articles from Saudi Journal of Anaesthesia are provided here courtesy of Wolters Kluwer -- Medknow Publications

RESOURCES