Validity and reliability of a virtual education satisfaction questionnaire from the perspective of cardiology residents during the COVID-19 pandemic

Abstract

BACKGROUND:

The coronavirus disease 2019 (COVID-19) pandemic has prompted the further virtualization of medical education. The satisfaction level of specific users such as cardiology residents with virtual education can augment its quality; hence, the significance of a valid and reliable questionnaire to obtain feedback is needed. This study aimed to design and measure validity and reliability of a satisfaction questionnaire for virtual education of cardiology residents during COVID-19 pandemic.

MATERIALS AND METHODS:

In this cross-sectional study, a self-administered questionnaire was developed by the faculty members of Rajaie Cardiovascular Medical and Research Center. Reliability was tested utilizing Cronbach's alpha and intercorrelation which was tested using Pearson's correlation coefficient test (ICC). Factor analysis was done by the Kaiser–Meyer–Olkin measure of sampling adequacy and Bartlett's sphericity test. The statistical analyses were performed with the SPSS software version 22.

RESULTS:

The face validity index was determined via an assessment of the relevance, clarity, and simplicity of each item, and values >0.79 were accepted. The total Cronbach's alpha coefficient was calculated 0.93. Concerning test–retest reliability, the correlation between two rounds of evaluation was >80 (P > 0.001) and ICC was 0.99 (P = 0.001). The content validity evaluation yielded an index of 0.95 and a ratio of 0.91. The principal component factor analysis, conducted to investigate construct validity, generated four domains.

CONCLUSIONS:

The study results confirmed the validity and reliability of the designed questionnaire to evaluate the level of satisfaction of cardiology residents with virtual learning in COVID-19 pandemic.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has significantly impacted educational systems the world over. Medical training with medical educationists now, faced with the unenviable task of altering hospital-based education given the contagious circumstances where clinical mentors and residents work.[1,2]

Virtual learning has proven itself to be the solution of choice during the current pandemic; nonetheless, from the perspective of medical training, reservations have been raised vis-à -vis familiarity with technology, face-to-face interactions, and control.[3] Indeed, unless virtual learning takes heed of the attitude of its users, it cannot be evolved. Specialized medical educational programs must always reflect the ever-changing and ever-increasing demands of society, at large, and the medical community, in particular.[4] Indubitably, fulfilling such needs in the COVID-19 era, in which the hitherto practiced methods of medical training—for instance, as regards angiography and echocardiography—have been rendered well-nigh unfeasible, requires the optimal use of virtual learning.[4]

Beside several benefits that virtual learning offers to educational system, some detriment points make immediate concerns among medical students, residents, and their attends such as impossibility of face-to-face interaction between the professor and the student, less control of the professors over the learners, and unfamiliarity of some professors with the online education system.[3]

The development of virtual learning will not be successful without considering the attitude of users toward this system. Success in virtual learning in medical fields depends, to a large extent, on the satisfaction of residents and professors and their willingness to use it. The quality of specialized medical education programs has not changed in line with the changing needs of society, changing the face of disease, and changing people's expectations of the medical community and healthcare systems.[4]

The idea that virtual education was a new learning domain is crucial. It means that virtual education is a domain in its own right, and it can be combined in a course with both face-to-face and distance learning.[5] Motivation and learning behavior are two very important factors in determining students' learning achievement. Perceived usefulness, management support, self-efficacy, technical support, and training are some factors of learning satisfaction.[6]

Aspects of learner motivation and learning behavior constitute a key factor in the achievement of competency standards in educational process. Final process of motivation is completing an action that can provide satisfaction.[7] Several studies show that both intrinsic and extrinsic motivation serves to reinforce attitudes toward behavior, or motivation can make a connection between attitudes and behavior.[8,9] Collected data from a valid and reliable questionnaire could improve quality of virtual education via promoting residents' motivation.

Research has shown that a systematic perspective toward receiving feedback from medical residents can promote higher quality standards of specialized education.[5] Given the exigencies of the COVID-19 pandemic, it seems advisable that a valid and reliable questionnaire be devised to obtain medical residents' feedback on virtual education. To that end, we endeavored to develop and validate a questionnaire on the level of satisfaction of cardiology residents with virtual learning in the COVID-19 pandemic.

Materials and Methods

Study design and population

This cross-sectional study evaluated the validity and reliability of a satisfaction questionnaire for cardiology residents regarding virtual education during the COVID-19 pandemic in a tertiary teaching hospital in the Iranian capital, Tehran, in 2020.

Data collection

A self-administered questionnaire was devised by the faculty members of Rajaie Cardiovascular Medical and Research Center on the basis of scientific resources and expert opinion. The instrument consisted of three parts. The first part comprised 23 questions about various virtual training-related factors such as the quality and content of virtual education, adequacy of teaching hours, interactions between professors and residents, interactions between residents themselves, internet facilities, willingness to learn practical skills such as angiography and echocardiography, and eagerness to conduct research. This part of the questionnaire evaluated the above areas through the format of a five-point Likert scale encompassing “strongly disagree,” “disagree,” “neutral,” “agree,” and “strongly agree” to allow respondents to express how much they agreed or disagreed with a particular statement. Thus, the scores obtained in this section ranged between 23 and 115. The second part consisted of four questions each demanding a choice between two options: “face-to-face learning” and “virtual learning.” The third part comprised two visual scales to enable respondents to specify their level of satisfaction with both “face-to-face learning” and “virtual learning” on a continuum from zero, denoting minimum satisfaction, to 10, indicating maximum satisfaction.

Content validity was determined using qualitative and quantitative methods. Qualitative content validity was assessed by 14 cardiologists, who commented on the questionnaire's grammar, sentence structure, and placement of phrases in the appropriate place, while quantitative content validity was evaluated using the coefficients of the content validity ratio (CVR) and the content validity index (CVI). For CVR evaluation, the expert group was asked to rate each question on a three-part “essential” scale: “essential;” “useful, but not essential;” and “not necessary.” The acceptable CVR, which depends on the number of specialists commenting on the instrument, was considered 0.49 in the present study.[6,7]

Construct validity was evaluated using principal component factor analysis to determine the domains of the designed questionnaire in the form of multiple-choice questions. For this purpose, a factor analysis using a varimax rotation was conducted, and the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was employed. Interpretation optimization was ensured through the use of a Promax rotation.

External reliability was assessed by five cardiology fellows in different cardiovascular fields. The fellows measured the reproducibility of the answers to the questions at 2 weeks' intervals via the test–retest method; then, the correlations between the results were calculated. Correlations exceeding 0.7 were considered to be reliable features.

Internal consistency varies from zero to 1 which was assessed using Cronbach's alpha. Scales with internal consistency coefficients >0.7 were regarded as acceptable results.

Ethical consideration

The study was approved by the Ethics Committee of Rajaie Cardiovascular Medical and Research Center (Ethical Code Number: IR.RHC.REC.1399.100). The purpose of the study was informed to participants and written informed consent was obtained. Confidentiality was assured by informing that the information recorded was used for research purposes only and that no personal details would be recorded or produced on any documentation related to the study.

Statistical analysis

Reliability was tested for internal consistency utilizing Cronbach's alpha, and inter-item correlation was tested using Pearson's correlation coefficient. In addition, the inter-rater consistency of the raters was determined by applying the intraclass correlation coefficient (ICC), the benchmark for which was as follows: >0.75: excellent; between 0.40 and 0.75: moderate; and <0.40: poor.[8] Internal consistency as regards the interscale correlation was examined using Cronbach's alpha. Cronbach's alpha coefficient for the entire questionnaire was considered to be 0.78 (0.73–0.77). Descriptive analysis was conducted to describe the data. For the purposes of factor analysis to investigate construct validity, two tests were applied: the KMO measure of sampling adequacy and Bartlett's sphericity test. The former is a statistical measure that denotes the proportion of variance among variables likely generated by underlying factors, with values of close to 1 generally taken to indicate the possibility of the usefulness of a factor analysis with the data and values less than 0.50 taken to indicate the inadequacy of the factor analysis results. The latter compares the observed correlation matrix with the identity matrix. Overall, values >0.50 in the KMO measure of sampling adequacy and P < 0.05 in Bartlett's sphericity test are considered acceptable.[9,10] The significance level was set at a P < 0.05. The statistical analyses were performed with the SPSS software, version 22, for Windows (SPSS Inc., Chicago, Illinois, U.S.A).

Results

As a measure of internal consistency, for the first part of the questionnaire (23 questions), Cronbach's alpha coefficient for each question ranged from 0.75 to 0.91 and the total Cronbach's alpha coefficient was 0.93, indicating the goodness of the overall reliability of the instrument. Furthermore, Cronbach's alpha coefficient for reliability was 0.95, denoting the stability of the whole questionnaire. Apropos the test–retest reliability, the correlation between the two evaluation rounds was more than 80 (r > 0.80 and P > 0.001) and ICC was 0.99 (P = 0.001), which demonstrated relatively good stability for the questionnaire.

With respect to the content validity of the questionnaire, the results showed a CVI of 0.95 and a CVR of 0.91. For face validity, the relevance, clarity, and simplicity of each item were evaluated, and values >0.79 were accepted.

The appropriateness of the instrument for evaluating the intended purpose, the difficulty level of the instrument for the target group (cardiology residents), and the adequacy of the questionnaire construction, which constitute the parameters that show the content validity of a questionnaire, were given high scores by the respondents. The cardiology residents in the present study had no difficulty understanding the questions; it can, therefore, be concluded that the content validity of this questionnaire was also high.

Principal component factor analysis for investigating construct validity was carried out to determine the domains of the designed questionnaire in the form of multiple-choice questions. Kaiser's criterion (eigenvalue = 1.0), considered to determine the main factors of the questionnaire, found four domains. The values of Bartlett's sphericity test were statistically significant (χ² = 2393.63, df = 435; P < 0.001). The KMO measure of sampling adequacy yielded a value of 0.921, indicating that the degree of common variance among the 23 items was acceptable. The results of the promax rotation, performed to optimize interpretation, are presented in Table 1. The first domain comprised 10 items on the quality and content of virtual education, the second domain consisted of eight items on hours/days in the week allocated to virtual education and also the facilities required for e-learning, the third domain contained three questions on practical learning for cardiology residents, and the fourth domain comprised two items on willingness to do research activity.

Table 1.

Factor analysis on the validity and reliability of the virtual education satisfaction questionnaire from the perspective of cardiology residents during the COVID-19 pandemic

Items number	Description	Rotated factor loadings matrix
		Factor 1	Factor 2	Factor 3	Factor 4
2	Possibility of communication with the professor	0.69
4	Possibility of communication between residents	0.61
7	Possibility of an accurate evaluation of residents by the professor	0.64
17	Ability to provide appropriate feedback by the professor	0.70
8	Good quality of virtual education	0.82
9	Consistency of the presented content with the course titles	0.75
12	Satisfaction with the method of education	0.88
13	Satisfaction with the scientific quality of education	0.89
3	Fulfillment of the expectations from virtual education	0.76
18	Interest in attending online classes	0.62
1	Ways to disseminate information about virtual education classes		0.81
5	Suitability of the days of the week for virtual lessons		0.74
6	Suitability of the hours of the day for virtual lessons		0.67
10	Ease of internet access		0.67
11	Uploading of links to virtual education classes on the website of the Deputy Minister of Education		0.75
14	Quality of the “online” audiovisual educational content		0.79
15	Quality of the “offline” audiovisual educational content		0.72
16	Rate of the use of the audiovisual teaching aids		0.87
19	Willingness to learn practical skills such as angiography			0.86
20	Willingness to learn practical skills such as echocardiography			0.84
21	Desirability of education in clinics			0.61
22	Willingness to conduct research “during the COVID-19 outbreak”				0.65
23	Eagerness to conduct research in the field of “COVID-19”				0.64

Discussion

This study aimed to design a valid and reliable questionnaire to evaluate the level of satisfaction of cardiology residents with virtual education in the COVID-19 pandemic.

A viable management strategy as regards satisfaction creation in a workforce is to prevent dissatisfaction,[11] which requires the regular measurement of these concepts with the aid of valid and reliable instruments. The fact that the training of medical specialists, as valuable human resources in the healthcare system, is a significant goal pursued in medical education prompted us to design a questionnaire with a view to evaluating satisfaction with virtual education among cardiology residents.

Reliability refers to the degree of the replicability of the results obtained by a measurement instrument. However, its significant contribution to assess validity of a questionnaire notwithstanding, it deemed a sufficient condition for the validity of the questionnaire.[12] The validity of a questionnaire, on the other hand, certifies the ability of its scales to measure the intended concept.[13]

The principal component factor analysis revealed four strong, clinically relevant domains. The first domain covered the quality and content of virtual education for cardiology residents, the second domain enquired about hours/days in the week allocated to virtual education and also the facilities required for e-learning, the third domain investigated the practicality of learning for respondents, and the fourth domain determined residents' eagerness to do research activity. Factor loadings were all within the acceptable range (0.61–0.89), transcending the recommendations of other reference studies (at least 0.4 for factor loadings).[14]

The results from the current study demonstrated that our questionnaire had good reliability and validity. Cronbach's alpha coefficient was greater than 0.9, indicative of good internal uniformity. In other words, the respondents had almost the same perception of the instrument. Moreover, with respect to the reproducibility and test–retest of our questionnaire, the results indicated not only an appropriate level of reliability but also the reproducibility, stability, and internal coherence of the items. Alternatively stated, if the measuring instrument is repeated in the same conditions, similar results will be obtained. Finally, the results also showed that the questionnaire had good validity in this study population.

In summary, five distinctive attributes of virtual education are (1) many-to-many (group communication); (2) any place (place-independence); (3) any time (time-independence); (4) text-based (enhanced by multiple media); and (5) computer-mediated messaging.[15] However, since face-to-face learning also has many benefits especially for cardiovascular residents who need clinical training, awareness of their willingness and satisfaction in changing methods of training after the end of COVID-19 pandemic and even permanent changes in their educational curriculum as a combined virtual and face-to-face method, considering a suitable instrument developed in the same field in various fellowship fields of cardiology seems important. Consequently, this questionnaire as a reliable and valid instrument can be used to assess satisfaction level of cardiology residents in virtual residents. This will help clinical professionals improve virtual educational plan for better training of cardiology residents.

Conclusions

The study results confirmed the validity and reliability of our Persian questionnaire, designed to evaluate the level of satisfaction of cardiology residents with virtual learning in the COVID-19 pandemic. Validated questionnaires that report medical residents' feedback on virtual education, especially in the period of the coronavirus outbreak, can assist educationists in opting for the most efficient medical training methods. We believe that the questionnaire introduced herein can help health policymakers and planners effectively devise comprehensive, yet easily comprehensible, educational programs.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.