Reliability and validity of a satisfaction questionnaire on virtual education in the coronavirus disease 2019 pandemic era aimed at cardiology faculty members

Abstract

BACKGROUND:

In the coronavirus disease 2019 pandemic era, clinical programs and mandatory hands-on activities have been supplanted by remote teaching to maintain the fundamental capabilities of medical training and to furnish medical students with quality education. Nonetheless, the satisfaction of faculty members with this training method in the current pandemic has yet to be assessed. The aim of this study was to design a Persian questionnaire with appropriate validity and reliability on cardiology professors’ satisfaction level with virtual education.

MATERIALS AND METHODS:

In this cross-sectional study, a questionnaire was devised drawing upon scientific sources and Iranian medical educators’ expertise. Seventeen faculty members in various specialties evaluated the questionnaire concerning face and content validity. Content validity was assessed through the calculation of the content validity ratio (CVR) (values >0.62 were considered acceptable) and the content validity index (CVI) (values >0.79 were considered acceptable), construct validity was evaluated through principal component factor analysis by the Kaiser–Meyer–Olkin (KMO) statistic and Bartlett's sphericity test, internal reliability was measured through the calculation of Cronbach's alpha coefficient, and consistency was appraised through the use of test-retest reliability at two different time points.

RESULTS:

The questionnaire had a reliability rate of 95%, indicating high internal validity. Concerning test-retest reliability, the intraclass correlation coefficient was 0.96 (P < 0.001), demonstrating relatively good stability. The CVI was 0.81, and the CVR was 0.85. The KMO measure of sampling adequacy was 0.954, indicating the acceptability of the degree of common variance among the all items.

CONCLUSIONS:

This Persian questionnaire on virtual education aimed at cardiology faculty members in the current pandemic with its low question count and appropriate domains had high reliability and validity. By knowing the level of professors’ satisfaction with the new method of education, it is possible to take steps to better provide specialized medical education to cardiology residents.

Introduction

In response to the coronavirus disease 2019 (COVID-19) pandemic, medical faculty members the world over are seeking to facilitate the transmission of knowledge and skills to the next generation of health professionals in the face of social distancing and other health protocol rules, which have proven to be strong disruptors of standard practices.[1] Indeed, since the outset of the current pandemic, medical educators have found themselves struggling against a range of issues, from course interruptions and teaching hours/days alterations to extra hours of online work for courses and webinars.[2]

Not only is the satisfaction of teaching closely related to faculty members’ retention but it also contributes to overall education cohesion and enhanced status of the teaching profession.[3] The factors contributing to faculty members’ satisfaction can be categorized into three groups: (a) Student-related, (b) instructor-related, and (c) institution-related.[4]

Nevertheless, what seems to have hampered medical faculty members’ endeavors to enhance e-learning programs is their relative reluctance in adopting new educational technologies, in tandem with a paucity of free time and the unfeasibility of the practical evaluation of their charges.[5] The situation is compounded in the case of cardiology residents’ syllabus, which demands essential practical skills such as electrophysiology studies, angiography, and echocardiography at patients’ bedside and in specialized laboratories.

Professors’ satisfaction and outcomes are the good indicators for assessing the quality and effectiveness of online programs. It is of concern for institutions to know whether its teachers, in general, are satisfied with their teaching experience.[4] Satisfaction and usefulness of virtual education from the perspective of faculty members can be depend on individual's character or may be related to the facilities such as the availability and accessibility of such online resources, educational methods, and the practical skills that they should teach.[6,7,8] One of the most important elements for quality online education is learner engagement to the learning process, to acquire knowledge and build his or her critical thinking.[9]

Faculty members and clinical educators are the key elements in the successful implementation of virtual education; still, no research has hitherto assessed their satisfaction level with e-learning in the COVID-19 pandemic era.

Satisfaction level is measured through two methods of conducting interviews and using a self-administration questionnaire. The first method, the interview, cannot be used in various studies due to its time-consuming nature. Therefore, in most studies, the second method, the self-administration questionnaire, is used to measure satisfaction.[10] It was in this context that we sought to design a Persian questionnaire with appropriate validity and reliability with a view to determining the satisfaction levels of cardiology professors with virtual education in the period of the coronavirus outbreak.

Materials and Methods

Study design and setting

This cross-sectional study was conducted in July of 2020 to January of 2021. Based on scientific sources including literature review and various Iranian faculty members’ opinion, a Persian questionnaire was devised to determine the satisfaction level of cardiology professors in Rajaie Cardiovascular Medical and Research Center with virtual education in the COVID-19 pandemic era.

Data collection tool and technique

First, the questionnaire questions were written based on the objectives of the study. These questions were a combination of a variety of questions including multiple choices, double choice and marking and also did not use negative and double negative that were clear and transparent. After determining the content of the questions, questions put into a meaningful order and format. This questionnaire was designed in three parts. The first part consists of 22 items grading on a five-points Likert scale (”totally agree = 5,” “agree = 4,” “ no difference = 3” “disagree = 2” and “totally disagree = 1) that all questions are of equal value; consequently, the sum of the scores can be from 5 to 25 for each question. The purpose is to evaluate the level of satisfaction that faculty members feel regarding different aspects of teaching such as content, adequacy of the teaching hours, interaction between professors and residents, Internet facilities, willingness to teach practical skills such as angiography and other imaging techniques, readiness to do research activities, and eagerness to participate in various webinars on cardiovascular diseases, COVID-19, or other fields. The second part posed only 1 question, demanding respondents to choose between 2 options: “Face-to-face learning” and “virtual learning.” The third part featured a visual scale for respondents to specify their level of satisfaction with the virtual education course on a continuum from 0 (minimum satisfaction) to 10 (maximum satisfaction). Demographic questions, including age, education level, academic degree, and specialty, were asked at the beginning of the questionnaire.

Study participants and sampling

First the content validity and then the face validity of the questionnaire were evaluated by 17 faculty members in such various specialties as cardiology, pharmacology, and urology from several medical schools from across Iran. For content validity, development (domain identification and tool construction), Judgment – quantification (determining the agreement among experts, determining the relevancy of questions, and comprehensiveness of the tool as a whole) were assessed. The experts’ opinions vis-à-vis the questionnaire's structure, grammar, relevance, clarity, and simplicity were brought to bear on each item before an expert panel, comprised of 10 professors, assessed each question on a 3-part “essential” scale: “Essential;” “useful, but not essential;” and “not necessary.” The answers served as a basis for a quantitative evaluation of the content validity ratio (CVR) and the content validity index (CVI). CVR values above 0.62 were accepted based on Lawshe's table,[11] and CVI values exceeding 0.79 were considered to denote acceptable relevance, clarity, and simplicity for each questionnaire item.[12,13,14]

Internal reliability was measured by determining internal uniformity, expressed as Cronbach's alpha coefficient. Reliability was considered to be good if the alpha value was not <0.7.[15] Test-retest reliability was drawn upon to assess consistency at two different time points: at baseline and after 10 days.

Principal component factor analysis, a statistical method for testing the construct validity of a scale, was applied to determine the domains of the designed questionnaire in the form of multiple-choice questions. For this purpose, a factor analysis using a Varimax rotation was conducted, and the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was employed. Further, interpretation optimization was ensured through the use of a Promax rotation.

Ethical consideration

Ethics Committee of Rajaie Cardiovascular Medical and Research Center approved the study (Ethical code number: IR. RHC.REC.1399.104). This study was approved by our local ethical committee according to the Helsinki Declaration of the World Medical Association.

Statistical analysis

With respect to internal consistency, Cronbach's alpha coefficient for the entire questionnaire was 0.78 (0.73–0.77). Inter-item correlations were tested through Pearson's correlation coefficient. The intraclass correlation coefficient (ICC) was employed to assess inter-rater consistency, with an ICC of > 0.75 considered excellent, between 0.40 and 0.75 moderate, and < 0.40 poor.[16]

Principal component factor analysis to investigate the construct validity was carried out via 2 tests: the KMO test and Bartlett's sphericity test. The former is a statistical measure that denotes the proportion of variance among variables likely generated by underlying factors, with values close to 1 generally taken to indicate the possibility of the usefulness of a factor analysis with the data and values <0.50 taken to indicate the inadequacy of the factor analysis results. The latter compares the observed correlation matrix with the identity matrix. Overall, values >0.50 in the KMO measure of sampling adequacy and P < 0.05 in Bartlett's sphericity test are considered acceptable.[17,18] The significance level was set at a P < 0.05.

Results

All full-time faculty members (143 people) participated in the current study. Concerning the reliability of the questionnaire, Cronbach's alpha coefficient was 95%, indicating high internal validity. The principal component factor analysis demonstrated a strong 5-factor domain, indicating face validity and high internal consistency and reliability. The results are depicted in Table 1. The first domain was comprised of five items on the quality and content of virtual education, the second domain consisted of four items on adequacy of e-learning facilities as one of the basic requirements, the third domain comprised five questions on the practicality of virtual teaching to cardiology residents, the fourth domain contained two items on willingness to do research activities, and the fifth domain featured six items on eagerness to attend various webinars during the COVID-19 pandemic. As regards test-retest reliability, the retest of inter-item correlations yielded a correlation coefficient of >0.70 throughout the 5-factor domain (P < 0.01) and an ICC of 0.96 (P < 0.001), demonstrating relatively good stability. Apropos of content validity, the results demonstrated an acceptable CVI of 0.81 and an acceptable CVR of 0.85.[7]

Table 1.

Factor analysis on the cardiology faculty members satisfaction questionnaire from virtual education during the coronavirus disease-2019 pandemic

Question number	Description	Rotated factor loadings matrix
		Factor 1	Factor 2	Factor 3	Factor 4	Factor 5
4	Possibility of the accurate evaluation of residents	0.73
5	Good quality of the virtual education course	0.72
2	Suitability of the days of the week for virtual lessons	0.69
3	Suitability of the hours of the day for virtual lessons	0.61
8	Fulfillment of expectations regarding virtual education	0.70
1	Possibility of communication with residents		0.81
20	Ways to disseminate information about virtual education classes		0.83
6	Ease of Internet access		0.71
7	Possibility to control the class		0.69
9	Willingness to teach practical skills such as angiography			0.78
10	Willingness to teach imaging techniques such as echocardiography			0.85
11	Desirability of education in clinics			0.79
12	Willingness to do daily rounds with residents			0.89
13	Satisfaction with having morning reports virtually at mid-day			0.72
14	Willingness to conduct research activities “during the COVID-19 outbreak”				0.77
15	Willingness to conduct research in the field of “COVID-19”				0.78
16	Eagerness to attend webinars on cardiovascular topics					0.75
17	Willingness to give lectures in webinars on cardiovascular topics					0.67
18	Willingness to attend webinars on “COVID-19” topics					0.70
19	Readiness to give lectures in webinars on “COVID-19” topics					0.87
21	Level of acceptance to participate in scientific webinars on cardiovascular topics					0.88
22	Level of acceptance to participate in scientific webinars on “COVID-19” topics					0.75

COVID-19=Coronavirus disease-2019

The 22 Likert-type items from the questionnaire were submitted to principal component factor analysis. The value of Bartlett's test of sphericity was significant (χ²= 476.52, df = 231; P < 0.001). The KMO (Eigenvalue = 1.0) measure of sampling adequacy was 0.954, indicating the acceptability of the degree of common variance among the 22 items. The rotation method accounted for 61% of the total variance for the 5-factor domain.

Discussion

The COVID-19 pandemic, as a global public health emergency, continues to generate tremendous challenges in the delivery of medical education, particularly in the clinical setting, in as much as it cannot be optimally substituted remotely.[19,20,21]

The purpose of the current study was to devise a valid and reliable questionnaire for the evaluation of cardiology faculty members’ satisfaction level with virtual learning in the COVID-19 pandemic era.

The current pandemic has prompted all medical faculty members to find and optimize ways and means to steer their specialty forward. Needless to say, the proper education and appropriate training of residents of various specialties require satisfaction on the part of both learner and educator with the available teaching methods. The COVID-19 pandemic has resulted in a rapid shift in medical education, from the conventional in-person method to virtual education, hence the significance of a valid and reliable tool to measure medical faculty members’ satisfaction level with this sudden transition.

A questionnaire is one of the most widely used instruments to collect relevant information in the most accurate and consistent manner, measured in terms of validity and reliability, correspondingly.[20] Validity refers to a questionnaire's amount of systematic or built-in error, and reliability denotes the replicability degree of the results acquired via a measurement tool and procedure.[22]

The results from the current study demonstrated that the questionnaire enjoyed relatively high reliability in the determination of the satisfaction level of respondents. With regard to validity, the results showed acceptable structural validity in accordance with hypothetical correlations. In addition, the results of the principal component factor analysis showed a strong 5-factor domain, denoting face validity and high internal consistency and reliability.

The novelty of this study is the design of a questionnaire in Persian for the first time during the COVID-19 pandemic. The short length and flexibility of our questionnaire are its strong points in the assessment of cardiology faculty members’ satisfaction levels with e-learning via items on attitudes toward factors implicated in virtual training such as content, teaching hours, professor-resident interactions, Internet facilities, willingness to teach such practical skills as imaging techniques, readiness to conduct research, and eagerness to participate in webinars.

Limitation and recommendation

The role of teaching satisfaction as motivating behavior was not evaluated by this questionnaire. There are a number of gaps in our knowledge around faculty member satisfaction of teaching in a critical situation like a pandemic that follow from our findings and would benefit from further research, including realist evaluation to extend and further study to design new questionnaire in order to evaluate the relationship between teaching motivation and learning satisfaction and also factors related to teaching satisfaction in details.

Conclusions

The reliability and validity of newly-designed Persian questionnaire for the evaluation of the satisfaction level of cardiology faculty members with virtual learning in the COVID-19 pandemic era were confirmed. Given the advantages of this questionnaire such as its small number of questions, appropriate domain, and high reliability and validity, we recommend its use with the aim of promoting medical e-learning during the COVID-19 pandemic. Further studies in order to promote virtual education in pandemic era could be beneficial.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.