Validity and reliability of activities coaching context questionnaire

Abstract

Background: Learning in a clinical environment is an inseparable part of a training program in medical education. To evaluate the quality of training in a clinical environment, a comprehensive questionnaire which is adjusted for local purposes is essential. This study was conducted to determine the validity and reliability of Activities Coaching Context (ACC)-questionnaire from the perspective of residents and students in a clinical learning environment.

Methods: This was a cross-sectional study conducted with 65 residents and students of Semnan University of Medical Sciences. The Content Validity of the questionnaire was confirmed by the judgment of a panel of ten experts in medical education. The construct validity of the questionnaire was determined by factor analysis. Data were collected and analyzed by SPSS₁₆.

Results: The ratio and index of content validity calculated by experts’ view were quite acceptable (0.31 and 0.7, respectively). Construct validity was examined by factor analysis which confirmed seven first order factors. Cronbach’s Alfa coefficient revealed a high degree of internal consistency (0.932). The reliability of the questionnaire was measured by test - retest (0.9).

Conclusions: The ACC questionnaire appeared to be a valid, reliable and also appropriate tool to evaluate the educational quality of a clinical learning environment.

Introduction

Training expert manpower, extending and promoting knowledge, developing research and paving the ground for the development of a country are the main elements included in the mission of each university endeavoring to have its mission completed. In this regard, universities should be able to identify problems, provide a plan and administer this plan appropriately. In the same line, medical education as a part of higher education plays an important role in protecting the public health, thereby showing special concerns about the qualitative and quantitative aspects of medical education (1,2). Evaluation of learning environment with the purpose of quality promotion has always been emphasized by curriculum designers; however, a tool is needed to provide comprehensive information about different aspects of clinical leaning environment having complex and multi-dimensional characteristics (3). There are a number of questionnaires designed for this purpose. Surgical Theatre Educational Environmental Measure (STEEM) (4), Anesthetic Theatre Educational Environment Measure (ATEEM) (5), postgraduate hospital educational environment measure (PHEEM) (6), Dundee Ready Education Environment Measure (DREEM) (7) and Activities Coaching Context (ACC) are some of them.

Among these questionnaires, ACC developed by Dectler et al (2011) is related to clinical training activities. It consists of 44 questions in seven sections including “Clinical Learning Experiences‏" (CLE), “Self – Directed Learning (SDL)‏", “Coaching and Teaching‏" (CT), “Welcoming Process‏" (WP), “Work organization ‏" (WO) , “Multi Profession Collaboration (MPC)‏" and Climate (C). In addition to these sections, it holds three sections related to instruction: “Coaching and Teaching”, “Activity” and “Cooperative Activity”. Although ACC seems to be an appropriate tool for quantitative and qualitative evaluation of a clinical learning environment, its psychometric characteristics should be confirmed before applying it. The present study is unique in this regard as no research study has yet evaluated these characteristic neither in Iran nor in other countries (8).

This study was conducted to determine the psychometric characteristics of ACC questionnaire and to present a standard tool to evaluate the clinical learning environment in Semnan University of Medical Sciences from the perspective of residents and interns.

Methods

This was a cross-sectional study in which the data was collected via census. The participants of the study were all the obstetric, internal and pediatric residents (N= 42), and obstetric, pediatric and surgery clerkships (N= 34) working in the main clinical wards of teaching and medical centers in Semnan University of Medical Sciences.

Data was collected by ACC questionnaire developed by Dectlar (2011) whose permission for using his questionnaire in the setting of Iran was acquired through an email. . The reliability and validity of this questionnaire were initially determined by using it in 150 wards of 160 affiliated hospitals in Louver University in Belgium. The questionnaire was examined with two groups of medical students who were in the year 6 of the program (N1= 531, N2= 402). The purpose of developing this instrument was to explicate the quality of learning in clinical wards, as a learning environment, by using a systematic and unique method.

For the purpose of the present study, ACC questionnaire was first translated from English into Persian through a standard back translation procedure. The questionnaire was then reviewed for clarity and simplicity and no major differences were found in the meaning of the concepts. In the next step, the Persian version of the questionnaire was sent to 12 medical educators to determine Content Validity Ratio (CVR) and Content Validity Index (CVI). Two options of “suitable” and “not suitable” were given to have the reviewers’ comments. After collecting the comments, no item was omitted, and there were some minor changes in the items to localize the questionnaire for clinical setting of Iran as follows:

“The survey of some pathology cases was on my duty.”

“My work program enables me to do what I expect to do .e.g; I spend one day in the lab, radiology, central pharmacy and in the ambulance.”

"Seminars held, was at a high scientific level, they were understandable, rich and full of encouragement.″

“The supervisor informed me, when an interesting event was about to happen in the field of pathology and clinical examination.”

″The supervisor gave me a useful feedback for my files."

"At the end of the period, I had a feedback dialogue with the supervisor".

"As the hospital had defined tasks, I knew what I was supposed to do".

″There was a balance between consultation, hardworking, working in operation theatre (the necessary times), delivery room and management in the weekly program ".

"There was always a person to answer my questions or resolve my problems. ″

'During this period, I encountered various syndromes and I had enough time to be familiarized with the symptoms of diseases".

"I had enough reference studies."

“I feel, I was well acknowledged due to my activities in the clinical ward ".

Consistency of the content and face validity of the questionnaire were confirmed by five experts and a test – retest method, with an interval of 10 days, was performed to examine the reliability of the instrument. The questionnaire was piloted with 10 students who were randomly selected to determine the internal consistency of the questionnaire. The Kolmogorov–Smirnov was used to test the normality of the data. The collected data were analyzed by SPSS version 16 using descriptive statistics presented by central statistics index (mean, variance, standard deviation, and graph).

The ACC questionnaire consisted of two parts. The first part was for collecting demographic information (age, sex, educational level, clinical ward). The second part consisted of 44 items and seven areas related to the followings‏:

• Clinical Learning Experiences (7 items)

• Self-directed Learning (7)

• Coaching and Teaching (16)

• Welcoming Procedure (3)

• Work Organization (3)

• Multi Professional Collaboration (2)

• Climate (6)

Each item was ranked on the basis of a 5-point Likert scale of “totally agree” (4 points), “agree” (3points), “no comments” (2 points), “disagree” (1point) and “totally disagree” (0 point). The maximum score in the questionnaire was 176. To have a better and clearer description, the scores were categorized into four categories: unfavorable (0-44), semi-favorable (45-88), favorable (89-132) and very favorable (133-176).

Statistical analysis

Data analysis was performed using SPSS software version 16. The required statistical methods include confirmatory factor analysis methods and Pearson correlation tests and Cronbach's alpha coefficient determination

Results

From 76 respondents, 65 (including 32 residentsand 33 clerkships) completed the questionnaire (85.5%). The respondents were both females (72%) and males (28%). The results of the study showed the CVI of 0.7 and the mean of 0.31 for CVR. These findings is most probably due to the high mean of experts’ judgment (1.5 for each question) leading to the acceptance of all questions in the ACC questionnaire.

The results of test-retest and Pearson Correlation Coefficient which was done to determine the reliability of ACC, showed that all items were strongly consistent (r =0.9, p=0.02) indicating a highly reliable questionnaire (Table 1).

Table 1 . Test-retest correlation coefficient related to seven - scale domain in the questionnaire .

Scale	Pearson ′s correlation co-efficient	No.	p
Clinical learning experiences	0/95	10	0/012
Self-directed learning	0/86	10	0/023
Coaching teaching	0/79	10	0/021
Welcoming procedure	0/88	10	0/001
Work organization	0/92	10	0/04
Multi-professional collaboration	0/9	10	0/03
Climate	0/93	10	0/04
Total score	0/9	10	0/02

Internal consistency of the items was determined by Cronbach Alpha (0.932) (Table 2).

Table 2 . Cronbach’s Alpha coefficient related to the seven-scale domains of the ACC Q .

Items	Alpha Cronbach co-efficient standardized	Number of questions in each area
Total items	0/932	44
Clinical learning experiences	0/791	7
Self-directed learning	0/734	7
Coaching teaching	0/908	16
Welcoming procedure	0/757	3
Work organization	0/415	3
Multi-professional collaboration	0/714	2
Climate	0/809	6

Construct validity was examined via factor analysis. Bartlett Test with an index of 2003.958 showed that the sample size was sufficient for factor analysis (df=949, p<0.001) (Table 3).

Table 3 . Test-retest correlation coefficient related to seven - scale domain in the questionnaire .

Kaiser-Meyer-Olkin Measure of Sampling Adequacy.	.376
Bartlett's Test of Approx. Chi-Square	2003.958
Sphericity df	946
p	<0.001

The Scree graph showed a rapid decrease from seven factors reaching to the zero variance. Each factor with the Eigen value of more than one was known as a factor or scale. Using this method, seven factors with the greatest role in describing the variance were then extracted (Fig. 1).

Graph1 .

Eigenvalues ​​of the questionnaire

The result of the confirmatory factor analysis showed the appropriateness of the exploration model. Seven factors were determined by values of more than 1.6 and the variance of 57.5. Rotation was done by Varimax and Normalization methods. The items of ACC questionnaire were supported by the seven – order factors pertaining to Clinical Learning Experiences (CLE), Self-Directed Learning (SDL), Welcoming Procedure (WP), Work Organization (WO), Multi-Professional Collaboration (MPC) and Climate (C) (Table 4).

Table 4 . Results loadings on Factor Analysis .

Questions	Matrix elements of the spin
		Factors
	1	2	3	4	5	6	7
CLE1	.66/.72
.72/80	CLE2
CLE3	.66/67
.75/74	CLE4
CLE5	.88/.88
CLE6	.70/.72
CLE7	.61/.63
SDL1		.45/.57
SDL2		.45/.56
SDL3		.58/.55
SDL4		.64/.50
SDL5		.78/.63
SDL6		.96/.57
SDL7		.79/.73
C&T1			.78/.67
C&T2			.76/.70
C&T3			.76/.80
C&T4			.76/.77
C&T5			.77/.83
C&T6			.72/.82
C&T7			.62./60
C&T8			.66/.66
C&T9			.72/.55
C&T10			.67/.70
C&T11			71/.76
C&T12			.49/.51
C&T13			.53/.55
C&T14			.56/.62
C&T15			53/.53
C&T16			.61/.63
WP1				.56/.59
WP2				.79/.79
WP3				.77/.79
WO1					.69/.68
WO2					.66/.63
WO3					.42/.47
MPC1						.71/.77
MPC2						.80/.88
C1							.73/.74
C2							.87/.90
C3							.85/.85
C4							.87/.85
C5							.82/.82
C6							.87/.90

CLE: clinical Learning / SDL: Self-Directed Learning /C&T: Coaching and Teaching / WP: Welcoming Procedure /WO: Work Organization /MPCS: Multi-Professional Collaboration

The results of confirmatory factor analysis (Table 4) and the correlation coefficient showed that some of the items belong to other domains. However, moving those items was not conceptually correct and they remained in the original place.

Discussion

The primary purpose of this study was to determine the reliability and validity of ACC among residents and clerkships in Semnan University of Medical Sciences. The results of this study revealed that the questionnaire was valid and reliable.

To our knowledge, reliability and validity of this tool have not been examined yet in Iran and other countries; therefore, data collected through this study was not comparable with similar studies.

In evaluating the quality of the clinical educational setting, the mean difference between males’ and females’ view

Regarding clinical learning experiences, self – learning and the role of supervisor revealed a significant statistical difference between the males’ and females’ view in 2 scales of the factors including Activity and Role of Supervisor. These differences might be due to the higher population of females and their intellectual and motivating states of mind in comparison with males. In the first factor scale, there was a statistical significant difference between peoples’ view about the social environment, clinical learning experiences, and work organization at pediatric, gynecological and maternity, internal and surgical wards. There was also a significant difference between the second factor scale in peoples’ view in pediatric, gynecological and maternity, internal and surgical wards this seems to be due to the favorable condition of internal wards in comparison with other wards. The high coefficient of occupied beds of over 75% could be due to the efficiency of management of internal wards by capable specialists including respiratory, urology, digestive and liver, blood and oncology, and rheumatology fields. Other factors such as physical condition of patients, food served free of charge, suitable work schedule and provision of rest breaks at work might be responsible for the better performance of internal ward.

Conclusion

The Persian version of ACC for evaluating the quality of clinical learning environment appeared to be a valid and reliable instrument in Iran and can also be used to evaluate the learning environments of other fields of medical education. It is suggested to use this instrument and to repeat factor analysis with a larger population to develop and remove the trivial problems which may be caused in other settings. A qualitative study in major clinical wards is suggested to extract in depth data for promoting the performance of educational environment.

Cite this article as: Koohpayehzadeh J, Haji Ahmadi M, Dehnad A, Soltani Arabshahi S.K, Bigdeli Sh, Yadollahi S. Validity and Reliability of Activities Coaching Context Questionnaire. Med J Islam Repub Iran 2014 (7 June). Vol. 28:41.