Applying structural equation modeling to Canadian Chiropractic Examining Board measures

Abstract

The purpose of this research project was to determine if structural equation modeling (SEM) can be successfully applied to the Canadian Chiropractic Examining Board (CCEB) measures to explore the inferential nature of the “causal” relationship between academic ability and success on the CCEB examinations; specifically the ability to make correct clinical decisions. As this was a time-series study (pre-chiropractic grade-point-average to licensure examination data), a latent variable path analysis was the SEM method of choice. The Comparative Fit Index for the model to data fit was 0.98. Inferences include: 1) the need to recruit students with strong academic abilities, 2) the need to hold back students who have not achieved a high level of understanding of the first two-years of work at chiropractic college, and 3) that the CCEB extended-matching, long-format questions are a better estimate of clinical reasoning ability than 5-option short-format questions or the OSCE.

Introduction

The Canadian Chiropractic Examining Board (CCEB) has been evaluating chiropractors wishing to be licensed/registered in Canada since 1962.¹ There have been many types of examinations offered by the CCEB since that time - from essay to Objective Structured Clinical Examinations (OSCEs).² The examinations currently provided by the CCEB include the written examinations, the clinical skills examination, and a practitioner assessment examination. The written examinations consist of Component A, generally a 5-option, best-answer multiple choice examination covering basic and applied sciences; and Component B, an extended-matching long-format assessment of Clinical Decision Making skills (up to 26 options) along with a five-option, best answer, multiple choice examination of diagnostic imaging skills. Candidate measures collected and assessed by the CCEB include: 1) pre-chiropractic grade-point-average (UGPA), 2) chiropractic grade-point-average while at chiropractic college (χρGPA), 3) standardized candidate measure on the basic science part of Component A, 4) standardized candidate measure on the applied science part of Component A, 5) standardized candidate measure on the clinical decision making part of Component B, 6) standardized candidate measure on the Diagnostic Imaging part of component B, and 7) standardized candidate measure on the Clinical Skills Examination OSCE.

Structural equation modeling (SEM), although only recently reported in the medical education literature, has a long history. Factor analysis was invented almost 100 years ago, path analysis about 75 years ago, and simultaneous equation models about 50 years ago. SEM first appeared about 25 years ago, combining models and methods from econometrics, psychometrics, sociometrics, and multivariate statistics.³ As the mathematical computations are somewhat complex, SEM has become increasingly popular as statistical packages have been modified to run on personal computers rather than mainframes. In contrast to correlational studies and factor analytic studies, SEM is a theory rich approach to multivariate analysis. SEM focuses on hypothesis testing of a structural theory as it applies to some phenomenon. The theory being evaluated generally represents a causal mechanism that generates observations on multiple variables.⁴ SEM uses pictures (the model) to clarify the underlying theory (the equations) that is being studied. Another advantage of SEM is that it can incorporate observed measurements and latent variables in its procedures. A latent variable is simply a construct that cannot be directly measured. As an example, we cannot directly measure clinical competency, thus clinical competency is latent. We do however have evaluations that measure some of the characteristics of clinical competency. SEM allows the researcher to, not only measure correlations between observed measures, but relationships between variables that cannot be directly measured – the latent variables. Kline summarizes the shared characteristics of SEM methods as: 1) SEM is a priori and requires researchers to think in terms of models, 2) SEM allows explicit representation of a distinction between observed and latent variables, 3) The basic statistic in SEM is the covariance, 4) SEM can be applied to correlational data (nonexperimental) and to data from experiments, 5) Many standard statistical procedures like multiple regression, canonical correlation, factor analysis, and ANOVA can be viewed as special cases of SEM, 6) SEM is a large-sample technique, 7) It is possible to test many different types of effects for statistical significance in SEM.⁵

The purpose of this research project was to determine if SEM can be successfully applied to the CCEB measures to explore the inferential nature of the relationship between academic ability and success on the CCEB examinations, specifically the ability to make correct clinical decisions (diagnosis and management). If SEM can be successfully applied, then the analysis should provide some new evidence of relationships between the variables. The primary objective was to determine if a theoretical model would “fit” the observable data. The secondary objective was to review the size of the “causal” effect of the observable and latent variables on the latent variable Clinical Reasoning Ability.

Methods

Subjects

Data were supplied by the Canadian Chiropractic Examining Board (CCEB) and consisted of anonymized candidate data (292 candidates) for its March and June 2004 Clinical Skills Examinations (CSEs). The data received from the CCEB were coded in such a way that it was impossible to track or identify an individual candidate, rater, or SP. As the data were completely anonymized and the research project met the policies of the CCEB for ethical research, approval was granted by the Board of Governors. The data represented candidates from 18 different chiropractic colleges. There were 188 graduates from Canadian institutions and 102 from US institutions and 2 from outside of North America.

Test format

There were six data variables available provided by the CCEB: standardized scores for the Basic Science (BS), Applied Science (AS), Clinical Decision Making (CDM), and Clinical Skills Examinations (an OSCE). The last two variables were UGPA and χρGPA. The UGPAs for all candidates were converted to a University of Calgary 4.0 grade-point-average and consisted of the average of all courses taken at post-secondary institutions in Canada. If candidates retook a course to improve their grade, both grades were included in the average. The χρGPA were not converted as no generally accepted method exists and consisted of the cumulative grade-point-average as reported by the candidate’s college of graduation. The BS and AS examinations consisted of single, best answer 5-option multiple-choice questions (MCQs). The AS examination was clinically based and consisted of clinical vignette questions in short-format. For the AS examination, the clinical vignettes consisted of statements that would lead the candidate to the correct diagnosis. The BS and AS examinations were taken by candidates after they had completed two-years of their chiropractic education. The CDM examination consisted of 26-option extended-matching long-format questions. Long-format questions consist of questions with an equal number of statements that lead candidates to the correct answer, and statements that typically would be received while taking a case history or performing a physical examination, but do not direct the candidate to the right answer. The CDM examination could be taken by candidates who were within 6-months of graduation. The OSCE consisted of ten stations through which all candidates rotated. The stations were administered twice a over multiple tracks, and multiple sites. Candidates could not contaminate other candidates as the morning candidates were not able to leave until the afternoon cycle started. Additional days of testing or new administrations consisted of new cases. Each station had a checklist for computer scoring of the data. Each checklist consisted of a series of 3-point rating scales (0 = not performed, 1 = performed but not correctly, 2 = performed correctly), one 5-point rating scale for professionalism, and one 10-point rating scale for overall technique. Both the professionalism scale and overall technique scale were anchored at the borderline pass and borderline fail levels. Stations 1 and 7 were case history stations, 2 and 8 were physical examination stations, and 5 and 6 were combined history and physical examination stations. At the end of each of these stations, candidates were required to communicate a diagnosis or differential, and a plan of management. Stations 3 and 9 were differential physical examination stations with multiple cases. Station 4 was an informed consent station, and station 10 was a chiropractic treatment station. The OSCE was taken by candidates who had completed all the requirements for graduation.

Statistics

As this was a time-series study (UGPA to licensure examination data), a latent variable path analysis was SEM method of choice. A correlational matrix of all variables was calculated for the development of a theoretical model. The theoretical model proposed that a latent trait called Academic Ability had a causal relationship to another latent trait called Professional Knowledge. Further the model proposed that the latent trait Professional Knowledge had a causal relationship to a third and final latent trait called Clinical Reasoning Ability. The model proposed that students arrive at chiropractic college with their Academic Ability already formed and thus the observable variable UGPA loaded onto Academic Ability. The latent trait Professional Knowledge was theorized to be established by the end of year two. Thus χρGPA, and the two CCEB examinations BS and AS were presumed to load onto the Professional Knowledge latent trait. The latent trait Clinical Reasoning Ability was theorized to be most complete at the end of clinical rounds. The CDM examination and the OSCE loaded onto the latent trait Clinical Reasoning Ability. As the AS also consisted of clinical vignettes, it also loaded onto Clinical Reasoning Ability. The model to data fit was analyzed with EQS version 6.1 (B83), a structural equation program by Multivariate Software Inc. and copyrighted by P.M. Bentler.

For the data to satisfactorily fit the model, comparative fit indices of greater than 0.97 are required.^6–8 Further, the analysis should complete in a small number of iterations (less than 15), and the percentage of residuals between 0.1 and −0.1 should be high.³ For a path coefficient to be meaningful, it should be greater than 0.40.⁵

Results

The correlational matrix with mean and standard deviations for all variables is shown in Table 1.

Table 1.

Bivariate Correlations, Standard Deviations, Means

	UGPA	χρGPA	BS	AS	CDM	OSCE
UGPA	1.00
χρGPA	0.34	1.00
BS	0.47	0.40	1.00
AS	0.46	0.38	0.78	1.00
CDM	0.38	0.31	0.63	0.67	1.00
OSCE	0.36	0.11	0.44	0.43	0.44	1.00
Std Dev	0.51	0.41	85.01	82.45	82.86	113.91
Mean	2.95	3.16	520.62	521.38	518.93	474.97

Footnote: UGPA = undergraduate grade point average, χρGPA = chiropractic college cumulative grade point average, BS = Basic Science, AS = Applied Science, CDM = Clinical Decision Making, OSCE = Objective structured clinical examination, Std. Dev = standard deviation, Mean = average.

The mean values for UGPA and χρGPA were 2.95 and 3.16 respectively. Table 1 reveals that both UGPA and χρGPA correlated highest with the BS examination (0.47 and 0.40 respectively) and lowest with the OSCE (0.36 and 0.11 respectively). UGPA correlated higher then χρGPA on all observable measures. BS correlated highest with AS (0.73) and AS correlated highest with CDM (0.67) than with the OSCE (0.43). As the correlations supported the theoretical model, an SEM model was created and the analysis performed. The grade-point-averages are all on a 4.0 scale. The examination scores are standardized to a theoretical population mean of 500 and a standard deviation of 100. Mean scores and standard deviations are not exactly these values as the two exam represent a sample of the candidates over the 5-year period.

Figure 1 provides the pictorial representation of the model and the path coefficients between the variables and the traits for the SEM analysis.

Figure 1. Three Factor Latent Variable Path Analysis.

Footnote: UGPA = undergraduate grade point average, χρGPA = chiropractic college cumulative grade point average, BS = Basic Science, AS = Applied Science, CDM = Clinical Decision Making, OSCE = Objective structured clinical examination, Std. Dev = standard deviation, Mean = average, Prof Know = professional knowledge (prior to clinical courses)

The Comparative Fit Index (CFI) for the model to data fit was 0.98. The analysis converged in 5 iterations, and ninety percent of the residuals were between −0.10 and +0.10. Figure 1 reveals that all path coefficients were greater than 0.40, except for the path coefficient between the latent variable Clinical Reasoning Ability and the observed variable AS.

Discussion

The stronger correlation of UGPA to the licensure examination measures reflects the lack of a consistent GPA from the 18 chiropractic colleges represented in the data. A CFI of 0.98; convergence in 5 iterations, and large number of residuals between 0.1 and −0.1 indicates that there is an adequate fit of the data to the theoretical model. The path coefficients between the latent variables are all strong. From Academic Ability to Professional Knowledge, the path coefficient is 0.84. From Professional Knowledge to Clinical Reasoning Ability, the path coefficient is 0.88. The strong path coefficients infer that: if chiropractors with strong Clinical Reasoning Ability are desired, then students should not move beyond the second year of their education unless they have demonstrated a substantial grasp of underlying concepts (the latent trait Professional Knowledge). Further, in order for students to be able to achieve a large measure of the latent trait Professional Knowledge, they should enter the chiropractic education process with a large measure of the latent trait Academic Ability. Although this may not be much of a surprise, it does conflict with anecdotal reports from previous CCEB administrators that unsuccessful CCEB candidates, their families, their spouses and their legal counsel, communicate that they would be excellent chiropractors; they just aren’t academically strong. When chiropractic programs are not fully subscribed (programs where there are more classroom seats than qualified applicants), the model further infers that should someone enter a program with less Academic Ability, they should not move through the system (past second year and the establishment of their level of the latent trait Professional Knowledge) unless they have demonstrated a suitable mastery level of their first two years work. Such demonstration could be measured by classroom evaluations or external measures.

The path coefficients from the observed variables reveal that the weakest path coefficient is between AS and Clinical Reasoning Ability (0.24). The CDM variable has a much stronger path coefficient (0.81). This infers that the 5-option short-format clinical vignette questions on the AS examination are not as good a measure of the latent trait Clinical Reasoning Ability as is the 26-option long-format clinical vignettes from the CDM examination. The path coefficient between the OSCE variable and the latent trait Clinical Reasoning Ability is moderate (0.54), but much less than the path coefficient to the CDM examination. This makes sense, as the OSCE is truly not a measure of clinical reasoning, but a measure of the skills of taking a case history, performing a physical examination, and communicating with patients. These path coefficients infer that the Practitioner Assessment Examination (for chiropractors who have ceased to practice in Canada for a period of time) should emphasize questions from the CDM examination and not simply utilize the OSCE.

Conclusions

The latent variable path analysis revealed good model to data fit, and revealed that inferences can be made from the data. These inferences include: the need to recruit students with strong academic abilities, the need to hold back students who have not achieved a high level of understanding of the first two-years of work at chiropractic college, and that lapsed practitioners should not be solely assessed with an OSCE. The relationship between academic ability and clinical reasoning skills was not previously available, and the stronger relationship between clinical reasoning and long-format clinical vignettes was not available. Structural Equation Modeling, in the form of a latent variable path analysis, was therefore successfully applied to data from the Canadian Chiropractic Examining Board.

This study is limited and not generalizable to medicine due to the heterogeneous nature of chiropractic undergraduate grade-point-averages. Students at medical colleges have a much more homogeneous entry level grade-point-average, and the results of this study may not be applicable.⁹ This study may also not be generalizable to single institutions. The heterogeneity of the data was enhanced by combining graduates of Canadian and US institutions with a 60% to 40% split. Students within a single institution may be more homogeneous. In addition, this study is at the lower bound of reasonable sample size. As a general rule, for each observable variable there should be 10 observations (lines of data), and 50 observations for each latent trait. Although this study exceeds this general rule (292 vs 210), the sample size is still small.