Development and Psychometric Evaluation of a Cardiovascular Risk and Disease Management Knowledge Assessment Tool

Abstract

PURPOSE

This paper describes the systematic construction and psychometric analysis of a knowledge assessment instrument for phase II cardiac rehabilitation (CR) patients measuring risk modification disease management knowledge and behavioral outcomes derived from national standards relevant to secondary prevention and management of cardiovascular disease.

METHODS

First, using adult curriculum based on disease specific learning outcomes and competencies, a systematic test item development process was completed by clinical staff. Second, a panel of educational and clinical experts used an iterative process to identify test content domain and arrive at consensus in selecting items meeting criteria. Third, the resulting 31 question instrument the Cardiac Knowledge Assessment Tool (CKAT) was piloted in CR patients to insure utility of application. Validity and reliability analysis were performed on 3,638 adult pre test administrations with additional focused analyses on 1,999 individuals completing both pre and post treatment administrations within 6 months.

RESULTS

Evidence of CKAT content validity was substantiated with 85% agreement among content experts. Evidence of construct validity was demonstrated via factor analysis identifying key underlying factors. Estimates of internal consistency, e.g. Cronbach’s Alpha = .852 and a Spearman-Brown split-half reliability = .817 on pre testing, supports test reliability. Item analysis, using point biserial correlation, measured relationships between performance on single items and total score (p<.01). Analyses utilizing item difficulty and item discrimination indices further verified item stability and validity of the CKAT.

CONCLUSIONS

A knowledge instrument specifically designed for an adult CR population was systematically developed and tested in a large representative patient population, satisfying psychometric parameters including validity and reliability.

Advances in the early recognition and aggressive medical treatment of cardiac disease have led to a sharp decrease in mortality in the predominately older adult population at risk¹. As mortality figures have decreased an increased burden has been placed on the health care system to provide the necessary preventive tools to effect long-term disease self-management in this large patient cohort². These same advances in the early recognition and aggressive medical treatment of cardiac events coupled with the dramatic decrease in time of hospital stay have limited in-hospital patient education. There is also evidence of a fundamental inability of the adult patient and family to comprehend the behaviors required of them to improve long-term health outcomes during an acute period of physical and psychological stress typical of post procedural inpatient hospital care³. During this shortened immediate post recovery period both patients’ and families’ perceived learning needs are directed at effecting survival of the serial event and not secondary preventive strategies⁴. The goal of outpatient cardiac rehabilitation (CR) from its initial inception has been to return the patient to optimal physical and psychosocial functional status post cardiac illness. Contemporary CR programs employ substantial resources teaching risk factor and disease management strategies. To this end education programs are formulated to help patients modify high-risk behaviors and assume personal responsibility for long-term disease management⁵. Results are encouraging; meta-analyses have revealed a 34% reduction in cardiac mortality and 29% reduction in recurrent myocardial infarction in CR settings where these focused psychoeducational programs have shown to be most effective⁶. Many CR programs have developed “home-grown” assessment tools to measure baseline disease and risk knowledge as well as quantify gains. Lacking is a test specifically developed and validated in the target population. Although instruments have been systematically developed to measure general knowledge in a variety of other cardiovascular settings they want for specificity of purpose and application in both pre and post testing CR use^7,8,9. In a more focused CR cliental Smith et al designed a 40 item test with good reliability and validity results¹⁰. However, design flaws included: development in a small, non representative sample of male phase II & III CR participants with & without diagnosed CAD; inclusion of a pilot-group of 38 spouses; and a Flesh Kincaide reading grade level of 11.3 and reading ease of 41.6 (both well outside the recommend levels of 8th grade or lower and readability ease of 75, respectively)^11–13. Also, test items included program specific knowledge including entry requirements, cardiopulmonary resuscitation technique and type A and B behavior analysis, further limiting general application. Missing were objective questions focused on specific, nationally established disease management goals including lipids, blood pressure, exercise, diabetes, diet and CHF management criteria¹⁴. More recently Bergman et al. developed a psychometrically sound test which claimed useful application in “adult populations” however, the test was developed using an urban undergraduate student cohort. A follow-up comparative study by the authors using an older adult cohort, mean age 57yrs., demonstrated a negative association between subject age and test score further calling into question the use of young adults in the original design, validation and interpretation of the instrument. Furthermore, a lack of specific focus on CAD risk modification and disease management knowledge limits utility of this instrument in CR adult clientele¹⁵.

Methods

Project overview

This work’s primary purpose was development and deployment of a valid clinical assessment tool for this specific adult patient population. The design strategy we describe was first validated as a clinical improvement project in a focused representative sample of 50 CR patients from 2000–2001. With local IRB approval, the results of this developmental project were expanded to include an additional 114 test applications which completed pilot analysis in 2002. During the ensuing years 2002–2011 this instrument fulfilled its primary goal of measuring clinically relevant knowledge in our patients thus enabling effective individual and group educational remediation. In January of 2011, with local IRB approval, we retrospectively conducted in-depth analysis to further validate its application and make any necessary psychometric adjustments to its construct.

Design approach

The formative stage of the project was initially influenced by the model developed by Gastafson and Branch (1997) which enumerates four major activities consistent of effective adult instructional development (AID). First, analysis (of the setting and learner needs); Second, design of a set of specifications for an effective, efficient, and relevant learner environment; Third, development of all learner and delivery materials and; Fourth, evaluation¹⁶. Richey (1986) also identified six core elements of AID that also influenced our design approach: (1) Determine learner needs; (2) Determine goals and objectives; (3) Construct assessment procedures; (4) Design/select delivery approaches; (5) Try-out instructional system; (6) Install and maintain system¹⁷.These models form the foundation of the World Wide Instructional Design System (WIDS) whose curriculum and evaluative models mandate a structured process with a primary focus on: Who the learners are (adults 40–85yrs.); What they need to know (heart disease risk and disease management knowledge and skills); When learning objectives are met (learning time lines); and How they’re going to get there (instructional strategies). The process then involves matching learning competencies to achievement criteria in an iterative item development format prior to piloting the test in the target population and performing reliability analyses¹⁸. Our work was purposed to follow this design utilizing a multidisciplinary team of experts including professionals representing nursing, medicine, exercise physiology, dietetics, pharmacy, psychology and adult education… then pilot, and rigorously investigate the designed psychometric constructs over time in our large CR patient population.

Test Construction

Instrument development was initiated in May of 2000 by multidisciplinary members of our program’s CR staff (SCR) using a systematic process suggested by Grant and Davis that involves step-wise content domain identification, item generation and instrument construction^19–22. Developers adhered to a process, pioneered by Bloom (1956) and refined by (Anderson 2010), which recognizes a taxonomy of learning objectives within cognitive and affective learning domains^20–22. Test item development focused on Bloom’s foundation learning objectives of knowledge, comprehension and application, all fundamental evaluative milestones in realizing the ultimate goal of behavioral change. The SCR program’s core curriculum consists of 36 instructional modules (Figure 1) developed according to guidelines established by the American Heart Association and American College of Cardiology for secondary prevention of heart disease and lifetime maintenance of heart healthy lifestyle practices²³. We followed the process that Lynn, 1986 defines content validity as “the determination of the content representativeness or content relevance of the elements/items of an instrument”. The two stage process first involved a structured developmental stage followed by a quantification stage²⁴. During the developmental stage SCR staff, utilizing their cumulative experience with the teaching plan objectives contained in each module were challenged to identify focused curriculum domains specific to desired learner based competencies and outcomes. Ten learning content domains evolved as a result of this process: 1) blood pressure control; 2) lipid management; 3) exercise; 4) heart failure; 5) cardiovascular disease risk; 6)medications; 7) anatomy & physiology; 8) diet; 9) diabetes and 10) stroke (Figure 2). Next, the SCR staff generated teaching plans with specific objectives and curriculum content for the individual teaching sessions contained within the ten identified learning domains. Test items were then systematically developed to measure desired competencies and objectives specific to each key learning domain. A pool of test items was constructed by working from a list of instructional objectives weighted to approximate and prioritize importance within each learning content domain. Care was taken to ensure that teaching content and learning objectives were tested in proportion to the relative importance they received during group instruction and individual remediation. Item relevance was then determined using a content validity index (CVI). Items were individually scored by the eight SCR developers on a 4-point scale of; 1= not relevant, 2=somewhat relevant, 3=quite relevant, 4=highly relevant. Scores above 1 connoted relevance. Item inclusion scores required .85 agreement among the CR raters^{24, 25}. From this item pool developers then weighted learning domain complexity based on a five point Likert scale from easy to difficult. Difficult domains, which required multiple learning objectives and more complex behavioral mastery, were thereby systematically allotted sufficient items to satisfy these needs. (Figure 1) The test structure focused on criteria referenced learning objectives. The instrument item structure consisted of a short phrased question followed by 5 multiple choice responses; one correct response and three incorrect interspersed distractors followed by a 5th “I don’t know” response, also scored incorrect. The “I don’t know” option was chosen due to the criteria based design of the instrument and the importance of clinicians to clearly evaluate baseline and temporal improvements in knowledge by avoiding guessing and test-taking acumen^26–29. Reading comprehension was kept at a basic 8th grade comprehension level via SMOG and Flesh readability analysis ^{30, 31}. This reading level was well within our patient education demographic (mean 13.6 years, sd 2.69) and generally accepted in the adult health literacy literature ^{32, 33}. Test developers avoided the use of jargon and obtuse medical terminology in item design. Test items were also screened for simplicity of sentence construct, redundancy and optimal clarity of interpretation. A panel of 7 expert judges was then recruited and using a table of specification format, these content experts evaluated the relevancy of the test item to the curriculum content and teaching objectives previously identified³⁴ (Table 1). If expert mean agreement on an item was 80% or less, the item was examined for either rewording or deletion. Fourteen items met this criterion, 9 were deleted and 5 were reworded. Test items were selected for inclusion in the final instrument using a cut-off criterion of at least 85% inter-rater (six of the seven) agreement among the judges²⁴ (Table 1). The 31 items chosen for inclusion were a result of this systematic process and compose the Cardiac Knowledge Assessment Tool (CKAT^©) developed to evaluate achievement of the learning objectives contained within this core CR curriculum (Table 2.).

Figure 1. Cardiac Rehab Instructional Modules.

36 instructional modules developed according to guidelines established by the American Heart Association and American College of Cardiology for secondary prevention of heart disease and lifetime maintenance of heart healthy lifestyle practices¹⁶.

Figure 2. Learning Content Domains and Corresponding Number of Test Items.

Ten focused curriculum domains, developed by multidisciplinary staff, and specific to the desired learner based competencies and outcomes suggested by the American Heart Association and American College of Cardiology guidelines¹⁶.

Table 1.

Table of Specification

		Reviewer #1		Reviewer #2		Reviewer #3		Reviewer #4		Reviewer #5		Reviewer #6		Reviewer #7
Discipline		BSN-EP/15yrs		MA-EP/15yrs		BSN/25yrs		MS-EP/10 yrs 10yrrs		BS-RD/16 yrs 16yrs		BSN/15yrs		MD/12yrs
Q#	Status	Domain	Ob/Met	Domain	Ob/Met	Domain	Ob/Met	Domain	Ob/Met	Domain	Ob/Met	Domain	Ob/Met	Domain	Ob/Met
#1	✓	BP	80%√	BP	80%	BP	90%	BP	90%	BP	100%	BP	100%	BP	50%
#2	✓	Chol.	95%	Chol.	80%	Chol.	90%	Chol.	95%	Chol.	85%	Chol.	98%	Chol.	100%
#3	✓	Ex.	95%	Ex.	80%	Ex.	100%	Ex.	95%	Ex.	100%	Ex.	98%	Ex.	100%
#4	X	BP	85%	BP	50%	BP	90%	BP	50%	BP	100%	BP	30%	BP	20%
#5	✓	Chol.	80%	CAD	50%	Chol.	100%	Chol.	90%	Chol.	80%	Chol.	80%	Chol.	100%
#6	✓	Ex.	95%	Ex.	80%	Ex.	100%	Ex.	90%	Ex.	60%	Ex.	95%	Ex.	100%
#7	✓	Ex./Sfty	80%	Ex./Sfty	80%	Ex./Sfty	95%	Ex./Sfty	100%	Ex./Sfty	100%	Ex./Sfty	85%	Ex./Sfty	100%
#8	✓	BP/Diet	90%	BP	88%	BP	90%	BP	85%	BP	80%	BP/Diet	58%	BP	80%
#9	✓	Chol.	90%	Chol.	50%	Chol.	100%	Chol.	95%	Chol.	80%	Chol.	93%	Chol.	100%
#10	✓*	Chol.	90%	Chol.	70%	Chol.	90%	Chol.	80%	Chol.	90%	Chol.	99%	Chol.	90%
#11	X	BP	80%	BP	80%	BP	90%	BP	70%	BP	50%	BP	69%	BP	20%
#12	✓	BP/Diet	85%	BP/Diet	80%	BP/Diet	90%	BP/Diet	90%	BP/Diet	10%	BP/Diet	93%	BP/Diet	100%
#13	✓	Ex.	95%	Ex.	80%	Ex.	90%	Ex.	90%	Ex.	80%	Ex.	99%	Ex.	100%
#14	✓	Chol.	85%	Chol.	80%	Chol.	90%	Chol.	90%	Chol.	100%	Chol.	89%	Chol.	100%
#15	X	Diet	70%	Diet	80%	Diet	90%	Diet	80%	Diet	100%	Diet	50%	Diet	50%
#16	✓	Ex.	85%	Ex.	80%	Ex.	90%	Ex.	85%	Ex.	80%	Ex.	85%	Ex.	50%
#17	✓	Chol.	85%	Chol.	80%	Chol.	85%	Chol.	80%	Chol.	80%	Chol.	62%	Chol.	100%
#18	✓	BP	80%	BP	70%	BP	80%	BP	85%	BP	80%	BP	90%	BP	100%
#19	✓	Chol.	95%	Chol.	80%	Chol.	90%	Chol.	100%	Chol.	100%	Chol.	100%	Chol.	50%
#20	X	Ex.	60%	Ex.	50%	Ex.	80%	Ex.	80%	Ex.	100%	Ex.	83%	Ex.	100%
#21	✓	BP	80%	BP	70%	BP	90%	BP	80%	BP	100%	BP	88%	BP	100%
#22	X	BP	65%	BP	70%	BP	80%	BP	90%	BP	10%	BP	63%	BP	50%
#23	✓*	Chol.	95%	Chol.	80%	Chol.	100%	Chol.	85%	Chol.	40%	Chol.	90%	Chol.	100%
#24	✓*	Meds	85%	Meds	80%	Meds	100%	Meds	100%	Meds	100%	Meds	99%	Meds	50%
#25	✓*	Ex.	85%	Ex.	90%	Ex.	90%	Ex.	90%	Ex.	40%	Ex.	82%	Ex.	100%
#26	✓	Chol.	80%	Chol.	80%	Chol.	100%	Chol.	85%	Chol.	80%	Chol.	93%	Chol.	100%
#27	✓	Ex.	90%	Ex.	80%	Ex.	100%	Ex.	75%	Ex.	100%	Ex.	100%	Ex.	100%
#28	X	Ex.	85%	Ex.	50%	Ex.	90%	Ex.	70%	Ex.	100%	Ex.	67%	Ex.	50%
#29	✓	Stroke	90%	Stroke	80%	Stroke	90%	Stroke	75%	Stroke	100%	Stroke	100%	Stroke	100%
#30	X	Stress	80%	Stress	80%	Stress	100%	Stress	55%	Stress	100%	Stress	79%	Stress	50%
#31	✓	Rsk/Rec	80%	Rsk/Rec	80%	Rsk/Rec	90%	Rsk/Rec	85%	Rsk/Rec	95%	Rsk/Rec	75%	Rsk/Rec	100%
#32	✓	CHF	80%	CHF	70%	CHF	80%	CHF	80%	CHF	100%	CHF	97%	CHF	100%
#33	✓	Ex.	90%	Ex.	80%	Ex.	100%	Ex.	85%	Ex.	100%	Ex.	90%	Ex.	100%
#34	X	Ex/Risk	60%	Ex/Risk	50%	Ex/Risk	90%	Ex/Risk	85%	Ex/Risk	100%	Ex/Risk	85%	Ex/Risk	20%
#35	✓*	Rsk/Rec	90%	Rsk/Rec	80%	Rsk/Rec	80%	Rsk/Rec	80%	Rsk/Rec	100%	Rsk/Rec	100%	Rsk/Rec	100%
#36	✓	A&P	85%	A&P	70%	A&P	100%	A&P	100%	A&P	90%	A&P	90%	A&P	100%
#37	✓	Diabetes	95%	Diabetes	90%	Diabetes	90%	Diabetes	80%	Diabetes	100%	Diabetes	85%	Diabetes	20%
#38	✓	CHF	85%	CHF	80%	CHF	100%	CHF	90%	CHF	80%	CHF	75%	CHF	100%
#39	✓	CHF	80%	CHF	80%	CHF	100%	CHF	90%	CHF	100%	CHF	100%	CHF	100%
#40	X	CHF	65%	CHF	40%	CHF	100%	CHF	100%	CHF	50%	CHF	50%	CHF	90%

^✓*= Item Revised & Accepted

^✓= Item Accepted

^X= Item Rejected

Table 2.

#	Test Item	Expert Mean Agreement
1	A term that is used to describe blood pressure higher than normal is:	84.3
	a. Hyperactive   b. Hypertension   c. Vasoconstriction   d. Hypotension   e. I don’t know
2	A bad form of cholesterol is:	91.8
	a. HDL (High Density Lipoprotein)   b. Hbg (Hemoglobin)   c. LDL (Low Density Lipoprotein)   d. CPK (Phosphokinase)   e. I don’t know
3	The best way to measure if you are working hard enough during exercise is to:	95.4
	a. Count the minutes of each exercise session   b. Count your pulse rate during exercise   c. Count the settings on the exercise equipment   d. Count your breathing rate during exercise   e. I don’t know
4	High amounts of Bad Cholesterol may lead to heart problems by blocking:	82.9
	a. Heart valves   b. Heart arteries   c. Heart electricity   d. Heart chambers   e. I don’t know
5	Each heart healthy exercise conditioning session should be not less than	88.6
	a. 15 minutes   b. 30 minutes   c. 60 minutes   d. 90 minutes   e. I don’t know
6	If you feel lightheaded during or after exercise, the best thing to do is:	91.4
	a. Continue to exercise   b. Monitor your pulse   c. Breath rapidly & deeply   d. Sit or lie down   e. I don’t know
7	The food item we eat or drink that most directly effects blood pressure is:	81.6
	a. Fat   b. Salt   c. Water   d. Sweets   e. I don’t know
8	The two best ways to decrease bad cholesterol is by:	86.9
	a. Exercise & medication   b. Cardiac cath & medication   c. Diet & medication   d. Chelation & angiograms   e. I don’t know
9	People who have had heart problems due to blocked arteries should keep their triglyceride levels less than	87.0
	a. 150 mg/dL   b. 200 mg/dL   c. 250 mg/dL   d. 200 mg/dL   e. I don’t know
10	The amount of sodium (salt) in the daily diet should be less than:	91.1
	a. 3,400 mg.   b. 1,500 mg.   c. 400 mg   d. 3,000 mg   e. I don’t know
11	To gain the most benefit from a heart healthy exercise program you should exercise at least:	90.6
	a. Once each week   b. Twice per week   c. Three times per week   d. Seven times per week   e. I don’t know
12	The type of cholesterol that is good and acts to remove harmful cholesterol from the body is:	90.6
	a. HDL (High Density Lipoprotein)   b. Hbg (Hemoglobin)   c. LDL (Low Density Lipoprotien)   d. VLDL   e. I don’t know
13	The best exercise for loosing body fat around the waist is:	82.1
	a. Leg lifts   b. Sit-ups   c. Crunches   d. Walking   e. I don’t know
14	Choose the food below this is most likely to increase triglyceride levels:	81.7
	a. Fried Foods   b. Sweets   c. Red Meat   d. Eggs   e. I don’t know
15	All of the below may help reduce high blood pressure except	83.6
	a. Weight lifting & resistance exercises   b. Body weight reduction   c. Aerobic “cardio” exercises   d. Stop smoking   e. I don’t know
16	The ideal goal for people who have had heart problems due to blocked arteries is to keep their LDL cholesterol less than:	87.9
	a. 50 mg/dL   b. 70 mg/dL   c. 150 mg/dL   d. 200mg/dL   e. I don’t know
17	High resting blood pressure levels increase the risk of all the below except:	86.9
	a. Stroke   b. High cholesterol   c. Heart attack   d. Diseased arteries   e. I don’t know
18	The type of food that is most likely to increase LDL cholesterol is:	88.6
	a. Fried Foods   b. Sweets   c. Pasta   d. Eggs   e. I don’t know
19	If you feel you are having side effects from your blood pressure medication you should:	87.7
	a. Decrease the dose   b. Call your physician   c. Ignore the symptoms   d. Stop taking the medication   e. I don’t know
20	All of the below choices are types of heart healthy exercises except	83.9
	a. Walking   b. Weight lifting   c. Swimming   d. Cycling   e. I don’t know
21	The health practice that is most effective in increasing (HDL) cholesterol is:	88.3
	a. Low fat diet   b. Exercise   c. Stress reduction   d. Weight reduction diet   e. I don’t know
22	When should you plan and begin a home or community based exercise program.	92.1
	a. During the Cardiac Rehab program   b. During the last session of Cardiac Rehab   c. Immediately after the Cardiac Rehab program   d. Two weeks after completing Cardiac Rehab   e. I don’t know
23	People who have had heart problems due to blocked blood vessels in the heart also are at a higher risk for stroke because:	90.7
	a. Most strokes are caused by weakened blood vessels   b. Most strokes are caused by blood thinners   c. Most strokes are caused by blocked vessels in the brain   d. Most stroke are caused by heart attacks   e. I don’t know
24	If you have chest pain that is caused by movement of or pressing on the painful area it is likely that the pain is:	86.4
	a. Heart related   b. Muscle or bone related   c. Breathing related   d. Stomach related   e. I don’t know
25	If you become short of breath when lying down or during rest it is most likely due to fluid build-up in the lungs caused by:	86.7
	a. Bronchial congestion   b. Emphysema   c. Exertion   d. Heart failure   e. I don’t know
26	The best drink to replace fluid lost when exercising in warm conditions is:	92.1
	a. Water   b. Sports drinks   c. Fruit Juice   d. Diet soda   e. I don’t know
27	The following are all controllable (modifiable) risk factors for heart disease except	90.0
	a. High cholesterol   b. High blood pressure   c. Family History   d. Smoking   e. I don’t know
28	The job of the coronary arteries is to:	90.7
	a. Supply the lungs with blood   b. Take blood away from the heart   c. Supply the heart muscle with blood   d. Deliver blood to the brain   e. I don’t know
29	It is possible to prevent Type II (adult onset) diabetes.	85.7
	a. True   b. False   c. I don’t know
30	The most effective method for you to evaluate heart failure is:	87.1
	a. Periodic checks with your doctor   b. Weighing yourself at the same time each day   c. Measure the amount of salt you eat each week.   D. Measure the amount of water you drink each day   e. I don’t know
31	A person may be having the signs and symptoms of heart failure if:	92.9
	a. They wake up from sleep with sever air hunger   b. They notice that they become short of breath when doing their normal daily activities   c. They notice, the first thing in the morning, that they have ½ inch pitted swelling around both ankles   d. All of the above may be signs of heart failure.   e. I don’t know

^aDuring the time period 2002–2011, two alterations have occurred to reflect changes in national guidelines. Item #10, correct response: Sodium 2400mg –changed to- 1500mg/day; and item #16, correct response: LDL cholesterol <100mg/dL- changed to- <70mg/dL. These changes did not affect response sequence or alter the substantive context of the correct answer or item distracters.]

Sample Characteristics

A convenience sample of 3,638 adults tested from 2002–2011 in our urban Midwestern phase II CR program was chosen for the study (Table 3).

Table 3.

Demographic Characteristics for Patients n = 3638

	Entry Age M/(SD)	Yrs Educ. M/(SD)	Race
			Caucasian	AA	Asian	Other
Male	64.58 (11.28) n=2416	13.81 (2.9) n = 2268	n = 2220 % = 94.9	n = 99 % = 4.2	n = 2 % = .1	n = 1 % = 0
Female	65.99 (11.28) n=1222	12.94 (2.27) n = 1149	n = 1060 % = 89.5	n = 109 % = 9.2	n = 2 % = .2	n = 2 % =.2
Total	65.05 (11.53) n=3638	13.52 (2.72) n =3417 334171928	% = 93.1 n = 3280	% = 5.9 n = 208	% = .1 n = 4	% = .1 n = 3

Abbreviations: AA, African American; M, mean; SD, standard deviation.

^an size variability in sub-groups due to missing data.

Test Administration

Subject testing was conducted in a controlled environment during intake and discharge sessions at our facility. Each subject was instructed regarding the importance of candid responses and use of the “I don’t know” answer option to provide an accurate assessment of their knowledge levels. Test design allowed subjects to self-select either paper-and-pencil or computer mouse-click application. Visual test instructions were the same for both formats. IRB approval was obtained for the pilot project in 2001. The test was piloted then in a group of 50 consecutive CR patients who, through the use of a formatted questionnaire, rated the test to be understandable, ‘quick and easy to take’ (mean administration time 13.5 minutes) and unambiguous in reading interpretation.

Data Collection

Data was prospectively collected in our CR clinical database then retrospectively deidentified and entered in a separate database created in SPSS version 16 which was used to perform descriptive, validity and reliability analyses³⁵.

Measures

Validity

While there are several types of validity the three most important in the educational literature are, content, construct and criteria-related^{36, 37}. There is agreement in the methodologic literature that content validity is largely a matter of judgment, involving two distinct phases; First, a-priori efforts by the test developer to enhance content validity through careful conceptualization and domain analysis prior to item generation, and Second a-posteriori efforts to evaluate the relevance of the test’s content through expert assessment^{24, 25, 38}. CKAT content validity was established by SCR developmental staff and clinical content experts, using this approach by use of a ‘table of specification’ described by Anastasi as ‘the systematic examination of test items to insure coverage of a representative sample of the content to be measured’³⁹. Construct validity was evaluated through the use of factor analysis, a statistical methodology employed to identify underlying constructs of the test. Predictive validation was analyzed using a hierarchical linear regression holding baseline knowledge scores and functional work capacity constant while predicting post CR program values⁴⁰. Also, correlation analysis was performed to evaluate concurrent performance with other recognized health care evaluative outcome measures, including the Medical Outcomes Study Short Form-36 (SF-36), Duke Activity Status Index (DASI),Mini-Mental State Exam (MMSE) and Beck Depression Inventory (BDI)^41–45.

Reliability

Cronbach’s Alpha is a widely used estimate to support internal consistency of items in a scale. It is a measure of the level of mean intercorrelation of items. Alpha increases as the inter-correlation among test items increase and is considered to be a robust internal consistency estimate of reliability of test scores. An Alpha of .70 is generally recognized in the literature as acceptable⁴⁶. We further measured internal consistency through use of the Spearman-Brown split half reliability coefficient, a method in which the items comprising a test are systematically split into two groups and scored independently the results of which are compared to compute a correlation coefficient³⁷.

Item Analyses

A point biserial correlation (r_pb) was used to measure the relationship between performance on an item (dichotomous variable) and the total score (continuous variable) and was calculated to determine the relative importance of test items⁴⁷. Individual items were further analyzed using an Item Difficulty Index which ranges from 0 to 1 and is the proportion of subjects who answered the test item accurately. The greater the difficulty of an item, the lower its index will be. If the item difficulty is more than .75 it is considered an easy item and if the difficulty is below .25, it is a difficult item. The Discrimination Index which ranges from +1 to −1 and is a comparison of how subjects obtaining high total scores on an instrument performed on individual items as compared to those with low total scores further analyzed individual items (Table 4)⁴⁸.

Table 4.

Item Analysis

Item #	Point Biserial		Item Difficulty Index		Discrimination Index
	Pre	Post	Pre	Post	Pre	Post
1	.446**	.357**	.83	.93	.32	.15
2	.590**	.534**	.49	.90	.75	.25
3	.505**	.298**	.81	.97	.38	.07
4	.439**	.305**	.92	.98	.18	.05
5	.411**	.451**	.45	.84	.46	.33
6	.350**	.243**	.80	.94	.24	.11
7	.384**	.315**	.81	.90	.27	.22
8	.229**	.386**	.54	.63	.19	.51
9	.469**	.477**	.39	.83	.54	.35
10	.434**	.501**	.32	.86	.48	.32
11	.414**	.183	.85	.95	.24	.07
12	.604**	.530**	.50	.91	.76	.23
13	.323**	.450**	.37	.83	.35	.35
14	.311**	.603**	.13	.53	.22	.77
15	.508**	.555**	.52	.85	.60	.37
16	.436**	.523**	.27	.76	.46	.48
17	.559**	.616**	.54	.81	.67	.46
18	.417**	.318**	.61	.87	.43	.18
19	.424**	.282**	.93	.98	.14	.04
20	.475**	.415**	.83	.96	.32	.11
21	.420**	.597**	.24	.68	.42	.66
22	.300**	.505**	.32	.76	.30	.48
23	.491**	.446**	.58	.85	.56	.30
24	.479**	.592**	.46	.75	.56	.57
25	.447**	.542**	.49	.79	.50	.47
26	.320**	.237**	.87	.97	.16	.07
27	.526**	.499**	.76	.91	.47	.22
28	.477**	.450**	.58	.78	.54	.41
29	.428**	.430**	.54	.81	.49	.38
30	.319**	.593**	.13	.53	.23	.77
31	.417**	.372**	.66	.86	.39	.26

^a**p <.01

Results

As previously described, content validity was systematically established by SCR developers through use of item CVI relevance analysis, domain assignment and strict adherence to a ‘table of specification’ by a seven member multidisciplinary team of content experts. Construct validity was determined via a principle component factor analysis with varimax rotation performed on 3,638 pre test scores⁴⁹. Simple structure was approached but not obtained. Seven underlying factors explaining 42% of the total variance were identified. Rotated factor loadings, were interpreted as the Pearson correlation between the test item and the factor (Only coefficients of .40 and above were retained for interpretation). Using these criteria, six items did not load significantly on any factor but were retained due to the underlying curriculum based content defining their inclusion in the instrument (Table 5). Test reliability was further supported via ‘Cronbach’s Alpha’ analysis, an estimate of internal consistency. Cronbach’s Alpha of .852 and .865 was obtained on the pilot and total test data set respectively (An Alpha of .70 is generally recognized in the literature as acceptable). Internal consistency was evaluated by use of the Spearman-Brown split half reliability coefficient. The results .817 pilot data, and .840 total data further confirmed the internal consistency of the instrument. Pre and post test, point biserial correlations measuring performance between each item (dichotomous variable) and the total score (continuous variable), determine the relative importance of test items. Values >.25, required for a “good” item, were significantly correlated with the total score at p<.01 for both the pre and post test with the exception of post test item #11. The Item Difficulty Index, which ranges from 0 to 1, reports the proportion of people answering a test item accurately (Table 4). The greater the difficulty of an item, the lower its index will be. If the item difficulty is more than .75 it is considered an easy item and if the difficulty is below .25, it is a difficult item. By this criteria 3 items would be considered difficult and 10 easy on the pretest. The remaining 18 items made up 58% of the test with an average difficulty score of 47.9 (50 being optimal). Post test difficulty scores ranged from .53 to .98, a trend that illustrates an overall expected decrease in item difficulty as a product of learning effect, described in educational psychology as the expected result of affective educational intervention^50–52. The Discrimination Index which ranges from +1 to −1 is a comparison of how subjects obtaining high total scores on an instrument performed on individual items as compared to those with low total scores (Table 4). All values pre and post were positive, which indicates good validity since the formula for this calculation is such that the value will be positive if more subjects in the high scoring group chose the correct answer than did subjects in the low scoring group. Analyses between paper and pencil, and computer modes of administration in 1,999 subjects completing both pre and post tests were not significantly different (pre-test, p<.064: post-test, p<.774), confirming uniform interpretation of results across application modes. Also this group of completer’s test scores positively correlated with attendance at group education sessions p<.035, further validating the curriculum based design of the instrument. CKAT pre testing correlates with the (SF-36) Physical and Mental Composite Scales (n =3298), MMSE (n=1164), BDI (n =1654), and the DASI (n =1875) of p≤ .01 demonstrated CKAT’s strong levels of concurrent validity with established health outcomes instruments. Concurrent validity was further demonstrated in a convenience sample of subjects completing two established adult health literacy tests; The Test of Functional Health Literacy in Adults (TOFHLA) and the Medical Term Recognition Test (METER), which were found to correlate significantly with CKAT; r = .299, p <.01 (n = 152) and r = .330, p<.01 (n = 152) respectively ^{53, 54}. Using hierarchical linear regression in a sample of 1,783 subjects completing both pre and post-testing, CKAT post-test scores were significant p<.001 in predicting CR program estimated mean FWC at program completion while controlling for pre-program CKAT scores and FWC results. The positive direction of this relationship strongly indicates higher scores on the CKAT predictive of FWC gain.

Table 5.

	Factor Analysis							Factor Description	Baseline % Correct
Question#	Component
	#1	#2	#3	#4	#5	#6	#7
5	.614							Factor #1 Self-Management Principles	51.8
24	.604								46.0
27	.563								75.6
17	.554								53.5
20	.525								83.1
23	.494								57.9
28*
12		.814						Factor #2 Risk Identification	49.4
2		.808							48.6
16		.519							26.8
18		.487							60.7
9		.486							39.7
4			.649					Factor #3 Health Improvement Issues	91.8
1			.599						82.5
3			.578						81.0
5*
11*
25				.686				Factor #4 Symptom Recognition	39.4
30				.685					12.6
31				.421					66.1
29*
10*
13					.580			Factor #5 Reasoning Ability	37.1
14					.452				13.0
21					.450				87.1
22*
26						.637		Factor #6 Self-Care Strategies	86.5
19						.617			92.7
8							.739	Factor #7 Intervention Identification	54.3
6							.457		80.2
7							.407		80.9

^aTable metric shows Pearson correlation between each test item and the corresponding factor.

^b*Indicates items that did not load significantly (.4 or greater) on any factor

Discussion

Cardiac disease-knowledge assessment instruments that have been developed in a variety of inpatient and outpatient settings, lack population specificity, systematic construct and rigorous reliability/validity analysis^{8, 9, 15}. We have described a comprehensive and systematic test development approach targeted for adults with cardiac disease learning needs which adheres to nationally recognized patient centered outcomes⁵. The design purposefully avoided parochial and non-evidenced based content, prevalent in previous instrument design¹⁰. The test development focused on fundamental adult cognitive learning objectives of knowledge, understanding, and application enumerated in Bloom’s and Anderson’s seminal work²². Test item development reflected systematic identification of CR specific learning content domains of blood pressure control, lipid management, exercise, heart failure, cardiovascular disease risk, medications, anatomy & physiology, diet, diabetes, and stroke¹⁴(Figure 1). Self-Management Principles, Risk Identification, Health Improvement Issues, Symptom Recognition, Reasoning Ability, Self-Care Strategies and Personal Intervention, a subset of highly desirable underlying personal care constructs immerged via factor analysis (Table 5). Validity and reliability analyses were robust. The test itself is unique in its use of a “don’t know” response set in multiple-choice format. Previous formats employed dichotomous (true/false) responses which increase the error effect of random guessing and test-taking acumen²⁸. The “don’t know” feature further adds to the instrument’s utility as a clinical barometer of baseline learning needs pretreatment and remediation post treatment, by reducing error of interpretation. Attention was specifically directed at readability and ease and speed of administration. Administration and scoring uniformity in either paper & pencil or computer formats add to its utility as a clinical tool in diverse settings. The instrument has been used as an effective clinical tool by staff during initial and pre-discharge administrations coupled with follow-up one-on-one goal setting and outcome assessment reviews. As a result clinicians are better prepared to analyze the learning specific content needs and focus individual educational remediation.

Limitations

The current findings were limited in several ways. First, our findings are based on a study population representative of a large Midwestern, hospital based CR program. Second, we included six items that did not significantly load on one of the underlying seven psychometric factors identified in factor analysis. The choice to include these six items was based on their strong content relevancy based on program learning objectives. Third, a memorization effect may have occurred although mitigated by a mean time of 90.4 days between pre and post-testing applications. Also, subgroup analysis and, application and validity analysis in educational settings devoted to effective management of heart disease risk in populations other than CR needs to be explored.

Implications

Accurate assessment and interpretation of clinically relevant data enhances intervention at every level. It is particularly important for clinicians to accurately assess knowledge in their adult cliental who are often facing the need to understand a myriad of baffling terms and concepts then make significant adjustments to personal behavior, all during times of high psychological stress. We have described development and testing of such an instrument specifically focused at the cardiovascular disease management needs of this older-adult population. Although designed and validated in this large CR cohort, the utility of this instrument in other settings directed at integrated management of cardiovascular disease risk is reasonable. However, further research confirming test validity is needed in these and CR settings representing a broader range of demographic diversity.

Conclusions

Instruments, designed to measure achievement of cardiovascular disease management knowledge and behavioral change concepts in patients post cardiac event, indicate a variety of psychometric design flaws^{7–10, 15}. Realizing this need we have systematically developed and tested an instrument specifically designed for this adult population that satisfies psychometric parameters and reliability/validity analysis in a large representative cardiac disease patient population.

What’s New and Important.

• Comprehensive and systematic development of an adult outpatient heart disease knowledge and health management assessment tool.

• Unique multiple choice format including correct response, three distracters and “Don’t Know” responses enhancing accuracy of measurement and clinical utility.

• Rapid test administration.

• Valid paper-and-pencil or computer application and scoring.

• Rigorous validity and reliability analyses in over 3500 patient applications.