Practice analysis and changes to the Chiropractic Board of Clinical Nutrition diplomate exam

Abstract

Objective

The main objective of this study was to report results of the practice analysis survey and to provide insights into the average levels of performance and the importance of professional tasks executed by chiropractic nutritionists. In addition, this study informs the chiropractic community of the changes made to the Chiropractic Board of Clinical Nutrition diplomate exam.

Methods

Seventy-eight practicing chiropractic nutritionists responded to the practice analysis survey. Their responses were analyzed, and conclusions about frequency and importance of performance tasks were reached. A panel of subject matter experts provided a qualitative review of the survey responses. The quantitative and qualitative analyses of the survey responses indicated that minor changes to the test plan were needed.

Results

Descriptive statistical techniques were employed to analyze the survey responses. The qualitative panel suggested reducing the number of domains on the nutrition exam from 7 to 6 by combining Laboratory and Nutrition-Specific Testing and Imaging and Other Special Studies domains. Additionally, the panel decided on the final distribution of weights combining the quantitative results with qualitative perspectives.

Conclusion

The practice analysis is a first step in the definition of the skills required for practicing chiropractic nutritionists. The analysis becomes one of the references and a decision-making tool used by the board for developing and administrating quality assessments.

INTRODUCTION

The Chiropractic Board of Clinical Nutrition (CBCN) collaborated with the National Board of Chiropractic Examiners (NBCE) to develop, administer, and score the CBCN exam. The CBCN was founded in 2009 as a credentialing and certifying agency for chiropractic nutritionists. The goal of the specialty group is to advance clinical nutrition while at the same time enhancing the health of chiropractic patients.

A diplomate of the Chiropractic Board of Clinical Nutrition is a licensed chiropractic physician who has successfully completed a postdoctoral program in nutrition of at least 300 credit hours from a chiropractic college or university, institution, foundation, or agency whose program is approved by an accrediting agency recognized by the US Department of Education. In addition, the diplomate has successfully passed the certifying examination administered jointly by the CBCN and NBCE.¹

The CBCN assessment is a 200-question exam administered electronically in various testing centers across the United States.² The exam was first developed in 2010 and administered in 2011. The original test plan encompassed 7 domains: Case History, Nutrition-Related Physical and Orthopedic/Neurologic Examinations, Laboratory and Nutrition-Specific Tests, Imaging and Special Studies, Diagnosis or Clinical Impression, Treatment and Specific Nutritional Interventions, and Case Management.

This study presents methodology and reports the results of the recent practice analysis survey conducted by the NBCE for the purpose of validity reaffirmation of the CBCN exam. The NBCE is responsible to its clients for the preparation of psychometrically sound and legally defensible licensure examinations. The periodic performance of practice analysis (ie, job analysis) studies assists the NBCE in evaluating the validity of the test plan, which guides content distribution of the licensure examination. Furthermore, practice analysis studies have long been recognized by measurement and testing professions as important sources of validity evidence for licensure examinations.²

Literature Review

Exam Validity

Assessment is an important component for ensuring public protection in professional licensing and certification. A model for test development assumes that test creators will ensure the validity of developed exams for the intended populations of test takers. The validation process begins with the formulation of detailed trait or construct definitions derived from theory, prior research, or systematic observation and analyses of the relevant behavior. Test items are then prepared to fit the construct definitions.³ Since the 1980s, test validity has increasingly become a significant concern, and evidence of validity in high-stakes assessment programs is now expected. Cronbach⁴ states that “validation was once a priestly mystery, a ritual behind the scenes, with the professional elite as witness and judge. Today it is a public spectacle combining the attractions of chess and mud wrestling.”

Kane⁵ points out that measurement is an inferential process in which limited samples are used to draw conclusions about people or organizational units. “To validate an interpretation or use of measurements is to evaluate the rationale, or argument, for the claims being made, and this in turn requires a clear statement of the proposed interpretations and uses and a critical evaluation of these interpretations and uses.”

The Standards for Educational and Psychological Testing require a verifiable close connection between the content of a test and the construct it intends to measure. “Evidence based on test content can include logical or empirical analyses of the adequacy with which the test content represents the content domain and of the relevance of the content domain to the proposed interpretation of test scores.”²

Practice/Job Analysis

Practice analysis performs a fundamental role in developing valid licensure and certification assessment tools.⁶ Knapp and Knapp⁷ define practice analysis as a systematic collection of data describing the knowledge, skills, and/or competencies required to practice a profession. The practice analysis is conducted to inquire about the work performed by professionals to document tasks essential to practice.^2,8 As a process of obtaining information about professions, practice analysis is the most widely used organizational data collection technique.⁹ Practice analysis helps to establish test validity by creating a profile of the profession, tracking trends in professional practice, and providing information vital to the development and refinement of professional programs. It also identifies current practice standards while anticipating future change.

Legal Standards for Practice Analysis

The legal recognition of practice/job analysis started in the area of employee selection. Legal cases related to job analysis are relevant to licensing and professional testing because “they characterize which types of procedures are viewed by the court as being appropriate for defining professional responsibilities and knowledge.”⁷ The Uniform Guidelines on Employee Selection Procedures provide guidance for employee selection procedures, helping employers to comply with federal laws pertaining to Title VII of the Civil Rights Act (1964). The guidelines provide many references to practice/job analyses. Specifically, Section 5 provides standards for validity studies demanding that selection procedures will have established evidence of criterion-related content and construct validity. The section requires that licensing tests, which are used to allow entrance into a profession, should be consistent with generally accepted professional standards for evaluating standardized tests and that validity studies should not be conducted by the test user.¹⁰

The Uniform Guidelines were adopted by 5 federal agencies: the Equal Employment Opportunity Commission, the Office of Personnel Management, the Department of Labor, the Department of Justice, and the Department of Treasury.¹¹ The leitmotif of the guidelines is the need for a close connection between a selection instrument (test) and the requirements of the profession for which the test is being used.¹²

Although there is no federal law that requires a practice analysis to be conducted,¹³ there have been several court cases that directly referred to the procedure. In 1971, a case was debated before the US Supreme Court, which ruled against a public utility corporation that required a high school diploma for its higher-paid jobs. The Court ruled that “tests must be reasonably related” to the job for which the test is required.¹⁴ Later, in 1983, in Kirkland vs New York Department of Correctional Services, the Court stated that “identification of the relative importance of the skills and tasks involved in a job and the competency required for the various aspects of a position are essential functions of a job analysis.” Further, “the cornerstone in the construction of a content valid examination is the job analysis.”¹⁵

In 1975, employees of the Albemarle Paper Co. claimed to have suffered from racially discriminatory hiring and promoting practices. The Court ruled that “job relatedness cannot be proven through vague and unsubstantiated hearsay” and that “limiting job analysis to selected jobs, that are unrepresentative of the full range of work performed, is inadequate for test development.”¹⁶

Professional Standards

Testing specialists in the field of educational measurement are very particular about the requirements for a high-stakes assessment. The requirements that pertain to practice analyses include Standards for Educational and Psychological Testing² and the Principles for the Validation and Use of Personnel Selection Procedures.¹⁷ Although the standards and principles are not legal documents, they frequently have been used by the courts to determine the appropriateness of validation procedures.¹⁸ Therefore, many licensing agencies have elected to adhere to the standards in their test development procedures.

The standards emphasize that job analysis is the primary basis for determining the content and assessing the validity of licensure examinations. A practice analysis study conducted in conjunction with exam development efforts should establish the frequency and importance of core professional tasks. For an exam that serves a purpose of public protection, based on practice analysis, test developers should be able to isolate and focus on professional core competencies that are important for the protection of the public.²

Surveys are often constructed to collect job-related data where respondents are asked to report the frequency and the importance of the tasks they perform at work. The collected data are analyzed and numerical estimates calculated for frequency of performance, importance to public protection, and necessity to develop competency in a task at the time of initial licensure.⁶

Current Study

The principal purpose of this study was to report results of the practice analysis survey and to provide insights into the average levels of performance and importance of professional tasks executed by chiropractic nutritionists. Coinciding with this purpose were more specific examinations of content validity in each domain of the exam. Finally, based on the quantitative results obtained by the survey and qualitative panel review of the test domains, the CBCN test plan was slightly modified. The final objective of this study was to inform the chiropractic community of these modifications.

METHODS

Several methodological approaches exist for conducting practice analyses, including observational studies, qualitative interviews, reviews of clinical incidents, surveying, review of job diaries, review of checklists, and others.¹⁹ When selecting a methodology for practice analysis, several aspects are usually considered, including cost, practicality, purpose, analyst experience, and the profession to be analyzed.¹¹ Nevertheless, a task survey questionnaire is the most common method of collecting information from practicing professionals.⁶ Therefore, surveying was used as the data collection methodology for this practice analysis.

Ethics

This study was approved by the NBCE Institutional Review Board. The board granted an exemption since the research involves survey procedures and the survey responses are anonymous.

The Survey

The 2019 survey relied on previously collected information to inform about trends in chiropractic nutrition and health care and to anticipate possible changes in the future of practice. A diverse group of chiropractic nutritionists developed and approved the content of the initial survey, which was administered in 2010.

A panel of subject matter expert (SMEs), in order to assist with the practice analysis study, was created internally at the NBCE. The panel included 3 doctors of chiropractic who also have extensive experience in test construction, a psychometrician, and a psychometric intern. One of the panel members is also a diplomate of the CBCN.

The CBCN board members and the NBCE panel of SMEs had thoroughly reviewed the survey administered in 2010, weighing the relevance of questions and providing necessary modifications. Additionally, the Role Description for a Diplomate of the CBCN was reviewed and confirmed as up to date and accurate by the CBCN Board.

The primary purpose of the 2019 survey was to inform and provide validity evidence for the content of the CBCN diplomate examination. Specifically, those responsible for test development should have empirical evidence to inform the selection of test content. The 2019 survey of clinical nutritionists instructed respondents to indicate the frequency with which they perform professional functions and specific nutritional interventions; respondents were also asked to provide an opinion of the risk to a patient's health or safety if the function or care was omitted or poorly performed.

Participants

The NBCE contacted a list of chiropractic nutritionists provided by the CBCN with a request to complete the survey. The original list contained 168 contacts; 78 survey responses were returned to the NBCE, which makes the survey response rate 46%. A variety of descriptive data were collected from the participants (Table 1).

Table 1-.

Descriptive Statistics for Demographic Variables Included in the Study

Variable	%	N
Gender		78
Male	69%
Female	31%
Ethnicity		74
Asian/Pacific Islander	0%
African American/Black	1%
White	92%
Hispanic/Latino	3%
Native American	0%
Other	4%
How many years have you been in practice?		76
Fewer than 2 y	0%
2–4 y	0%
5–15 y	12%
16–25 y	21%
More than 25 y	67%
What is the highest level of nonchiropractic education you have obtained?		77
High school diploma	0%
Associate degree	8%
Bachelor's degree	52%
Master's degree	20%
Doctoral degree	17%
Other	4%
What are your gross practice earnings?		77
Less than $200,000	36%
$200,000–$300,000	23%
$300,000–$400,000	9%
$400,000–$500,000	12%
More than $500,000	20%
What is your postgraduate nutrition diplomate status through an ACA or ICA boards?		77
Nutrition diplomate status in progress	4%
Nutrition diplomate status complete	96%
Do you currently include nutritional evaluation counseling?		77
Yes	97%
No	3%
How many hours per week you practice chiropractic and/or nutritional evaluation, counseling, supplementation, testing, etc.?		75
9 or fewer	11%
10–19	21%
20–29	27%
30–39	27%
40–49	12%
50–59	3%
60 or more	0%
How many hours in your week are dedicated to nutritional evaluation, counseling, supplementation, testing, etc. in your practice (this refers to patient management and not teaching, research, etc.)?		73
9 or fewer	33%
10–19	44%
20–29	15%
More than 40	8%
How many years have you been in practice?		74
Fewer than 2 y	0%
2–4 y	0%
5–15 y	11%
16–25 y	19%
More than 25 y	70%
How many years have you included nutritional evaluation, counseling, supplementation, testing, etc. in your practice (this refers to patient management and not teaching, research, etc.)?		75
Fewer than 2 y	0%
2–4 y	1%
5–15 y	16%
16–25 y	31%
More than 25 y	52%
What is the total number of patients (not patient visits) you personally treat per week with chiropractic care and/or with nutritional evaluation, counseling, supplementation, testing, etc.?		72
Fewer than 50	56%
50–99	28%
100–149	13%
150–199	3%
200–249	1%
250–300	0%
More than 300	0%
What is the total number of patients (not patient visits) you personally treat using only nutritional evaluation, counseling, supplementation, testing, etc. each week?		72
0	3%
1–49	86%
50–99	7%
100–149	3%
150–199	0%
200–249	1%
250–300	0%
More than 300	0%
What is the size of the community in which your practice is located?		74
City	37%
Suburb of city	32%
Large town	12%
Small town	16%
Rural	3%
Do you have staff privileges at a hospital?		73
Yes	0%
No	100%
How much of your time is spent on nutrition?		74
None	0%
1%–25%	38%
26%–50%	23%
51%–75%	22%
76%–100%	18%
How much of your time is spent documenting care?		73
None	0%
1%–25%	63%
26%–50%	29%
51%–75%	3%
76%–100%	6%
How much of your time is spent in business management? (personnel, marketing, etc.)		74
None	8%
1%–25%	74%
26%–50%	16%
51%–75%	1%
76%–100%	0%
What percentage of your patients are male?		75
None	0%
1%–25%	0%
26%–50%	31%
51%–75%	67%
76%–100%	3%
What percentage of your patients are female?		74
None	0%
1%–25%	0%
26%–50%	30%
51%–75%	51%
76%–100%	19%
What percentage of your patients are age 5 or younger?		74
None	19%
1%–25%	81%
2%– 50%	0%
51%–75%	0%
76%–100%	0%
What percentage of your patients are ages 5–17?		75
None	4%
1%–25%	93%
26%–50%	3%
51%–75%	0%
76%–100%	0%
What percentage of your patients are ages 18–30?		75
None	3%
1%–25%	55%
26%–50%	41%
51%–75%	1%
76%–100%	0%
What percentage of your patients are ages 31–50?		74
None	0%
1%–25%	23%
26%–50%	58%
51%–75%	19%
76%–100%	0%
What percentage of your patients are ages 51–64?		75
None	0%
1%–25%	22%
26%–50%	51%
51%–75%	25%
76%–100%	1%
What percentage of your patients are ages 65 and older?		75
None	1%
1%–25%	72%
26%–50%	21%
51%–75%	3%
76%–100%	3%

Measures

The survey contained 122 items; 28 items inquired about demographic characteristics of the respondents of the patients they see. The bulk of the 2019 survey of clinical nutritionists collected information from respondents regarding the frequency with which they perform professional functions and specific nutritional interventions (frequency of professional functions). Additionally, respondents were asked to provide an opinion of the risk to a patient's health or safety if the function or care was omitted or poorly performed (risk factor).

The professional functions were presented in a logical order in the survey, beginning with the clinical nutritionist initially obtaining a case history, followed by performing examinations, then performing or ordering additional nutrition-specific studies and tests and interpreting the results. Next, respondents considered the following professional functions: developing differential diagnoses, determining a probable prognosis, and formulating a treatment plan regarding the types of nutritional interventions available, giving special consideration to interactions between herbs, medications, and other nutrients. They also reported on case management functions, such as providing nutritional counseling on lifestyle habits, diets, exercise, and overall well-being. Nutrition-specific considerations related to documentation and objective outcome measures were evaluated.

Scales

The frequency of professional functions section of the survey asked clinical nutritionists to indicate how frequently during the past 12 months they had performed each of the professional functions and how frequently they had utilized various specific nutritional interventions for patient care. The frequencies were reported on a 5-point scale ranging from never to routinely. The risk factor portion presented a context where the respondent was asked to consider a patient who needed a clinical nutritionist to perform each function. The respondent was asked to assess the risk to a patient's health or safety if a clinical nutritionist omitted or poorly performed the function. The respondent was instructed to assess risk independently of how frequently they may perform the function. Risk was assessed on a 5-point scale of no risk to severe risk.

Importance Index

The creation of the importance index was done by multiplying the frequency of responses in professional functions by the risk. The professional functions responses were dichotomized in the following way:

where Response_ij is the survey response i, i = 1, 2, …, 78, on a professional function item j, j = 1, 2, …, 88. The risk responses were dichotomized in the following way:

where Response_ik is the survey response i, i = 1, 2, …, 78 on a risk item k, k = 1, 2, …, 88.

Then the importance index was calculated for each combination of professional function and risk in the following way:

where n(Response_ij = 1) is the count of “routinely” or “frequently” responses on professional function items and n(Response_ik = 1) is the count of “severe risk” or “significant risk” responses on risk items.

Calculation of Category Weight

The calculation of the category weights on the CBCN exam was performed in 2 steps. The first step was to compute the overall risk for the original 7 categories represented in the exam: 1 = Case History, 2 = Physical and Nutrition Examination, 3 = Laboratory and Nutrition-Specific Testing, 4 = Imaging and Other Specific Studies, 5 = Diagnosis or Clinical Impression, 6 = Treatment and Specific Nutritional Interventions, and 7 = Case Management. The following is a formulaic representation of the calculation:

where Importance_tc is the total importance for exam category c, c = 1, 2, …, 7, and Importance_ic is the importance index for item i = 1, 2, …, I_c, in category c.

The overall importance for the exam was calculated in the following way:

where Importance_TE is the importance index for the entire exam. Finally, the category percentage (weight) was calculated in the following way:

where weight_c is the percentage of category c on the exam.

Content Validity

The concept of content validity is a concern with the extent to which the content of items in a scale adequately represents the complete range or breadth of the construct under consideration.²⁰ The concept of content validity addresses the comprehensiveness and representativeness of an exam. Content validity is the representativeness of sampling adequacy of the content of a measuring instrument. It is not possible to draw a random sample of items from the hypothetical “universe” of content. Such a universe exists only theoretically; therefore, content validity is judgmental. Each item on a test must be studied, and each domain must be weighted for its presumed representativeness of the content universe.²¹ Lawshe²² developed a quantitative approach to content validation. In the original study, the participants were asked to rate each item on a 3-point scale, where each item is judged to be either essential, important but not essential, or unimportant. The number of essential responses was counted, and the content validity ratio (CVR) was computed in the following way:

where n_e is the number of panelists who rated the item as being essential and N is the overall number of panelists. The CVR index can range between −1 and +1 with values closer to 1 indicating better agreement.²³

For the purposes of current research, we calculated the CVR values for each domain of the CBCN exam. For this, we referred to the calculations previously made (see the section “Importance Index”) using n for items classified as “routinely” or “frequently” as n_e.

Lawshe²² had never reported the critical values for CVR and how they relate to various sample sizes. Ayre and Scally²⁴ evaluated critical values for CVR, producing a table of exact values including the minimum number of panel members required for the agreement to be above chance. They reported the critical CVR = .30 for a panel size of N = 40, the highest sample size considered in that study.

RESULTS

According to the survey, the specialty of chiropractic nutrition appears to remain a field dominated by white (92%) males (69%). The majority of the respondents indicated that they have been in practice for more than 15 years (88%), while 12% reported to be in practice for 5 to 15 years. The descriptive statistics associated with the demographic variables included in the study are presented in Table 1.

Case History (α = .95, CVR = .60)

It is noted that alpha is a reliability coefficient ranging from 0 to 1.0. and that the CVR is a coefficient of agreement between the judges on how essential a particular item or group of items is. Ratings of case history activities relating to the frequency and risk appear in Table 2. Chiropractic nutritionists reported routinely obtaining personal health history, obtaining a list of prescription and nonprescription drugs and supplements, reviewing the nervous system, reviewing the musculoskeletal system, obtaining a family health history, and inquiring about exercise habits, M_{Chiropractic History} = 4.4. The average risk involved to the patient's health or safety if the tasks were poorly performed was reported to be significant, M_{Risk Chiropractic History} = 3.3.

Table 2-.

Case History Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you obtain a personal health history?	4.9 (.4)	3.9 (.8)
How often do you obtain a list of prescription and nonprescription drugs and supplements?	4.9 (.4)	3.9 (.9)
How often do you review the nervous system?	4.8 (.4)	3.7 (.8)
How often do you review the musculoskeletal system?	4.8 (.5)	3.5 (.8)
How often do you obtain a family health history?	4.8 (.6)	3.4 (.8)
How often do you inquire about exercise habits?	4.8 (.4)	3.1 (.8)
How often do you inquire about the use of alcohol?	4.7 (.7)	3.7 (.9)
How often do you review alimentary and gastrointestinal systems?	4.7 (.5)	3.6 (.8)
How often do you inquire about the use of caffeine?	4.7 (.7)	2.9 (.8)
How often do you obtain vital signs?	4.6 (.8)	3.7 (.9)
How often do you review endocrine and metabolic systems?	4.6 (.6)	3.5 (.8)
How often do you review skin conditions and allergies?	4.6 (.7)	3.3 (.9)
How often do you inquire about any other specific diets used within the last 12 mo?	4.6 (.6)	3 (.8)
How often do you review the hematological/hematopoietic system?	4.5 (.8)	3.5 (.8)
How often do you utilize a health questionnaire?	4.5 (1)	3.5 (1)
How often do you review genitourinary and reproductive systems?	4.5 (.8)	3.4 (.7)
How often do you inquire about use of recreational/illicit drugs?	4.4 (.9)	3.7 (1)
How often do you review psychological /psychiatric/mental health status?	4.3 (.9)	3.4 (.8)
How often do you obtain a psychosocial history?	4.3 (1)	3.3 (.9)
How often do you review the respiratory system?	4.1 (1)	3.3 (.7)
How often do you review other problems or information not listed above?	4 (1)	3 (.9)
How often do you review a diet diary with patient?	4 (1)	2.9 (1)
How often do you obtain a 24-h diet recall?	3.7 (1.2)	2.7 (.9)
How often do you inquire about religious or ethnic dietary restrictions?	3.6 (1.3)	2.6 (.9)
How often do you obtain a 7-d diet diary?	3.5 (1.2)	2.8 (.9)
How often do you obtain a 3-d diet diary?	3.4 (1.2)	2.7 (.9)

Additionally, chiropractic nutritionists routinely inquire about the use of alcohol, review alimentary and gastrointestinal systems, inquire about the use of caffeine, obtain vital signs, review endocrine and metabolic systems, and review symptoms of skin conditions and allergies. Inquiring about religious or ethnic dietary restrictions and obtaining a 3-day or 7-day diet diary is sometimes performed by chiropractic nutritionists, and there is some risk to the health and safety of a patient if omitted or poorly performed.

Physical and Nutritional Examination (α = .90, CVR = .32)

Ratings of physical and nutritional activities relating to the frequency and risk are detailed in Table 3. Survey respondents reported occasional performance of all aspects of physical and nutritional examinations, M_{Physical and Nutritional Examination} = 3.6. Some risk was reported if the tasks were poorly performed, M_{Risk Physical and Nutritional Examination} = 3.0.

Table 3-.

Physical and Nutritional Examination Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you obtain basic anthropometric measurements (height, weight, body mass index)?	4.6 (.8)	3 (.9)
How often do you inspect hair, nails, skin, tongue, etc.?	4.2 (.9)	3.1 (.8)
How often do you perform an abdominal examination?	3.6 (1)	3.2 (.7)
How often do you perform a nutrition-focused orthopedic/neurologic examination?	3.6 (1.1)	3 (.9)
How often do you perform a cardiopulmonary examination?	3.4 (1.2)	3.2 (.9)
How often do you obtain advanced anthropometric measurements (body composition, waist-to-hip circumference ratio, etc.)?	3.4 (1.2)	2.7 (.9)
How often do you perform nutrition specific examination procedures (Ragland's test, Rogoff, Lowenberg's, etc.)?	3.3 (1.1)	2.7 (.8)
How often do you perform an ear, nose, and throat examination?	3.1 (1.1)	3.3 (.9)
How often do you obtain a basal temperature chart?	2.9 (1.2)	2.4 (.7)

Laboratory and Nutrition (α = .72, CVR = .53)

Respondents reported that they frequently obtain and interpret blood and urine laboratory test results and that they interpret hair, saliva, stool, or other laboratory tests. The overall lower category average of 3.1 reflects the low frequency of genetic testing and live cell analysis performed by chiropractic nutritionists. Ratings of laboratory and nutrition activities relating to the frequency and risk are presented in Table 4.

Table 4-.

Laboratory and Nutrition Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you obtain and interpret blood laboratory test results for nutritional implications?	4.4 (.8)	3.8 (.9)
How often do you obtain and interpret urine laboratory test results for nutritional implications?	3.9 (1.1)	3.3 (.8)
How often do you obtain and interpret hair, saliva, feces, or other laboratory test results for nutritional implications?	3.6 (1)	3.3 (.9)
How often do you obtain genetic testing?	2.2 (.9)	2.5 (.9)
How often do you obtain and interpret live cell analysis with dark-field microscopy?	1.3 (.6)	1.9 (1)

Chiropractic nutritionists reported occasional performance of all aspects of laboratory and nutrition, M_{Laboratory and Nutrition} = 3.1. There is some risk involved to the patient's health or safety if the assessment tasks are poorly perform, M_{Risk Laboratory and Nutrition} = 3.0.

Imaging and Other Special Studies (α = .69, CVR = .47)

Respondents reported they sometimes obtain and interpret diagnostic imaging studies (eg, bone density, x-ray, and computed tomography) to evaluate for nutritional implications and that there is significant risk to the patient's health or safety if the assessment task is poorly performed. Respondents sometimes obtain and interpret other specialized studies (eg, electrocardiogram and ultrasound) for nutritional implications, and there is significant risk to the patient's health or safety if the assessment task is poorly performed. Obtaining and interpreting an electrodermal analysis is rarely performed and has little risk to the patient's health or safety if poorly performed. Table 5 presents results of performance frequency and associated risk for imaging and other special studies activities.

Table 5-.

Imaging and Other Special Studies Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you obtain and interpret diagnostic imaging studies (eg, bone density, x-ray, computed tomography, etc.) to evaluate for nutritional implications?	3.4 (1)	3.3 (.8)
How often do you obtain and interpret other specialized studies (eg, electrocardiogram, ultrasound, etc.) for nutritional implications?	2.8 (1.1)	3.1 (.9)
How often do you obtain and interpret electrodermal analysis?	1.7 (.9)	2.1 (.9)

Survey participants reported rare in-office performance of imaging and other special studies, M_{Imaging and Other Special Studies} = 2.6, which may be due to poor cost-effectiveness. Some risk was reported to be associated with poor performance of imaging/other special studies tasks, M_{Risk Imaging and Other Special Studies} = 2.8.

Diagnosis or Clinical Impression (α = .78, CVR = .79)

Table 6 presents results for diagnostic or clinical impression. Chiropractic nutritionists reported that they routinely perform diagnosis or clinical impression, M_{Diagnosis or Clinical Impression} = 4.6, and that there is significant risk to the patient's health or safety if they poorly perform these assessment tasks, M_{Risk Diagnosis or Clinical Impression} = 3.8.

Table 6-.

Diagnosis or Clinical Impression Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you develop a differential diagnosis or clinical impression?	4.7 (.5)	3.8 (.9)
How often do you review possible interactions of nutrients, foods, drugs, and herbs associated with a patient's care?	4.5 (.8)	3.8 (.9)

Treatment and Specific Nutritional Interventions (α = .86, CVR = .43)

Treatment and specific nutritional intervention appear in Table 7. Respondents reported that they routinely recommend therapeutic levels of vitamins and minerals and frequently recommend or help develop special diets for various conditions (eg, diabetes, kidney disease, allergy elimination, gluten intolerance, and lactose intolerance). They frequently utilize nutritional detoxification protocols, specific amino acid therapy (eg, arginine, glutamine, and tryptophan), and phytochemical supplementation (eg, isoflavones and epicatechins). Furthermore, the respondents indicated utilization of weight loss management programs, glandular therapy, and supplementation for classic nutritional deficiency diseases (eg, iron, pernicious anemia, and rickets). Chiropractic nutritionists rarely manage anorexia and bulimia, but poor management is associated with significant risk.

Table 7-.

Treatment and Specific Nutritional Interventions Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you recommend therapeutic levels of vitamins and minerals?	4.5 (.7)	3.3 (1)
How often do you assess the existence of risk factors and contraindications to nutritional intervention?	4.4 (.8)	3.8 (.9)
How often do you recommend/help develop special diets (eg, diabetes, kidney disease, allergy elimination, gluten or lactose intolerance)?	3.9 (.9)	3.5 (1)
How often do you utilize nutritional detoxification protocols?	3.6 (1)	3.3 (.8)
How often do you utilize herbal therapy using American and European herbs (extracts, teas tablets, etc.)?	3.6 (1.1)	3.2 (.9)
How often do you use specific amino acid therapy (eg, arginine, glutamine, tryptophan)?	3.4 (.8)	3 (.8)
How often do you use phytochemical supplementation (eg, isoflavones, epicatechins)?	3.4 (1)	2.7 (.9)
How often do you utilize weight loss management programs?	3.3 (1.1)	3.2 (.9)
How often do you use glandular therapy?	3.2 (1.2)	3 (1)
How often do you recommend supplementation for classic nutritional deficiency diseases (eg, scurvy, beri, iron, pernicious anemia, Rickets)?	2.9 (1.1)	N/A
How often do you recommend/provide nutrition—focused patient education classes?	2.8 (1.2)	2.7 (1.2)
How often do you use hormone therapy (eg, growth, melatonin, progesterone)?	2.7 (1.1)	3.4 (.9)
How often do you use homeopathy?	2.6 (1)	2.5 (1.1)
How often do you utilize herbal therapies using traditional Chinese herbs?	2.4 (1)	2.9 (1)
How often do you utilize anorexia/bulimia management programs?	1.8 (.8)	3.7 (1.1)

On average, chiropractic nutritionists sometimes perform treatment and specific nutritional interventions, M_{Treatment and Specific Nutritional Interventions} = 3.2, and there is some risk to the patient's health or safety if they poorly perform an assessment task, M_{Risk Treatment and Specific Nutritional Interventions} = 3.2.

Case Management (α = .96, CVR = .46)

The averages for case management activities are presented in Table 8. Respondents reported that they frequently perform all aspects of case management, M_{Case Management} = 4.2. There is some risk involved to the patient's health or safety if the assessment tasks is poorly performed, M_{Risk Treatment and Specific Nutritional Interventions} = 3.3.

Table 8-.

Case Management Means for Activity Frequency and Risk

Item	Frequency Mean (SD)	Risk Mean (SD)
How often do you advise patients regarding nutritional/dietary habits?	4.8 (.5)	3.6 (1)
How often do you legibly document each patient contact?	4.8 (.6)	3.5 (1)
How often do you review with a patient his or her relevant history, examination findings, diagnosis, prognosis, and case management plan options?	4.8 (.5)	3.4 (1.1)
How often do you monitor a patient's progress or response to treatment utilizing follow-up?	4.7 (.6)	3.3 (.9)
How often do you develop a case management plan that includes nutritional and lifestyle changes?	4.6 (.6)	3.5 (.9)
How often do you provide instruction advice regarding activities daily living?	4.6 (.6)	3.3 (1)
How often do you counsel patients on proper sleep quantity and quality?	4.5 (.6)	3.3 (1)
How often do you advise patients to change risky or unhealthy behaviors?	4.4 (.8)	3.7 (.9)
How often do you advise or counsel patients regarding disease prevention and early detection?	4.4 (.7)	3.4 (.9)
How often do you create complete, legible documentation of a patient's disease history and examination findings as well as diagnosis, prognosis, and treatment plan?	4.4 (1)	3.4 (1)
How often do you develop a prognosis?	4.4 (1)	3.2 (1)
How often do you provide education or advice regarding health promotion and wellness care?	4.4 (.7)	3.1 (1)
How often do you reexamine a patient with physical examination procedures, either periodically or when the patient s condition materially changes?	4.3 (.8)	3.3 (.8)
How often do you monitor a patient's progress or response to treatment utilizing follow-up lab tests?	4.2 (.9)	3.5 (.9)
How often do you advise patients in proper physical fitness and exercise techniques?	4.2 (.7)	3.2 (.9)
How often do you monitor a patient's progress or response to treatment utilizing objective outcome measures (eg, questionnaires etc.)?	4.2 (.8)	3.2 (.9)
How often do you advise patients in proper ergonomics/posture?	4.1 (.9)	3 (.9)
How often do you advise patients regarding smoking cessation?	4 (1)	3.7 (1.1)
How often do you review genitourinary and reproductive histories?	4 (.9)	3.1 (.9)
How often do you provide instruction in label reading and food shopping?	4 (.9)	3 (.9)
How often do you provide advice regarding food preparation?	4 (.9)	2.9 (.8)
How often do you re-examine a patient with nutrition-focused orthopedic/neurologic examination procedures, either periodically or when the patient's condition materially changes?	3.9 (1.1)	3.3 (.8)
How often do you provide stress management guidance?	3.8 (1)	3.3 (.9)
How often do you monitor a patient's progress or response to treatment utilizing follow-up testing of body impedance analysis, functional tests, etc.?	3.8 (1)	3.1 (.8)
How often do you monitor a patient's progress or response to treatment by documenting physical changes in hair, skin, nails, etc.?	3.7 (1)	2.9 (.9)
How often do you provide patients with recommendations regarding their personal hygiene?	3.4 (1)	2.8 (.9)
How often do you refer a patient to a specialist for consultation or comanagement?	3.2 (.8)	3.6 (.9)
How often do you write a narrative report (not daily notes)?	2.6 (1)	2.4 (1)

Qualitative Panel and Test Plan Revisions

On July 2, 2019, a qualitative panel convened to review the survey results. The panel reviewed the quantitative methodology and suggested revisions to the test plan. The panel suggested reducing the number of domains on the nutrition exam from 7 to 6 by combining the Laboratory and Nutrition-Specific Testing and Imaging and Other Special Studies domains. Additionally, the panel decided on the final distribution of weights combining the quantitative results with qualitative perspectives. The final distribution of weights across domains on the CBCN exam is presented in Table 9.

Table 9-.

Revised Test Plan for the CBCN Exam

Category Name	Weight
Case History	20%
Physical and Nutritional Examination, including Anthropometrics and Neuromusculoskeletal	10%
Laboratory and Nutrition-Specific Testing; Imaging and Other Special Studies	15%
Diagnosis or Clinical Impression	20%
Treatment and Specific Nutritional Interventions	15%
Case Management	20%

DISCUSSION

This study detailed the practice responsibilities of chiropractic nutritionists, ascertaining the knowledge and skills required to effectively carry out the responsibilities of the practice. Among many factors contributing to the decision to readdress the content of an exam is the change in practice itself. An incongruity between the exam and the practice may pose a threat to content validity. Therefore, a time-to-time thorough review of the practice is strongly recommended.²

The present investigation established validity evidence for the content of the CBCN examination. The results of the practice analysis survey have provided insights into the frequencies and importance of professional tasks performed by chiropractic nutritionists. Using the importance index, weights were calculated for all domains in the CBCN test plan. The outcomes led to the reduction of the number of domains on the CBCN exam from 7 to 6 by combining Laboratory and Nutrition-Specific Testing and Imaging and Other Special Studies.

The calculation of within-domain CVR indices provided further evidence of domain-level validity. The reliability of the survey instrument was assessed by estimating internal consistency indices.²⁵

Chiropractic nutritionists responding on the practice analysis found the items listed on the survey to be representative of the activities routinely performed in their practice. Although minor adjustments were made to the weight distribution, in general, the analyses of the responses to the survey items supported the logic of the test plan.

The present validation efforts relied on a mixed-method methodology, which, is defined as the “research in which a team of researchers combines elements of quantitative and qualitative research approaches for the broad purposes of breadth and depth of understanding and corroboration.”²⁶ The combination of quantitative and qualitative results provides an opportunity to examine corroboration and convergence of findings across multiple data sources. Furthermore, the qualitative approach may help to augment statistical results derived by the quantitative part.²⁷

The quantitative elements in this study encompassed the statistical data analyses of the survey responses. The qualitative element was the panel of subject matter experts who reviewed the quantitative results and, utilizing their personal expertise, suggested the final distribution of the weights on the CBCN exam.

Limitations

Findings of this study should be considered in the context of potential limitations. Raymond²⁸ advocates that although practice analysis serves as the primary source of evidence supporting the validity of scores from licensure and certification exams, there is surprisingly little consensus in the measurement community regarding suitable methods for conducting job analyses and translating the results into test plans. Furthermore, Dierdorff and Wilson²⁹ conducted a meta-analysis of job analysis reliability using 46 various job analysis studies. The researchers could not reach a general conclusion, stating that “more research is clearly needed if a more accurate picture is to be drawn. Yet the investigators revealed that task data had higher interrater and intrarater reliabilities.

The practice analysis alone is not enough to fully support the content validity for a licensing exam. Therefore, it was imperative to conduct the qualitative panel. Knapp and Knapp⁷ explained that the practice analysis data play a key role in guiding decisions regarding the critical responsibilities, skills, and knowledge and that the consensus of the subject matter experts represents “the last word.” The subject matter experts provide a qualitative link between the practice analysis and the test plan, thus improving the content validity of the licensure examination.

From a statistical perspective, the findings may not reveal the complete picture due to the limited sample size. The nutrition specialty within the chiropractic profession includes fewer than 200 members. The NBCE reached out to all of them but was able to collect only 78 responses. We consider this the most significant limitation to the study.

The study utilized a survey as a methodology for data collection. Yet surveys are afflicted by many types of errors.³⁰ As most surveys are sampling instruments designed to estimate responses of larger populations, the quality of data collected by surveys is usually related to the survey sampling design. Thus, the central concern of survey designers is to provide sample estimates of a population that are as accurate as possible.²⁰

CONCLUSION

Practice analysis provides information on the core tasks and risks that are most critical for competent performance by a chiropractic nutritionist and provides a viable and defensible content validation of the licensure examination. This study signifies a first step in the definition of the skills required for practicing chiropractic nutritionists. The 2019 NBCE survey based on the practice analysis becomes one of the references and a decision-making tool for developing and administrating quality assessments. Based on evidence and rigorous methodology, this study can be used to evaluate and support the CBCN test plan.