ThyroidEx: Development and Preliminary Validation of a Thyroid Surgery Expectations Measure

Abstract

Objective.

To address the lack of validated patient-reported outcome (PRO) instruments that define and quantify patient expectations in thyroid cancer surgery, we developed and initially validated ThyroidEx, a novel disease-specific PRO instrument.

Study Design.

Survey study.

Setting.

Single-institution tertiary care cancer center.

Methods.

An expert panel drafted an initial set of thyroid cancer–specific concepts, which was used in semistructured concept elicitation interviews with patients with thyroid cancer 4 weeks before and 8 weeks after surgery. Candidate items were generated per patient responses and refined via cognitive interviewing and additional review by the expert panel. The draft ThyroidEx was then preoperatively administered to a separate cohort undergoing thyroid cancer surgery to establish a final item set and initial psychometric evidence.

Results.

Prospective concept elicitation interviews generated 358 patient-elicited concepts (n = 15 patients). These were then placed into 70 unique subcategories from which 41 items were generated for cognitive interviews with 20 patients preoperatively and 28 postoperatively. After expert panel review, ThyroidEx included 18 items across 2 scales (Expectations and Concerns), with an additional item about beliefs. In the preoperative cohort in phase 2 (n = 67), internal consistency Cronbach’s α values ranged from 0.81 to 0.89. Descriptive analysis showed significant differences between patients’ concerns and expectations and clinicians’ perceptions.

Conclusion.

Defining expectations represents an important modifier in the measurement of PROs. Preliminary validation of ThyroidEx revealed incongruent expectations between expert opinion and patients. Future development and implementation of ThyroidEx may affect preoperative consultation and the consent process.

Optimizing outcomes among patients with low-risk thyroid cancer hinges on the accurate assessment and management of preoperative patient expectations. The extent of surgery (lobectomy vs total thyroidectomy and/or with central neck) and surgical technique (neck scar vs scars outside of the neck) for thyroid cancer vary and commonly use patient satisfaction as an endpoint. Patient-reported outcome (PRO) instruments are considered the gold standard for capturing the subjective patient experience.^1–4 However, unrealistic patient expectations in the surgical management of thyroid cancer can result in patients dissatisfied with surgical outcomes, independent of the technical and oncologic success of the surgery.

The Affordable Care Act ushered in payment reform, which now includes quality outcomes as a metric for payment. In thyroid cancer surgery, the pretest probability of standard measures of quality—including hospital readmission, complications, recurrence, and death—is rare and difficult to use as a sensitive measurement of quality. PROs and patient satisfaction, both influenced by patient expectations, have therefore emerged as measures of quality. However, the integration of quality measured by PROs and patient satisfaction into payment models requires an understanding of the influence of expectations, which is limited by a lack of tools to measure expectations in patients with thyroid cancer. Although thyroid disease–specific PRO tools exist, none currently assess patient expectations.^5–8 The most important initial step for developing a PRO tool is establishing content validity in accordance with the rigorous methodological recommendations included in the 2009 US Food and Drug Administration’s “Guidance for Industry—Patient-Reported Outcomes Measures: Use in Medical Product Development to Support Labeling Claims.”⁹

Herein we describe the content validation and preliminary psychometric validation of ThyroidEx, a first-of-its-kind PRO instrument that we developed for the rigorous measurement of preoperative expectations in patients undergoing thyroid cancer surgery. The goal of ThyroidEx is to improve communication and education preoperatively and the interpretation of satisfaction and health-related quality of life (HRQoL) postoperatively.

Methods

Participants

Patients were eligible if they were adults (age ≥18 years) who were scheduled to undergo thyroid lobectomy or total thyroidectomy for thyroid cancer at Memorial Sloan Kettering Cancer Center (MSK), a National Cancer Institute–designated Comprehensive Cancer Center in New York, New York. Patients with planned neck dissection were excluded. Patients meeting eligibility criteria were invited by their head and neck surgical oncologist to enroll. The study’s clinical research coordinator approached eligible patients to obtain informed consent. Figure 1 outlines the development of ThyroidEx. An expert panel of MSK attending physicians and support staff (ie, head and neck surgeons, nurses, physician assistants, and nurse practitioners) was recruited to provide initial input on concepts related to thyroid surgery, as well as to complete a review and critique of the draft for ThyroidEx. Approval from the MSK Institutional Review Board, in accordance with an assurance filed with and approved by the US Department of Health and Human Services, was obtained prior to the study’s inception.

Figure 1.

Development of ThyroidEx.

Procedure

Phase 1: Establishment of Content Validity via Content Elicitation and Cognitive Interviewing.

Members of our expert panel were surveyed via email to provide perceived concerns and expectations of patients undergoing thyroid surgery based on their clinical experience and training. Concepts identified through this survey were then used by the study’s qualitative methodology specialist (QMS) to develop an interview guide for one-on-one concept elicitation interviews. Concept elicitation is an exploratory qualitative method that allows participants to spontaneously describe their expectations and concerns related to surgery without the use of formal prompts, followed by asking them whether they recognize concepts from the expert-derived list as being important to their surgical-related expectations and concerns. This method is considered best practice in the initial establishment of content validity.¹⁰ Semistructured interviews focused on expectations and concerns were conducted within 4 weeks before and 8 weeks after thyroid surgery to determine if patients’ postoperative experience was congruent with preoperative concerns and expectations.

A data analysis team consisting of the clinical research coordinator and QMS analyzed the interview data using established inductive thematic text analysis methods and employing ATLAS.ti, a qualitative data analysis software program.¹¹ Candidate items were generated by the sorted concepts, which resulted in a draft instrument, ThyroidEx. To refine and establish content validity, the draft instrument was pre- and postoperatively administered to a separate cohort of patients via the rigorous process of cognitive interviewing. Cognitive interviewing is an iterative process that allows patients to review a set of items and identify issues with comprehension, missing content, and other problematic areas. This qualitative method is considered to be essential in the confirmation of a PRO tool’s content validity.⁹ Scripts to guide interviews were developed by the study’s QMS based on best practices in cognitive interviewing.¹²

Phase 2: Preliminary Psychometric Validation and Item Reduction.

Preliminary field testing of the draft items was conducted in an additional cohort of patients. Participants in this phase of the study completed ThyroidEx preoperatively in an electronic format at the time of their consent visit in clinic. Demographics and clinical characteristics (eg, age, sex, surgery type) were also collected to characterize the sample. Consultation with the expert panel was used to create the final set of ThyroidEx items and subscales. We computed internal consistency using Cronbach’s α for each subscale. Cronbach’s α values >0.70 are an indicator of good internal consistency.¹³

Comparative Analysis

Using the preliminary field-testing cohort responses, we descriptively analyzed patient responses and compared these with the clinician responses. Clinicians (surgeons, n = 11; endocrinologists, n = 5; trainees, n = 5) were electronically administered ThyroidEx and asked to complete it based on their “perceptions regarding the beliefs, expectations, and concerns of your patients.” The clinician and patient responses were compared and analyzed with Fisher’s exact test.

Results

A total of 15 participants completed pre- and postoperative concept elicitation interviews. Across all interviews, patients identified 358 concepts representing concerns and/or expectations. Sample concerns and expectations are listed in Table 1. On the basis of structural coding, these concepts were placed into 70 unique subcategories (eg, concerns about voice quality, expected need to adjust to new metabolism) and were sorted by using the a priori thematic coding scheme (eg, long-term postoperative concerns, overall expectations regarding surgery). Candidate items were subsequently generated by the sorted concepts, resulting in 41 items in 11 subcategories for concerns and 22 items in the 8 subcategories for expectations. Cognitive interviews were then completed with 20 patients preoperatively and 28 different patients postoperatively. Item content was generally understood, with item reduction based on patient input and expert review, resulting in a finalized ThyroidEx that included 2 scales (Expectations and Concerns) and 18 items, with an additional item about beliefs. For each symptom, patients rate expectations on a 4-point Likert scale and concerns on a 5-point Likert scale.

Table 1.

Selected Concerns and Expectations: Phase 1—Concept Elicitation.

Selected concerns	Selected expectations
Involvement of lymph nodes	Lymph nodes will be removed during surgery
Possible removal of parathyroid glands	Temporary dysfunction to parathyroid glands
Surgery may affect vocal cords	Very little pain
Length of stay in hospital	A lot of numbness
Pain will be worse than described	Quick recovery after surgery
Nausea for anesthesia	Will be in the hospital overnight
Scar will become infected	Voice will change temporarily after surgery
Care of surgical area	Will not be allowed to lift anything heavy
Bleeding in the surgical area	Will be unable to exercise
Stitches will break	Will be energetic as before surgery
Voice will change after surgery	Will be able to return to usual activities
Voice will become hoarse after surgery	Will be on thyroid medication for the rest of life
Long-term functioning will not be the same	Doctor may need to adjust medication dosage
May not feel the same as before surgery	Will take time to adjust to metabolism after surgery
Will affect my everyday life	Will experience mood changes while adjusting to dosage
	Will need to establish routine to take medication

For phase 2, ThyroidEx was administered preoperatively to 67 patients with planned thyroid surgery. Patient characteristics are listed in Table 2. For expectations and concerns after thyroid surgery Cronbach’s α values were 0.81 and 0.89, respectively (n = 67), indicating good internal consistency for both scales. Descriptive analysis of patient responses and clinicians was done to assess for clinical trends and generate hypotheses (Table 3). For beliefs, there was no significant difference between patient responses and clinician perceptions. However, for expectations and concerns, there were significant differences between patient responses and clinician perceptions for items related to pain (P = .003), “time for me to get adjusted to my metabolism” (P = .001), bleeding (P = .01), scar (P = .01), side effects (P = .03), feeling the “same” after surgery (P = .03), and lifelong medication (P = .001). Figure 2 illustrates patient and clinician (trainee, endocrinologist, surgeon) responses as measured by frequency percentage for items that demonstrate significant differences.

Table 2.

Patient Characteristics (n = 67).

	Mean ± SD or No. (%)
Age, y	43.9 ± 14.5
Sex
Female	46 (68.7)
Male	21 (31.3)
Type of surgery
Thyroidectomy total	24 (35.8)
Thyroidectomy total, cervical lymphadenectomy	18 (26.8)
Thyroid lobectomy	20 (29.9)
Thyroid lobectomy, cervical lymphadenectomy	4 (6)
Isthmusectomy	1 (1.5)

Table 3.

Participant Responses to ThyroidEx (n = 67).

	No. (%)
Beliefs about thyroid cancer
Thyroid cancer is one of the most curable cancers.
Disagree	0 (0.0)
Somewhat agree	8 (11.9)
Agree	53 (79.1)
Don’t know	6 (9.0)
Thyroid surgery expectations
I expect that there will be temporary dysfunction to my parathyroid glands.
Definitely will not	2 (3.0)
Probably will not	27 (40.3)
Probably will	35 (52.2)
Definitely will	3 (4.5)
I expect that I will be self-conscious of the scar.
Definitely will not	21 (31.3)
Probably will not	17 (25.4)
Probably will	19 (28.4)
Definitely will	10 (15.0)
I expect that my voice will change temporarily after thyroid surgery.
Definitely will not	1 (1.5)
Probably will not	29 (43.3)
Probably will	34 (50.7)
Definitely will	3 (4.5)
I expect that I will experience a lot of numbness after thyroid surgery.
Definitely will not	5 (7.5)
Probably will not	30 (44.8)
Probably will	28 (41.8)
Definitely will	4 (6.0)
I expect that I will experience a lot of pain after thyroid surgery.
Definitely will not	3 (4.5)
Probably will not	35 (52.2)
Probably will	27 (40.3)
Definitely will	2 (3.0)
I expect that I will experience a lot of fatigue after thyroid surgery.
Definitely will not	7 (10.4)
Probably will not	37 (55.2)
Probably will	19 (28.4)
Definitely will	4 (6.0)
I expect that it will take time for me to get adjusted to my metabolism after thyroid surgery.
Definitely will not	0 (0.0)
Probably will not	8 (11.9)
Probably will	38 (56.7)
Definitely will	21 (31.3)
I expect that I will experience some changes to my mood while I am adjusting to the dosage of the medication.
Definitely will not	0 (0.0)
Probably will not	26 (38.8)
Probably will	33 (49.3)
Definitely will	8 (11.9)
I expect that I will need to establish a routine to take the thyroid medication.
Definitely will not	1 (1.5)
Probably will not	4 (6.0)
Probably will	31 (46.2)
Definitely will	31 (46.2)
Thyroid surgery concerns
I am concerned about the length of time I will stay in the hospital.
Not at all	41 (61.2)
A little	12 (17.9)
Moderately	9 (13.4)
A lot	5 (7.5)
Extremely	0 (0.0)
I am concerned that my pain will be worse than what has been described to me.
Not at all	29 (43.3)
A little	17 (25.4)
Moderately	15 (22.4)
A lot	3 (4.5)
Extremely	3 (4.5)
I am concerned that I will have bleeding.
Not at all	33 (49.3)
A little	18 (26.9)
Moderately	9 (13.4)
A lot	4 (6.0)
Extremely	3 (4.5)
I am concerned about the scar that I will get from the surgery.
Not at all	27 (40.3)
A little	21 (31.3)
Moderately	9 (13.4)
A lot	3 (4.5)
Extremely	7 (10.4)
I am concerned people at work or school will look at me differently after thyroid surgery.
Not at all	42 (62.7)
A little	14 (20.9)
Moderately	6 (9.0)
A lot	4 (6.0)
Extremely	1 (1.5)
I am concerned about the side effects of the thyroid medication.
Not at all	20 (29.9)
A little	14 (20.9)
Moderately	14 (20.9)
A lot	10 (14.9)
Extremely	9 (13.4)
I am concerned whether I will feel the same as I did before my thyroid was removed.
Not at all	18 (26.9)
A little	23 (34.3)
Moderately	10 (14.9)
A lot	11 (16.4)
Extremely	5 (7.5)
I am concerned that my cancer will come back.
Not at all	12 (17.9)
A little	20 (29.9)
Moderately	14 (20.9)
A lot	11 (16.4)
Extremely	10 (14.9)
I am concerned about being on thyroid medication for the rest of my life.
Not at all	22 (32.8)
A little	21 (31.3)
Moderately	9 (13.4)
A lot	8 (11.9)
Extremely	7 (10.4)

Figure 2.

Patient and clinician responses.

Discussion

Good outcomes are the goal of patient care, represent the numerator in value-based health care equations, and can be strongly influenced by patient expectations.¹⁴ We report the content validation and preliminary development of a first-of-its-kind PRO instrument to measure the expectations and concerns of patients undergoing surgery for low-risk thyroid cancer. ThyroidEx consists of 2 independently functioning scales with 18 items that were rigorously developed and validated following extensive qualitative patient interviews and expert input.

The incidence of thyroid cancer has about tripled in 50 years, largely related to overdiagnosis of incidentally found thyroid cancers in asymptomatic patients.¹⁵ The majority of patients with thyroid cancer are treated with surgery, which is most commonly performed through a neck incision. Surgery for thyroid cancer can result in lifelong thyroid hormone supplementation or replacement and carries a risk of hypocalcemia and voice change. Preoperative consultation and consent are centered on reviewing the sequela of surgery, assessing the risks and benefits, and setting expectations in often asymptomatic patients. Although complications (vocal cord paralysis) are frequently focused on, expected outcomes of surgery, such as lifelong thyroid replacement hormone after total thyroidectomy (ThyroidEx, item 19), may be less anticipated. Alternative options to thyroid cancer surgery are also reviewed at the time of preoperative consultation, including active surveillance, nonsurgical procedures (radio-frequency ablation), lobectomy, and total thyroidectomy via neck versus remote access options.^16,17 These are often compared by using satisfaction and HRQoL as endpoints in support of a specific approach.^18,19 Patients may present with preexisting expectations from external sources, including the internet, family, and friends. Mancuso and colleagues’ work on expectations among patients undergoing orthopedic surgery procedures illustrates the fluidity of expectations and how they can be modified with focused teaching modules.^20–22 Using validated PRO instruments to measure expectations in a couple of randomized controlled trials of hip and knee arthroplasties, the authors showed that expectation-focused preoperative teaching modules result in a closer alignment of patients’ and surgeons’ expectations.^21,22 Expectations have been measured in oncology,²³ supporting a role in this field, but are limited by validated instruments with rigorous qualitative development.

A rigorous evaluation of patient expectations allows for the optimization of shared decision making during preoperative consultation. Measuring how effectively preoperative consultation manages expectations in patients undergoing surgery for thyroid cancer has been limited by a lack of tools. In a small sample, our work demonstrates significant differences between patients’ expectations and concerns and clinicians’ perceptions of the expectations and concerns. Once additional psychometric validation is completed (ie, construct validity, test-retest reliability, establishment of clinically meaningful score changes), ThyroidEx will enable more robust study and optimization of preoperative consultation and consent and will allow for rigorous comparisons among thyroid cancer treatments.

Alternative payment models, such as the Medicare Merit-Based Incentive Payment System and bundled payment programs, use PROs and satisfaction as measures of quality and value. As pay-for-performance initiatives advance, an understanding of influences on these measures will be essential to compete in the marketplace. Satisfaction and HRQoL are influenced by expectations.¹⁴ In a systematic review on the relationship of patient expectations and PROs in surgery,¹⁴ Waljee et al found that identified fulfillment of expectations was associated with greater satisfaction in 18 of 36 studies that measured satisfaction; level of expectations correlated to postoperative satisfaction in 9 studies. De Groot and colleagues’ work in lumbar surgery suggests that patients who expect more pain, longer recovery times, and delayed return to work (items also included in ThyroidEx) are more likely to be less satisfied with surgery,²⁴ which would support that preoperative expectations should be considered a “modifier” when using satisfaction and HRQoL as outcome measures. Although Waljee et al reported that fulfillment of expectations correlated with improved PROs in 24 (40%) of the 60 studies in their systematic review, there was no correlation in 20%.¹⁴ The authors concluded that a contributing factor is a lack of robust PRO tools that measure expectations, underscoring the need for the development of ThyroidEx.

There are some limitations to our study. First, our sample was primarily cross-sectional, but phase 1 included concept elicitation interviews in a single population within 4 weeks preoperatively and 8 weeks postoperatively. Future research will examine differential item functioning (eg, age and sex). Second, the test population was heterogenous, representing different cancer types and surgical treatment options (thyroid lobectomy and total thyroidectomy). Although this was done to incorporate a spectrum of patients and treatment options, the sample was too small to perform subgroup analyses comparing the different surgical approaches. Finally, we have provided preliminary psychometric data and acknowledge that additional construct validity, test-retest reliability, and item responsiveness testing are necessary to define the underlying psychometric properties for ThyroidEx. However, we have rigorously demonstrated content validity of this tool using state-of-the-art patient-centered qualitative methodology and believe that this tool is ready for deployment in clinical practice.

Additional research with ThyroidEx, including further validation, is needed to assess how preoperative expectations influence satisfaction and HRQoL in patients treated for thyroid cancer, as well as how expectations should modify quality measures in value-based payment models. The eventual implementation of ThyroidEx in clinical care and clinical trials may also allow for the accurate assessment of novel approaches to surgery and may improve preoperative education and shared decision making.