The Patient Perspective on Patient-Reported Outcome Measures Following Elective Hand Surgery: A Convergent Mixed-Methods Analysis

Abstract

Purpose

Patient-reported outcome measures (PROMs) have traditionally been used for research purposes, but are now being used to evaluate outcomes from the patient’s perspective and inform ongoing management and quality of care. We used quantitative and qualitative approaches to evaluate the short-version Disabilities of the Arm, Shoulder, and Hand (QuickDASH) and the Patient-Specific Functional Scale (PSFS) with regard to patient preference and measurement of patient goals and their responsiveness after treatment.

Methods

Patients 18 years or older undergoing elective hand surgery received the QuickDASH and PSFS questionnaires before and at 6 weeks after surgery. Two additional questions intended to elicit patients’ preferences regarding the QuickDASH and PSFS were included. Responsiveness was measured by change in pre- to postoperative score. We analyzed patients’ responses to the 2 additional questions to identify themes in PROM preferences. Results from the quantitative and qualitative analyses were combined into a convergent mixed-methods (eg, quantitative and qualitative) analysis.

Results

Thirty-eight patients completed preoperative questionnaires; 25 (66%) completed postoperative questionnaires. Seventeen patients (77%) preferred the PSFS, 3 (14%) had no preference, 2 (9%) preferred the QuickDASH. The average change from pre- to postoperative QuickDASH was −10 (SD, 20), and that of the PSFS was −27 (SD, 26). Ten patients (40%) reported QuickDASH score changes above the minimal clinically importance difference (MCID), 17 patients (68%) reported PSFS score changes above the MCID. Content analysis revealed 4 themes in preference for a PROM: instrument simplicity (ease of instrument understanding and completion), personalized assessment (individualization and relevance), goal directed (having measurable aims or objectives), distinct items (concrete or specific instrument items or functions).

Conclusions

Most patients felt the PSFS better measured their goals because it is a simple, personalized instrument with distinct domains.

Clinical relevance

Whereas standardized PROMs may better compare across populations, physicians, or conditions, employing PROMs that address patient-specific goals may better assess aspects of care most important to patients. A combination of these 2 types of PROMs can be used to assess outcomes and inform quality of care.

AS HEALTH CARE SYSTEMS MOVE toward delivering value-based care, there is growing enthusiasm for the measurement of outcomes from the patient perspective. Patient-reported outcome measures (PROMs) are designed to evaluate aspects of health from the patient’s perspective, in contrast to physician-measured or objective outcomes (eg, range of motion). Although PROMs have traditionally been used to assess outcomes for research, they are increasingly used as part of health care strategy to inform ongoing care, measure outcomes from the patient perspective, and also for reimbursement by insurance carriers.^1–3 For example, it is possible that some well-validated PROMs that demonstrate stable psychometric properties may be appropriate to assess patient outcomes for research, but may not be understandable or actionable for use at point of care to inform management decisions to improve quality of care. As such, understanding how patients perceive PROMs and their perspective on which best reflect their own experience and symptoms can improve our understanding of which PROMs may be better for point of care use (eg, those deemed relevant and actionable by the patient and physician).⁴

Patient-reported outcome measure designs vary by domain and by what is measured. Several validated PROMs are used in the field of hand and upper extremity surgery, and although the majority of standardized PROMs ask about the ability to complete functional tasks, such tasks may not be important to certain patients. For example, a recent qualitative study found that patients define their goals for treatment based on specific functional tasks (eg, yoga, hiking) that may not be queried on a standardized PROM.⁵ Indeed, just because information from PROMs is “reported by the patient” does not mean it is important or valued by the patient.

Patient-centered outcome measures (PCOMs), a new concept in outcome measurement, are outcome measures that align with the values of the patient and measure the impact of a disease and/or treatment based on what is important to the individual patient.^6–8 Patient-centered outcome measures aim to translate an outcome score into an interpretable measure that is important to the patient and actionable. In contrast to the short-version Disabilities of the Arm, Shoulder, and Hand (QuickDASH) (a commonly used hand and upper extremity outcome measurement tool), the Patient-Specific Functional Scale (PSFS), which has been validated for patients with upper extremity musculoskeletal problems,⁹ allows patients to identify functional goals important to him or her in an open-ended manner. It is currently unknown how each instrument assesses aspects of care that align with the individual patient’s values and preferences while also assessing the patient outcome as a whole (eg, a PROM may measure 1 part of a patient’s outcome as a whole).¹⁰ Although the QuickDASH allows for comparison across diagnoses and patients and has been validated in multiple conditions, it may be limited for use at point of care because some questions may not be relevant or important to all patients. Although a relatively new and less-utilized entity, prior research has shown that several tasks frequently listed on the PSFS are not present on the QuickDASH including fastening buttons, nail trimming, and holding a book.⁵ Eppler et al,¹¹ using a qualitative approach to better understand quality from the patient perspective, also noted that many functional goals described by patients were specific and may be difficult to capture using standardized PROMs.

Although the psychometric properties of the QuickDASH and the PSFS have been studied at length,^5,9,12,13 no studies have evaluated which PROM patients feel more appropriately measures their goals in treatment—which prior work has shown can aid in patient communication and involvement in care when used at point of care. As the health care system moves toward an approach that encourages patient-centered and goal-directed care, it is increasingly important to recognize individual patient goals and ensure these goals are reflected in outcome measurement instruments. In this pilot study, we aimed to evaluate patient perspectives of the QuickDASH and PSFS through a mixed-methods (quantitative and qualitative) approach by evaluating which instrument patients prefer and the responsiveness to change of each instrument.

METHODS

Patient selection

After obtaining institutional review board approval, we recruited patients undergoing elective hand and upper extremity surgery in the preoperative holding area on the day of their surgery. Patients of 1 hand and upper extremity–trained surgeon (R.N.K.) were enrolled by a research assistant via consecutive sampling between March 2018 and May 2018. Inclusion criteria were age equal to or greater than 18, elective surgery, and English literacy. We excluded those having surgery for trauma and patients that would be immobilized for greater than 4 weeks after surgery. We chose immobilization time of less than 4 weeks to ensure patients had an opportunity to realize a functional improvement by their 6-week follow-up visit. As an initial pilot investigation, we aimed to enroll at least 35 patients. Accounting for 20% loss to follow-up, we intended to have at least 25 patients with completed questionnaires, based on prior work achieving saturation in qualitative analysis.^14–17

Data collection

After consent, patients were given the QuickDASH and the PSFS questionnaires prior to undergoing surgery. The QuickDASH is an 11-item questionnaire that was adapted from the 30-item DASH as a region-specific measure of disability and symptoms in patients with upper extremity disorders.^18,19 The QuickDASH asks patients questions about their ability to conduct 6 specific, predetermined tasks, their limitations in work and social activities, and their level of pain, tingling, and interferences with sleep. The QuickDASH is scored from 0 to 100, 0 indicating no disability and 100 indicating extreme disability. The PSFS asks patients to identify between 3 and 5 activities with which they have difficulty because of their hand or upper extremity condition and rate them on a scale from 1 (unable to perform activity) to 10 (able to perform activity at same level as before injury or problem). The scores for each activity are averaged to produce a score from 1 to 10 (1 indicating extreme disability and 10 indicating no disability). Patients were subsequently given the same questionnaires at their second follow-up visit, at least 6 weeks from the date of their surgery. The PSFS was given prior to the QuickDASH to prevent bias (so that the standardized items provided on the QuickDASH did not affect the open-ended items patients provided on the PSFS). After completion of the questionnaires, they were asked 2 follow-up questions: (1) Which questionnaire (the QuickDASH or the PSFS) better addressed your goals during this clinical experience? and (2) Please explain why you chose that questionnaire.

Quantitative data analysis

Patient demographics were recorded, including the procedure performed. The PSFS score was standardized (so that a higher score represented greater disability)²⁰ and normalized to match the QuickDASH (so that each score was out of 100)^21–23 for analysis. Pre- and postsurgical QuickDASH and PSFS scores were calculated. Descriptive statistics (mean and SD) of the change in score for each questionnaire were recorded. The frequency and percent of questionnaire preference was recorded. Change in score was used as a proxy for responsiveness. We calculated the frequency and percent of patient changes above the minimal clinical importance difference (MCID) for each questionnaire based on prior work (eg, proportions of patients that reach the MCID threshold). Because MCID refers to the smallest difference between scores that a patient can detect to be beneficial or harmful, we used MCID thresholds for responsiveness of the instrument. An MCID of 14 and 25 were used for the QuickDASH and the PSFS, respectively, based upon previous literature.^9,24–31

Qualitative data analysis

We analyzed the patients’ answers to the open-ended questions regarding why they chose a specific questionnaire using qualitative content analysis.^32–34 The authors followed the consolidated criteria for reporting qualitative research (COREQ) (Appendix A; available on the Journal’s Web site at www.jhandsurg.org),³⁵ a 32-item checklist to ensure study rigor and aid in interpretation. Two members of the research team (R.N.K., L.M.S.) independently analyzed the open-ended responses to identify themes. Three phases of coding were conducted: open coding, selective coding, and development of themes for answers to the question, “Please explain why you chose that questionnaire”. During open coding, the 2 coders reviewed patient responses, took notes on these responses, and identified concepts of interest that they labeled with subcodes. Reviewers met after coding 5 responses to create an agreed-upon codebook and then completed open coding of all responses. The codebook was applied to all of the patient responses, new codes were included until there were no further new codes identified (saturation), and the final codebook was reapplied to all transcripts. Discrepancies in coding were resolved by discussion to consensus.

Convergent data analysis

Results from the quantitative and qualitative analyses were integrated and arrayed side-by-side with a convergent mixed-methods analysis in accordance with prior investigations.^36–38 The integration involved merging the results from the quantitative and qualitative data so that a comparison could be made and a more complete understanding could emerge than what was provided by the quantitative or the qualitative results alone.

RESULTS

Thirty-eight patients completed the preoperative questionnaires (no patients declined to participate). Twenty-five patients (66%) completed the postoperative questionnaires. Patient demographics, procedure type, and number of procedures are noted in Table 1.

TABLE 1.

Demographics, procedure type, and number of procedures^*

Characteristic	Count, n (%)
Total	25
Mean Age, y (SD)	54.8 (18.9)
Sex, n (%)
Male	10 (40)
Female	15 (60)
Annual household income, n (%)
<$49,000	1 (7.7)
$50,000–$99,999	4 (30.8)
$100,000–$149,999	0 (0)
$150,000–$199,999	3 (23.1)
$200,000–$249,999	2 (15.4)
>$250,000	3 (23.1)
Race/ethnicity, n (%)
White/Caucasian	17 (73.9)
Black or African American	1 (4.3)
American Indian or Alaska Native	0 (0)
Asian	1 (4.3)
Hispanic	3 (13.0)
Native Hawaiian or other Pacific Islander	0 (0)
Other	1 (4.3)
Employment status, n (%)
Full-time	7 (41.2)
Part-time	1 (5.9)
Retired	6 (35.3)
No work outside the home	0 (0)
Disabled	1 (5.9)
Unemployed	1 (5.9)
Student	1 (5.9)
Highest level of education, n (%)
Elementary school	0 (0)
High school	2 (12.5)
2-Year college	3 (18.8)
4-Year college	5 (31.3)
Postcollege/graduate	6 (37.5)
Relationship status, n (%)
Married	14 (82.4)
Domestic partnership	0 (0)
Single, never married	1 (5.9)
Single, divorced or separated	1 (5.9)
Single, widowed	1 (5.9)
Primary insurance type, n (%)
Medicaid/Medi-Cal	1 (5.6)
Medicare	8 (44.4)
Military	0 (0)
Privately insured	9 (50)
San Mateo County Health Insurance	0 (0)
Uninsured	0 (0)
Procedure, n (%)
Carpal tunnel release	9 (36)
Trigger finger release	7 (28)
DeQuervain release	2 (8)
Other	7 (28)
Number of procedures, n (%)
1	23 (92)
2	2 (8)
3 or more	0 (0)

^*Some patients did not fully complete the demographics questionnaire; therefore, not all raw numbers add up to 25.

Quantitative results

When asked which PROM they preferred, 17 patients (77%) preferred the PSFS, 3 patients (14%) had no preference, and 2 patients (9%) preferred the QuickDASH. The average change in score from pre- to postoperative QuickDASH was −10 (SD, 20) and that of the PSFS was −27 (SD, 26) (Table 2).

TABLE 2.

PROM Scores

Measurement Instrument	Preoperative Score Mean (SD)	Postoperative Score Mean (SD)	Mean Change in Score (SD)
QuickDASH	43.8 (23.8)	33.5 (19.7)	−10.26 (20.4)*
PSFS	67.4 (17.7)	40.2 (20.3)	−27.24 (26.2)*

^*Indicates significant difference (P < .05).

Ten patients (40%) reported a change in QuickDASH above the MCID, 17 (68%) reported a change in PSFS above the MCID. Among 15 patients who did not demonstrate QuickDASH score changes above the MCID, 9 (60%) demonstrated changes greater than the MCID of their PSFS score. Among 6 patients who did not demonstrate changes greater than the MCID on their PSFS questionnaire, 1 demonstrated a change in the QuickDASH greater than the MCID. Table 3 demonstrates scenarios in which the MCID of the QuickDASH and PSFS vary by calculation method and patient population.

TABLE 3.

Results Displayed as Number and Percentage of Patients Above MCID for the QuickDASH and PSFS, Respectively

	QuickDASH MCID
		6.830	829	1128	1424
PSFS MCID	2031	15 (60) | 20 (80)	14 (56) | 20 (80)	13 (52) | 20 (80)	10 (40) | 20 (80)
	2531	15 (60) | 17 (68)	14 (56) | 17 (68)	13 (52) | 17 (68)	10 (40) | 17 (68)
	299	15 (60) | 11 (44)	14 (56) | 11 (44)	13 (52) | 11 (44)	10 (40) | 11 (44)
	3031	15 (60) | 11 (44)	14 (56) | 11 (44)	13 (52) | 11 (44)	10 (40) | 11 (44)

Qualitative results

Representative responses as to why patients preferred a specific instrument are listed in Figure 1. Analysis of the open-ended responses revealed 4 themes: (1) instrument simplicity, defined as the ease of understanding and completion of the instrument; (2) personalized assessment, defined as the individualization and relevance of an instrument to a specific patient; (3) goal-directed, defined as having a measurable aim or objective that a patient wishes to achieve; and (4) distinct items, defined as concrete or specific tasks or functions that are recognizably different from other tasks or functions.

FIGURE 1:

Illustration of the convergent mixed-methods analysis allowing integration of quantitative and qualitative data.

DISCUSSION

Patient-reported outcome measures are transitioning from being used solely for research purposes toward informing care decisions to improve quality of care. As such, we sought to understand the patient perspective on 2 PROMs that measure different constructs (eg, functional goals vs upper extremity disability) to assess whether one construct and instrument was preferred by patients. Our quantitative results suggest that patients believe the PSFS addresses their goals for hand surgery more frequently than the QuickDASH. Based on responses to open-ended questions, we also identified why patients believe the PSFS better addresses their goals and potentially why it can be more useful as a PROM for use and communication at point of care: it is a simple, personalized, and goal-directed instrument with distinct domains. Whereas traditional PROMs such as the DASH/QuickDASH can be important for assessing outcomes as a whole (eg, disability of the arm), their results may not be understandable and actionable for individual patients. As such, using instruments like the PSFS that allow patients to track their own progress toward their functional goals may be more actionable in improving quality of care and may be able to be used in conjunction with those that assist in comparisons across physicians or conditions or in clinical research efforts.

Importantly, we are not advocating for the sole use of the PSFS to evaluate outcomes, but instead, that outcomes assessment may be enhanced with the combined use of instruments to assess multiple facets of a patient’s outcome. The PSFS (and other non-standardized PROMs) do have limitations, some of which are mitigated by the concurrent use of a standardized PROM. Given that the PSFS asks patients to identify activities that they have difficulty performing, it may miss important patient symptoms like pain and numbness that are related to only certain activities. Additionally, the PSFS is less well studied with regard to its psychometric properties (eg, MCID, validity) and the populations/conditions in which it has been studied (compared with legacy instruments like the QuickDASH). Also, important to note is that evaluating change over time and comparisons between patients for each instrument differs—the QuickDASH outcome is a score based upon patients answering the same questions, whereas the PSFS outcome is a score based upon patient-selected and, therefore, varying goals. In addition, individualized and patient-specific PROMs are a relatively new entity and the barriers to their use, integration into clinical practice, and advantages/disadvantages war-rant further study.^39–41

In a value-based delivery model in which physicians are reimbursed based upon outcomes and cost of care, understanding how outcomes are defined is critical. Quality of care can be measured from various perspectives. The importance of the patient perspective has led to the use of PROMs to assess quality in hand and orthopedic surgery. For example, the Centers for Medicare and Medicaid services is collecting data on quality measures using PROMs (Hip disability and Osteoarthritis Outcome Score [HOOS] and Knee Injury and Osteoarthritis Outcome Score [KOOS]) for total joint arthroplasty through Patient-Reported Outcome Performance Measures (PRO-PMs).¹ Private payers have also begun implementing similar payment models.^2,3 Our results suggest that this area requires further analyses prior to implementation of PRO-PMs by payers.

To our knowledge, no studies have sought to evaluate which outcome measures patients feel better addresses their goals in undergoing hand surgery. Prior studies have evaluated and established the psychometric properties of the PSFS for various hand and upper extremity conditions.^9,12,13,42 McMillan and Binhammer¹² evaluated the responsiveness of the DASH, Michigan Hand Outcomes Questionnaire (MHQ), and PSFS for various hand and wrist conditions and found the MHQ to be the most responsive at 6 months after surgery for those with carpal tunnel syndrome and the PSFS to be the most responsive for finger contracture. It is important to note that psychometric properties of various PROMs vary by patient population and condition studied.^43,44 In addition, responsiveness assumes that constructs being measured are of equal weight to each individual patient. For example, when outcomes are measured based upon domains insignificant or irrelevant to the patient, a measure may be highly responsive and yet not important to the patient.

Our results should be viewed within their limitations. We used MCID thresholds as a proxy for minimum levels of responsiveness. The MCID varies by patient population and calculation method and should be interpreted with caution.^43–45 The authors chose an MCID of 25 and 14 for the PSFS^9,25–27 and QuickDASH,^24–31,46 respectively; based upon prior literature however, the pathologies and patients from which these numbers were derived are different from those in this study. In addition, there is variation in the exact calculation method (eg, anchor-, distribution-, Delphi-based methods) used in the literature, each having their own advantages and disadvantages. Table 3 demonstrates scenarios in which MCIDs of the QuickDASH and PSFS vary by calculation method and patient population. Indeed, the variation in MCIDs in the published literature for various PROMs may, in part, reflect the variation in importance patients place on the items of the PROM. Our inclusion criteria and 66% follow-up rate represents a study limitation as well. We aimed to avoid e-mail or phone follow-up to avoid introducing bias into the results. In addition, we do not believe that those patients lost to follow-up represent a cohort different enough in their instrument preferences or surgical goals to affect our results. Given the different numerical outcomes scales of the QuickDASH and PSFS, direct comparison of the 2 is difficult. We attempted to align the 2 scales to achieve the best comparison with regard to responsiveness for the purpose of this investigation, in line with prior investigation.^20–23

Limitations of the mixed-methods approach should be noted as well. Investigator bias and the researchers’ preconceived notions may affect the results for the qualitative analysis. We aimed to minimize such biases by including 2 reviewers (L.M.S. and R.N.K.). In addition, the patient transcripts were gathered in a standardized manner, which minimizes bias related to data collection and data interpretation, but lacks the depth that could be achieved by an interview. Other limitations include the small sample size, lack of a priori sample size estimate, and the heterogeneous population evaluated, which limit the external validity of the study. The goal of this study was not to validate the responsiveness of either instrument for a specific condition but, instead, to assess the quantitative responsiveness of an instrument in conjunction with the qualitative rationale for delivering care that is based on individual patients’ values and preferences. In addition, we did not measure the time needed to complete each instrument and the potential effect of response burden. It is possible that having patients derive their own goals and limitations requires more thought and effort; however, this is not reflected in the qualitative feedback received. In addition, we recognize that asking patients which instrument better addressed their goals in treatment may bias the results toward the PSFS (because it asks patients to identify their own goals); however, as the health care system moves toward an approach that encourages patient-centered and goal-directed care, we believe the recognition and alignment of patient goals with outcome measurement instruments becomes increasingly important. Prior work has shown the PSFS to display a floor effect (high number of patients recording minimum scores),⁴² which reduces the responsiveness of the instrument; however, no patient in this study recorded lower than a score of 28 on the preoperative PSFS. Lastly, the results of qualitative research are not necessarily meant to be generalizable, nor did we evaluate the PSFS from a population perspective, nor across various contexts (eg, language, region). Instead, the goal was to study a process and generate hypotheses about the mechanism.

Despite these limitations, we reason that a patient-specific PROM and standardized PROM may be used in a complementary manner or based upon the question at hand (eg, employing patient-specific PROMs to evaluate individual patient outcomes, employing standardized PROMs to evaluate populations of patients, and using both PROMs to evaluate quality of care delivered and support value-based reimbursement models).

APPENDIX A. COREQ PSFS

Topic	Guide Questions/Description	Reported on (Page No.) and Comments
Domain 1: Research Team and Reflexivity
Personal characteristics
Interviewer/facilitator	Which author/s conducted the interview or focus group?	S.L.E. and L.M.S. distributed questionnaires
Credentials	What were the researcher’s credentials? eg, PhD, MD	L.M.S. — MD and S.L.E. — MPH (Title page)
Occupation	What was their occupation at the time of the study?	Resident and social sciences research professional
Sex	Was the researcher male or female?	Female
Experience and training	What experience or training did the researcher have?	Both facilitators have prior quantitative and qualitative research experience, S.L.E. has an MPH
Relationship with participants
Relationship established	Was a relationship established prior to study commencement?	No prior relationship was established
Participant knowledge of the interviewer	What did the participants know about the researcher? eg, personal goals, reasons for doing the research	The purpose of the research and research process were discussed during the consent process
Interviewer characteristics	What characteristics were reported about the interviewer/facilitator? eg, bias, assumptions, reasons and interests in the research topic?	No specific characteristics, biases, assumptions were reported
Domain 2: Study Design
Theoretical framework
Methodological orientation and theory	What methodological question was stated to underpin the study? eg, grounded theory, discourse analysis, ethnography, phenomenology, content analysis	Content analysis (page 7)
Participant selection
Sampling	How were participants selected? eg, purposive, convenience, consecutive, snowball	Convenience sample of those meeting inclusion criteria
Method of approach	How were participants approached? eg, face-to-face, telephone, mail, e-mail	Participants were approached in person (page 5)
Sample size	How many participants were in the study?	38 patients were initially included (page 8)
Nonparticipation	How many people refused to participate or dropped out? Reasons?	25 patients completed postoperative surveys. 13 patients did not come to their postoperative appointment (page 8)
Setting
Setting of data collection	Where were the data collected? eg, home, clinic, workplace	Baseline data were collected in the preoperative holding area. Follow-up data were collected in the clinic
Presence of nonparticipants	Was anyone else present besides the participants and researchers?	Occasionally patients’ family members were present in the preoperative holding area or in the clinic room during survey completion
Description of sample	What are the important characteristics of the sample? eg, demographic data, date	Patient demographics are listed in Table 1
Data collection
Interview guide	Were questions, prompts, guides provided by the authors? Was it pilot-tested?	Question prompts were provided by the authors. The question prompts were not pilot-tested
Repeat interviews	Were repeat interviews carried out? If yes, how many?	No repeat interviews were carried out
Audio/visual recording	Did the research use audio or visual recording to collect the data?	No audio or visual recording was used
Field notes	Were field notes made during and/or after the interview or focus group?	No field notes were made, analyzed data were based upon patient responses
Duration	What was the duration of the interviews and focus groups?	Survey completion took about 5–10 minutes; however, this was not timed
Data saturation	Was data saturation discussed?	Data saturation was discussed during the analysis period (page 5)
Transcripts returned	Were transcripts returned to participants for comment and/or correction?	Researchers did not use transcripts. Surveys were not returned to patients
Domain 3: Analysis and Findings
Data analysis
Number of data coders	How many data coders coded the data?	Two researchers coded the data (L.M.S. and R.N.K.) (page 7)
Description of the coding tree	Did authors provide a description of the coding tree?	A codebook was developed but was not provided (page 7)
Derivation of themes	Were themes identified in advance or derived from data?	Themes were identified from the data not in advance
Software	What software, if applicable, was used to manage the data?	Web-based and HIPAA (Health Insurance Portability and Accountability Act)–compliant RedCap (Research Electronic Data Capture) was used to store data. Microsoft Excel was used to organize and code data
Participant checking	Did participants provide feedback on the findings?	Participants did not provide feedback
Reporting
Quotations presented	Were participant quotations presented to illustrate the themes/findings? Was each quotation identified? eg, participant number	Participant quotations were presented to illustrate themes, findings. Fig. 1. These quotations were kept anonymous
Data and findings consistent	Was there consistency between data presented and the findings?	Yes, there was consistency between presented data and our findings
Clarity of major themes	Were major themes clearly presented in the findings?	Major themes are clearly presented—Fig. 1 (page 8)
Clarity of minor themes	Is there a description of diverse cases or discussion of minor themes?	The researchers did not analyze minor themes