Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Jun 27.
Published in final edited form as: J Manipulative Physiol Ther. 2019 Jun 27;42(5):307–318. doi: 10.1016/j.jmpt.2019.02.006

Researching the Appropriateness of Care in the Complementary and Integrative Health Professions Part 3: Designing Instruments With Patient Input

MD Whitley 1, ID Coulter 2,*, RW Gery 1, RD Hays 3, C Sherbourne 1, PM Herman 1, LG Hilton 4
PMCID: PMC6698201  NIHMSID: NIHMS1528914  PMID: 31255308

Abstract

Objectives.

The purpose of this article is to describe how we designed patient survey instruments to ensure that patient data about preferences and experience could be included in appropriateness decisions. These actions were part of a project that examined the appropriateness of spinal manipulation and mobilization for chronic low back pain and chronic neck pain.

Methods.

We conducted focus groups, cognitive interviews, a literature review of measures in prior chiropractic and Complementary and Integrative Health (CIH) research and a pilot study to develop questionnaires of patient preferences, experiences, values, and beliefs.

Results.

Questionnaires were administered online to 2024 individuals from 125 chiropractic clinics. The survey included 3 long questionnaires and 5 shorter ones. All were administered online. The baseline items had 2 questionnaires that respondents could complete in different sittings. Respondents completed shorter biweekly follow-ups every 2 weeks, and a final questionnaire at 3 months. The 2 initial questionnaires had 81 and 140 items, the 5 biweekly follow-up questionnaires had 37 items each, and the endline questionnaire contained 121 items. Participants generally responded positively to the survey items, and 91% of the patients who completed a baseline questionnaire completed the endpoint survey 3 months later. We used “legacy” measures, and we also adapted measures and developed new measures for this study. Preliminary assessment of reliability and validity for a newly developed scale about coping behaviors indicates that the items work well together in a scale.

Conclusions:

This article documents the challenges and the efforts involved in designing data collection tools to facilitate the inclusion of patient data into appropriateness decisions.

INTRODUCTION

Appropriateness of care decisions have been based on the published literature on safety and efficacy and the judgments of experts, both clinical and scientific experts. What is missing is the voice of patients in this process. However, in an era of patient-centered care reflected in organizations such as the Patient-Centered Outcome Research Institute (PCORI),1 inclusion of patient input should be considered essential. The Center of Excellence for Research in Complementary and Alternative Medicine (CERC) was established at RAND specifically to develop a method for studies on appropriateness that included patient input and costs.2 While it is now self-evident that patient input should play a role in decisions that impact them, it is important to do that while at the same time ensuring the decisions are clinically appropriate and safe. In developing a method at RAND/UCLA to measure appropriateness (the RAND/UCLA Appropriateness Method),3,4 considerable effort was made to make sure the decisions were evidence-based or based on clinical experience that could be agreed upon by a panel of experts. Hence, the patient component should be equally evidence-based, that is, based on actual data collected from patients.

The CERC national study collected data to assess patient beliefs and preferences, patient-reported outcomes, costs and resource allocation. These data were provided to the study’s expert panels so that expert panelists could take these findings into account when determining their ratings about the appropriateness of manipulation and mobilization for chronic low back and neck pain.5,6

Although we describe experiences from a research study, our lessons learned may be applicable to Complementary and Integrative Health (CIH) providers as well. Complementary and Integrative Health providers and researchers both need rigorous patient measures to help them collect reliable and valid data that are relevant and not burdensome to patients. Our research team prioritized parsimony and survey items that were relevant to our respondents to improve participation rates and engagement with the study.

In this paper, we share lessons learned from our literature review, cognitive interviews, pilot study, and national study about 1) how to identify appropriate existing instruments to measure beliefs, preferences, and experiences with chronic pain and coping among CIH patients, 2) how to decide whether to modify a tool to better fit one’s study or clinical circumstance, 3) how to develop new measures and evaluate their reliability and validity, and 4) how to assemble multiple measures together into a single questionnaire.

THE PROBLEM

Researchers need rigorous methodologies and reliable and valid self-report measures to evaluate efficacy and effectiveness of therapies and to understand patients’ experiences and beliefs in chiropractic and other areas of CIH. Rigorous patient measures are also essential for clinicians. Clinicians may want information about patients’ perceptions of care, adherence to recommendations, and health-related quality of life.710 Measures can be useful as a research tool and for patient care. The past half century has seen a gradual shift away from exclusive reliance on clinical and laboratory measures of illness or disease-specific outcome measures towards the development and utilization of comprehensive indices of patient health status, including patient self-report (what might be considered a more holistic approach to measurement).11,12

There are various options when creating a patient survey:

  • Using all or part of existing measures

  • Revising items from an existing measure

  • Creating new items

Many researchers and clinicians prefer to use existing measures that have already been evaluated and published, as this is often more efficient than designing a tool from scratch.13,14 Using an existing tool enables them to compare results to other studies and other practitioners that used the same tool, and this provides a helpful point of comparison across research or patient subgroups. A challenge is that there are few existing measures of patient beliefs and coping that have been evaluated in chiropractic. It is important for CIH researchers and practitioners to know how to find measures, decide whether a measure will suite their needs, and understand other options if no existing measures are appropriate.

CIH researchers may opt to design their own data collection tools. This has the major advantage of enabling them to ask exactly the question they want, perhaps addressing a topic that no one else has attempted to rigorously study or measure. It gives them the opportunity to cover all their domains of interest, and to word items in a way that will make sense and be relevant to their target population.

Creating an instrument is a complicated and lengthy process. How can a researcher or clinician be sure s/he has identified all the relevant domains that a tool should capture? Which are the best response options to use and how will they affect analytic options later? How can one feel confident that their respondents will understand and respond to the items in the way that the study team intends? If researchers want to measure multiple constructs using multiple tools within the same questionnaire, how can they make sure that fatigue or confusion among respondents are not adversely affecting their responses? Lastly, how can researchers test a novel set of items to be confident that the items are reliable and valid? There are ways to address all these questions, but they require careful planning.

To summarize, our primary questions in this study were:

  1. How do we choose instruments that are patient-centric and relevant to their experiences and that will capture their preferences and values?

  2. How do we choose between, on the one hand, utilizing legacy measures1517 that have been widely used in previous studies so we can compare our study to previous work, and on the other hand, designing new instruments specific to this study?

  3. How do we choose data collection instruments that are comprehensive but concise, reliable, valid, relevant, and non-burdensome?

THE SOLUTION

In the following sections are the steps we took and the solutions we arrived at in answering the three questions above. In the results section, we discuss what the outcomes were.

METHODS

This study was approved by RAND’S Institutional Review Board, referred to as the Human Subjects Protection Committee (HSPC). This study was registered as an observational study on ClinicalTrials.gov ID:

THE RAND CERC STUDY

The solution to our data collection instrument challenges was based on exploratory interviews, focus groups, literature review, cognitive interviews, a pilot study, a survey of 125 clinics and over 2000 patients, protocol documents, minutes and emails from 4 years of project activities, along with the experiences reported by our project staff who were involved in developing, testing and administering the questionnaire.

Exploratory Interviews

A researcher’s first step to developing an instrument is having a clear understanding of the constructs and domains she or he intends to measure. We conducted exploratory phone interviews with 40 chiropractic patients with low back or neck pain. Our exploratory interviews took 30–45 minutes using open-ended questions to guide patients to tell us the story of their pain condition. The responses were analyzed using pile sorting, which identified key domains in this area and laid them out in a framework.18 The purpose of the exploratory interviews was to determine what issues were relevant to patients and what we needed to measure to ensure their perspectives and concerns were captured. They helped to determine the constructs for which we needed to find or create questions.

Focus Groups

Based on qualitative methods used by members of the research team in previous studies,19,20 we conducted focus groups and cognitive interviews prior to the main study. First, we conducted 6 focus groups (2 in Los Angeles, 2 in Chicago, and 2 in Boston) with patients age 18 and older to identify key aspects of experiences with chiropractic care. We included patients with a range of prior chiropractic experience and sociodemographic characteristics (age, gender, race/ethnicity). Participants were recruited with the help of a focus group vendor.

The focus groups were led by a senior Principal Investigator using a semi-structured guide and notes were taken by a research assistant. The groups began with open-ended questions inquiring about experiences with care and desired outcomes. Then a summary of the CAHPS and PROMIS measures was shown to participants to get their opinion about their relevance and importance in representing their own experience. The research team probed to identify important areas not captured by the existing measures. The groups were audio-taped, transcribed verbatim and analyzed using Atlas.ti software. Focus group themes were identified and compared to the hypothesized domains. We documented feedback about possible problems with existing items and identified new item content. New items were written as needed.

The purpose of the focus groups was to evaluate the applicability of standardized Patient Reported Outcomes (PROs) that assess patient experiences of care (e.g. Consumer Assessment of Healthcare Providers and Systems or CAHPS®) and health-related quality of life (Patient-Reported Outcomes Measurement and Information System or PROMIS®) for chiropractic patients who had experienced manipulation or mobilization or back pain. The focus groups contributed to the process of developing instruments because they evaluated whether the existing questions were perceived to be important by chiropractic patients and reflective of what is important about the care they receive and outcomes of care. The groups also explored possible gaps in the content of the existing measures that are important to patients. This information was used to determine if modifications to existing measures were needed and if there were gaps that require newitems. Revisions to existing items and new items were then drafted and subjected to cognitive interviews with chiropractic patients who have experienced back pain. The revised and new items were finalized based on cognitive interviews (see below).

Literature Review

While we collected exploratory and pilot data, we were also conducting a comprehensive search of the literature to identify existing tools (legacy instruments) that measured patient experiences, beliefs and preferences related to chiropractic care and/or chronic pain. We identified 49 existing tools (see list in Appendix A). Four researchers on our team with expertise in anthropology, psychology, sociology and public health read through the items. We included in our final set of measures some commonly-used outcomes such as the Oswestry Disability Index,21,22 Neck Disability Index23 and the PROMIS-29 v 2.0 profile measure24 because those tools measured constructs relevant to patient experiences such as the impact of pain on a person’s everyday life. Two other tools that we selected for our questionnaire were the Survey of Pain Attitudes25,26 and the Chronic Pain Self-Efficacy Scale.27 We felt those tools addressed unique aspects of patient experiences and beliefs that were not already captured by other tools already identified.

Reviewing the 49 existing tools helped us learn about constructs and existing measures. The literature review and the exploratory data were used to create a framework. Our framework provided a clear list of constructs for which we needed measures, and it acted as a theoretical guide for selecting and developing our instruments. The literature review pertains to question 2, how do we choose between, on the one hand, utilizing legacy measures which have been widely used in previous studies so we can compare our study to previous work, and on the other hand, identify where designing new instruments specific to this study would be necessary. To that extent they help answer question 1, what is relevant.

Cognitive Interviews

We conducted face-to-face cognitive interviews to ensure the patient experience items we developed were understood by patients. We probed about item stem content and clarity of the response options. Each item was reviewed by at least 5 adult (18 and older) subjects. We limited our interview to 40 items per subject with a goal of a 60-minute interview. We tested 80 items overall, and we conducted 10 cognitive interviews in total to obtain 5 interviews per item. Half of the interviews were conducted on the west coast of the United States and the other half on the east coast. We conducted some cognitive interviews using intermittent probes and others using retrospective probes following completion of all items. The interviewer asked open-ended questions and probed about item stems, response options, and time frames. After completion of each cognitive interview, the items were discussed with the research assistant conducting the interviews. Any new items developed were documented and potential problems with the existing CAHPS and PROMIS measures noted. Where we identified problems with an existing item, we created a revised version of the item. For example, in the original version of the CAHPS Health Plan Study we inserted “chiropractor” as an exemplar in the instructions that refer to primary care providers. In the national study, we administered both the standard item (for comparability with existing studies) and the revised item. A total of about 20 additional items (revised items and new items) were included in the national sample following the cognitive interviews.

Cognitive interviews allowed us to determine if constructs we had identified could be understood by patients and communicated to the researchers. They also contributed to our understanding of respondent burden.

Pilot study

Once the items had been programmed into a web survey, the research team tested the web tools multiple times to ensure that they had been programmed correctly, that the skip patterns were correct, and that there were no other unforeseen issues with the web surveys. After the questionnaires were fully programmed, we conducted a pilot study with 89 low back or neck pain patients from 7 local chiropractic clinics. We gathered information at the end of an online questionnaire in our pilot study to assess patients’ reactions to our data collection tools and to identify potential problems with the length of the questionnaire or comprehensibility of the items. The items assessed participants’ experiences signing up for the study in their chiropractor’s office, their thoughts about completing the questionnaire in general and about the length of the questionnaire, and whether any of the items were confusing. We counted how many participants dropped out of the study at different phases of the pilot. The fact that drop out was low from one stage to the next indicated that the questionnaires were not overly burdensome, and that the incentives were sufficient. The pilot study contributed to all three of our questions.

RESULTS

Existing Tools

A drawback of using existing tools in their entirety was that some items were not relevant to all respondents. For instance, one of the 10 Oswestry Disability Index21,22 items asks about the impact of pain on the respondent’s sex life, and the item does not include a “does not apply” option. Two pilot study respondents commented that because they were not sexually active, they did not know how to answer that question. One stated, “On the sex question, for example, I am single and celibate. It would have been better to put an “other” choice or left a comment box. Don’t just assume that everyone has a sex life.”

In some instances, we used subscales from an existing instrument rather than the entire instrument. For instance, the Chronic Pain Self-Efficacy Scale includes items that cover 3 distinct self-efficacy domains: pain management, coping and physical function.27 We decided to use the items for only the pain management and coping domains because our questionnaire already included many items about physical function.

Selecting specific subscales of a larger tool gave us more control over which items we included and helped us avoid having a very long, repetitive questionnaire. A disadvantage of this approach is that if we wanted to compare our results with results from other published papers, we could only refer to studies that reported scores for each subscale. In the case of the Chronic Pain Self-Efficacy Scale, for instance, some authors have reported subscale scores,28,29 but in other cases, especially if the tool is not the primary outcome measure, authors reported only the overall scale score.30

Modifying Existing Instruments

In some instances, we took an existing item or set of items and revised the wording to better fit our study population. For example, we used the Credibility items from the Credibility/Expectancy Questionnaire,31 and one of those items was “At this point, how successful do you think this treatment will be in reducing your trauma symptoms?” Many CIH patients with chronic pain conditions reported seeking care from multiple types of providers, such as a chiropractor, primary care provider, and a massage therapist. We believed that if a questionnaire item referred generically to “this treatment,” and patients were completing the survey at home and not in a clinic, they may not know which treatment to report about. We changed the term to “your chiropractic treatment” to make this clear. Similarly, we thought the term “trauma symptoms” would be confusing because these patients may not have experienced anything that they or their providers would call traumatic. So, we changed that phrase to “your pain symptoms.” While these changes affected comparability with prior studies, we felt that relevance to our study population was more important.

In other cases, we used a combination of existing items and new items. For example, we used focus group input and the literature review to select Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician & Group Survey 3.0 items relevant to chiropractic care (3 access to care items, 4 communication items, and 1 global rating of the provider item). We supplemented these items with 2 additional access to care items, 5 additional communication items, 1 global rating of office appearance item, 4 items assessing office assistants, 1 item on insurance coverage, and 3 items assessing perceived outcomes of care.

Practical Considerations with Using Existing Tools

There is strong support for the use of existing measures and measures from established item pools such as PROMIS,24 rather than creating new measures. In our study, there were practical and logistical details to consider in using existing instruments. For instance, sometimes the exact language of a full instrument was easily accessible in published scholarly articles (e.g., Credibility/Expectancy Questionnaire31) or even on a website (e.g., the Center for Epidemiological Studies Depression Scale32), but in other cases, the items were only available by request (e.g., Survey of Pain Attitudes25,26). Some instruments required permission from the original authors, completion of a usage agreement, and/or payment to use. Hays and colleagues33 provide a useful discussion on copyright restrictions of survey instruments. We strongly encourage researchers to contact the authors of any tool you intend to use to ask about usage requirements. We found that the instruments’ authors were sometimes valuable resources when we encountered questions later about how to score items, for example.

The articles that we identified in the initial search of the literature were useful resources once we had collected our data. We referred to them when we scored items and wanted to compare our scores with a reference population. (See Appendix A for a list of all tools reviewed.)

Creating new tools: An example about measure coping behaviors

When there were constructs that we wished to study, often because our exploratory phone interviews had revealed a richness and variability in the ways that patients talked about those constructs, and we could not find any instruments in the literature that adequately measured those domains, we created new items. One example of this from our study was measuring coping activities. In general, we followed established tenets for developing survey items.3436 Here we share our process of developing items to assess coping behaviors and quantitively assessing study participants’ responses to those items.

We understood from the exploratory interviews that people coped with pain in many ways beyond visits to healthcare providers. While some tools measured the degree to which people felt they could cope with pain in a general sense (e.g., Chronic Pain Self-Efficacy Scale27), we aimed to measure the degree to which people coped with pain across the various domains of their life – for example, by changing or controlling their emotions and thoughts, by engaging in self-care, and by manipulating the environment around them. No existing tool captured this adequately. We identified broad domains of coping based on empirical data from our exploratory interviews and based on logic (e.g., we assumed that if patients were coping by modifying one part of their physical environment, like their home. they might also be modifying their work environment). We identified examples of coping behaviors in each domain from examples cited by respondents in the exploratory interviews and in the coping literature.37,38 Figure 1 illustrates how our empirical findings, logical assumptions and literature review led to a set of 26 items assessing coping behaviors.

Figure 1.

Figure 1.

Exploratory study findings, logical assumptions and information from the literature that informed CERC questionnaire items about coping behaviors

We created 3–4 items per domain measuring the frequency of coping activities. For most items, we applied a commonly-used 5-point response set: “never,” “rarely,” “sometimes,” “often,” and “always.” However, we used dichotomous yes/no response choices for 6 of the items (made large changes at home, made small changes at home, wore a lifting belt, and the 5 items about coping at work) because a frequency response did not make sense. For the item assessing the use of opioid medication, we referred to the Chronic Low Back Pain Taskforce Minimal Item set39 for a list of example medications.

We cognitively tested these new items with a sample of 4 individuals. The items were also tested as part of our pilot survey with 89 patients. Table 1 shows the final items that were included to assess coping activities.

Table 1.

Chronic Pain Coping Behaviors Scale, Factor Loadings for Single Factor Solution

Item Rotated factor loading
1 Meditated or used guided imagery 0.329
2 Ignored my pain* 0.063
3 Thought about what 1 need to do for my pain 0.336
4 Psychological counseling 0.524
5 Exercised (including yoga, walking, going to the gym, stretching, etc.)* −0.035
6 Got injections/shots (including steroids, epidurals, cortisol, etc.) 0.482
7 Took over the counter pain medications 0.328
8 Took herbs, other supplements or vitamins specifically for pain 0.402
9 Took non-opioid prescription pain medications 0.557
10 Tookopioid prescription pain medications 0.533
11 Used hot pads/ice packs at home 0.542
12 Rested 0.458
13 Reduced the amount of time 1 spent with friends 0.730
14 Avoided social activities 0.718
15 Talked to someone who listened or gave me advice about my pain 0.500
16 Asked or received support from someone to help with my daily tasks 0.633
17 Received emotional support for my pain from family and friends 0.585
18 Did fun things with people to help get my mind off of the pain 0.278
19 Larger changes to your home to increase comfort and accessibility, like installing a ramp or getting a new chair or bed 0.414
20 Smaller changes like installing hand bars or buying different pillows 0.358
21 Wore a lifting belt, girdle or truss to prevent injury 0.286
Factor Eigenvalue Proportion of variance
Factor 1 4.597 63.1%

Note: Factor loadings come from a single factor solution based on a polychoric correlation matrix using responses from n=1808 respondents. Rotated (oblimin oblique) factor loadings are shown.

*

Items were not included in reliability analysis due to low factor loading

Analyses of Our New Measure of Chronic Pain Coping Behaviors

We analyzed data for the 21 chronic pain coping items as a preliminary assessment of their validity and reliability. Validity refers to the degree to which a measure actually captures the construct that the researcher intends to measure, while reliability refers to whether or not responses are consistent and stable.40 We used responses from our national survey, which was conducted with patients from 125 chiropractic clinics in 6 cities across the US. The coping items were administered within our baseline online survey to n=2024 patients with chronic low back and/or chronic neck pain. After excluding 216 respondents who missed one or more of the 21 items, we had an analytic sample of n=1808 respondents.

To assess whether the items measured 1 or multiple dimensions of coping, we conducted an exploratory factor analysis using polychoric correlations to accommodate non-continuous variables. We examined the scree plot to of eigenvalues to identify the appropriate number of factors. After deciding to proceed with a single factor of coping, we identified 2 items that did not load highly onto that factor, and we proceeded with the other 19 items. We assessed internal consistency reliability of the 19 items using Cronbach’s coefficient alpha.41 Then, we assessed the fit of the single factor using confirmatory factor analysis.

Based on the results of the factor analysis, we proposed to keep 19 items in the Chronic Pain Coping Behaviors scale. In a confirmatory factor analysis using a probit structural equation model with a single latent variable (results not shown) we observed that all 19 items had statistically significant (p<0.05) factor loadings. Lastly, the alpha coefficient for the 19 items (Table 2) was 0.78, indicating acceptable reliability.

Table 2.

Chronic Pain Coping Approaches scale descriptive statistics and internal consistency (n=1808)

Descriptive statistics for sum score based on 19 items
Mean, SD 43.825
Range (out of possible 19–95) Minimum= 20, Maximum=77
Alpha coefficient 0.780

This preliminary study suggests the items work well together in a scale. The team’s next step will be to conduct additional quantitative analyses focusing on the validity of these items, such as measuring the correlation between responses on these items to response on existing legacy measures.

Advantages of creating novel items

We created new items to address various other domains, including what respondents think it means for pain to be chronic. Creating novel items was helpful to our project because it allowed us to measure behaviors that no other existing instruments captured. Because we crafted these items specifically with this patient population in mind, they were more relevant to respondents. This was reflected in the responses to the study experience items in the pilot study which we asked respondents at the end of the survey. Multiple respondents commented on how the questions made sense to them and their conditions.

Bring It All Together: Weaving Tools and Items Into One Survey

Once the 3 parts of the Center (Patient Outcomes; Patient Preferences; Resource Allocation) had identified the measures that they wanted to include, we brought all the measures and items together into one survey, assessed them for appropriate ordering, length, and consistent wording, and then conducted a series of tests in preparation for our national study.

Ordering

First, we had to consider the order of the items. We spent considerable time putting the items into a thematically logical order. Items taken directly from an existing tool were kept in the same order as they appear in the original tool. We carefully implemented skip patterns to ensure that people were not asked questions that were irrelevant to them. Also, we avoided ordering items in a way that could create bias. For example, in one part of the survey, we presented a list of factors that could influence a person’s decision to get chiropractic care, and we asked respondents to rank each item in terms of importance. Because respondents may be more likely to endorse items that appear at the beginning of a list,42 we randomized the order of those items across all respondents.

Questionnaire Length

This was a 3-month longitudinal study, and it included 3 longer questionnaires and 5 shorter ones. All questionnaires were administered online. There were many measures that we wanted to administer at baseline, and in order to reduce burden on the respondents, we separated the baseline items into two questionnaires that respondents could complete in different sittings. Next, respondents completed shorter biweekly follow-up questionnaires every two weeks, and a final questionnaire at 3-months follow-up. For respondents who had both low back and neck pain, the 2 initial questionnaires had 81 and 140 items, the 5 biweekly follow-up questionnaires had 37 items each, and the endline questionnaire contained 121 items. The questionnaires were shorter for respondents who only reported having low back or neck pain (not both).

Consistency

We tried to be consistent across all our questionnaire items in terms of how items were worded and the response categories offered. One challenge was that some legacy items were written in first person (“I”) while others were written in third (“you”). Wherever possible, we modified items to make this consistent, but because we needed to include several existing measures using the original wording, it was not possible to make this consistent everywhere. Whenever possible, we also tried to use the same response categories, such as using the same 5-point response scale across multiple items rather than switching from a 5-point to a 7-point scale, whenever possible.

DISCUSSION

We noted earlier that this study set out to answer three main questions with regard to collecting data from chiropractic practices:

  1. How do we choose instruments that are patient-centric and relevant to their experiences and that will capture their preferences and values?

  2. How do we choose between, on the one hand, utilizing existing (legacy) measures which have been widely used in previous studies so we can compare our study to previous work, and on the other hand, design new instruments specific to this study?

  3. How do we choose data collection instruments that are comprehensive but concise, reliable, valid, relevant, and non-burdensome?

As noted here, answering those three questions requires considerable effort and multi-method solutions; literature review, exploratory interviews, focus groups, cognitive interviews, pilot study, national survey. At first the questions seem quite simple, but they are woven into the broader research question of this Center. In this instance, the broader question is the appropriateness of chiropractic manipulation and mobilization in the treatment of chronic low back and neck pain. This is only the second study ever conducted in chiropractic to try to calculate a rate of appropriate care. Traditionally appropriate care was thought to be that which was efficacious and safe, and this was decided by research, researchers and clinicians.4,43 However, the question is now being raised in an era of patient-centered care and patient-centered outcomes, or outcomes that are significant to the patient.44 That raises a further question of what patient data can be collected within chiropractic treatment clinics that will allow researchers to answer that. In that broader context, the question then becomes how do we ensure that Evidence-Based Practice is truly Practice-Based Evidence. The significance of the study goes well beyond the three simple questions posed in this paper, but without solutions to those three, the broader questions cannot be answered,

This study has demonstrated that it is possible to collect a wealth of patient-centered data from chiropractic clinics, but the process is multi-faceted and quite demanding in terms of effort and resources. It can be difficult to find appropriate measurement tools to use with chiropractic/CIH patients. There may not be existing questionnaires that address a particular study’s constructs of interest, or existing tools may not apply to or have been validated with this population. Here we have shared our experiences with developing a questionnaire to assess multiple domains of patient experiences, beliefs and preferences about chiropractic care for chronic pain to demonstrate a set of approaches that researchers can use to identifying and creating tools.

We have also presented a detailed example showing how we used these methods to create a scale for measuring coping behaviors. We showed how the exploratory data collection and literature review findings, combined with logical assumptions, led us to identify key domains and key patient perspectives that we needed to capture if we wanted to understand what patients do from day to day to cope with their pain. From there, we created and tested 21 items for the general pain population and 5 additional items for people who are employed. Although the validity analysis is ongoing, we have presented preliminary findings using data from our national study to show that 19 of the 21 items general population items worked well together in a scale and had acceptable reliability. We believe these novel items are a useful contribution to the existing array of legacy measures related to coping with chronic pain, such as the Chronic Pain Self-Efficacy Scale.27

The purpose of describing these methods is to encourage researchers and clinicians to consider the many possible approaches at their disposal for collecting information from patients. We are not suggesting that everyone should combine all these approaches the way we did, but rather that they should think carefully about which approach fits their needs best.

LIMITATIONS

The limitations are that this study was conducted only within the United States. While this should not affect the methods used in that many of the instruments selected were not developed in the US and have been used on other populations (and the literature review was not restricted to US articles), it does mean this study was only focused on the US.

While not a limitation for this study and the results reported here, the generous funding of this study by NCCIH would make it difficult for others to replicate the approach we have described at least in its totality. We were able to use a very comprehensive approach to develop the instruments (literature reviews, cognitive interviews, a pilot study, a national online survey, etc.). While this type of approach will ensure that the instruments developed have been rigorously tested, clearly this level of work would be beyond most research projects in CIH. At its peak, some 16 researchers were employed on this project. This level of funding for chiropractic research to date has not been replicated outside of the US.

CONCLUSION

It is important to collect valid data about patients’ experiences and beliefs for research and clinical care. In many instances, as with our study, the best approach may be to use existing measures for some constructs, to modify existing measures for other constructs, and to create entirely new measures for constructs where the existing measures are insufficient. In this paper, we have described how we used multiple qualitative methods as well as a review of the literature to identify constructs and then design questionnaires that were successfully administered as part of a national survey of chiropractic patients with chronic low back and neck pain. We have presented preliminary reliability and validity data for one of our novel measures, which addresses coping behaviors. We have also outlined suggestions for CIH researchers and providers who want to collect this sort of information from patients.

FUNDING SOURCES AND CONFLICTS OF INTEREST

This study was funded by the NIH’s National Center for Complementary and Integrative Health Grant No: 1U19AT007912–01. All authors report that they were funded by a grant from the National Center for Complementary and Integrative Health during the study. No conflicts of interest were reported for this study.

Appendix A.

Results of Initial Search for Tools about Patient Experiences, Beliefs and Preferences for Treatment for Chronic Pain

Tool
1. 100-point Modified Von Korff Pain and Disability Scales45
2. Acute Low Back Pain Screening Questionnaire46
3. Arthritis Self-Efficacy Scale47
4. Back Beliefs Questionnaire48
5. Bath Ankylosing Spondylitis Functional Index49
6. Beck Anxiety Inventory50
7. Beck Depression Inventory- II51
8. Borg’s rating of perceived exertion scale52
9. Bournemouth Questionnaire53
10. Center for Epidemiological Studies Depression Scale54
11. Chronic Pain Acceptance Questionnaire55
12. Chronic Pain Coping Inventory56
13. Chronic Pain Self-Efficacy Scale27 - Some subscales were used.
14. Coping Strategies Questionnaire57
15. Disabilities of the Arm, Shoulder and Hand Questionnaires58
16. General Health Questionnaire-2859
17. General Self-Efficacy Scale60
18. Global Perceived Effect Scale61
19. Global Rating of Change Scale62
20. Goal Pursuit Questionnaire63
21. Health Related Quality of Life Survey64
22. Injustice Experiences Questionnaire 65
23. MedRisk Instrument for Measuring Patient Satisfaction with Chiropractic Care66
24. Multidimensional Health Locus of Control Scale67
25. Neck Disability Index23 - Complete tool was used.
26. Northwick Park Neck Pain Questionnaire68
27. Oswestry Disability Index (ODI)21,22 - Complete tool was used.
28. Pain Catastrophizing Scale69-Some items were used.
29. Pain Disability Index70
30. Pain Self-Efficacy Questionnaire71
31. Patient Practitioner Orientation Scale72
32. Patient Satisfaction Scale73
33. Patient Specific Functional Scale74
34. Photograph Series of Daily Activities75
35. PROMIS-29 v 2.0 profile measure19-Complete tool was used.
36. Psychological Inflexibility in Pain Scale76
37. Quebec Pain Disability Scale77
38. Roter Interaction Analysis System78
39. Self-Efficacy Scale79
40. Sense of Coherence80
41. SF-6D81
42. Shoulder Pain and Disability Index82
43. Sport Injury Rehabilitation Adherence Scale83
44. Survey of Pain Attitude25,26 - Some domains were used.
45. Tampa Scale of Kinesiophobia84
46. Vanderbilt Pain Management Inventory85
47. Western Ontario and McMaster Universities Osteoarthritis Index86
48. Working Alliance Inventory87
49. ZUNG self-rating depression scale88

Note: While most of these tools did not get integrated in our questionnaire, nearly all of them provided us useful insight into the types of domains being assessed in the pain literature.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • 1.Patient-Centered Outcomes Research Institute (PCORI). 2018; https://www.pcori.org/. Accessed Nov 19, 2018.
  • 2.Coulter I, Herman PM, Ryan GG, Hays RD, Hilton L, Whitley MD. Researching the Appropriateness of Care in the CAM Professions: Part I. Journal of Manipulative & Physiological Therapeutics. (Forthcoming). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Coulter I, Elfenbaum P, Jain S, Jonas W. SEaRCH™ expert panel process: streamlining the link between evidence and practice. BMC research notes. 2016;9(1):16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Fitch K, Berstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corporation; 2001. [Google Scholar]
  • 5.Coulter I, Whitley M, Hurwitz E, Vernon H, Shekelle P, Herman PM. Determining the Appropriateness of Spinal Manipulation and Mobilization for Chronic Low Back Pain: Indications and Ratings by a Multidisciplinary Expert Panel Santa Monica, CA: RAND Corporation; 2018. [Google Scholar]
  • 6.Coulter I, Whitley M, Vernon H, Hurwitz E, Shekelle P, Herman PM. Determining the Appropriateness of Spinal Manipulation and Mobilization for Chronic Neck Pain: Indications and Ratings by a Multidisciplinary Expert Panel. Santa Monica, CA: RAND Corporation; 2018. [Google Scholar]
  • 7.Fung CH, Hays RD. Prospects and challenges in using patient-reported outcomes in clinical practice. Quality of Life Research. 2008;17(10):1297–1302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Deshpande PR, Rajan S, Sudeepthi BL, Abdul Nazir CP. Patient-reported outcomes: A new era in clinical research. Perspectives in Clinical Research. 2011;2(4):137–144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Bitton A, Onega T, Tosteson ANA, Haas JS. Toward a better understanding of patient-reported outcomes in clinical practice. The American Journal of Managed Care. 2014;20(4):281–283. [PMC free article] [PubMed] [Google Scholar]
  • 10.Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: A review of the options and considerations. Quality of Life Research. 2012;21(8):1305–1314. [DOI] [PubMed] [Google Scholar]
  • 11.Khorsan R, Coulter ID, Hawk C, Choate CG. Measures in chiropractic research: choosing patient-based outcome assessments. Journal of Manipulative and Physiological Therapeutics. 2008;31(5):355–375. [DOI] [PubMed] [Google Scholar]
  • 12.Wood-Dauphinee S Assessing quality of life in clinical research: From where have we come and where are we going? Journal of Clinical Epidemiology. 1999;52(4):355–363. [DOI] [PubMed] [Google Scholar]
  • 13.Cherepanov D, Hays R. Health and quality-of-life-outcomes: The role of patient-reported measures. Outcomes measurement in the human services 2nd ed. Cross-cultural issues and methods in the era of health reform: NASM Press, Washington (DC); 2011. [Google Scholar]
  • 14.Ganz P, Hays RD, Kaplan RM, Litwin MS. Measuring health-related quality of life and other outcomes In: Kominski GG, ed. Changing the U.S. Health Care System. 4th ed. San Francisco, CA: Jossey-Bass; 2014:307–341. [Google Scholar]
  • 15.Shahgholi L, Yost KJ, Carter RE, et al. Correlation of the Patient Reported Outcomes Measurement Information System with legacy outcomes measures in assessment of response to lumbar transforaminal epidural steroid injections. American Journal of Neuroradiology. 2015;36(3):594. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Sikorskii A, Victorson D, O’Connor P, et al. PROMIS and legacy measures compared in a supportive care intervention for breast cancer patients and caregivers: Experience from a randomized trial. Psycho‐Oncology. 2018;27(9):2265–2273. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Clover K, Lambert SD, Oldmeadow C, et al. PROMIS depression measures perform similarly to legacy measures relative to a structured diagnostic interview for depression in cancer patients. Quality of Life Research. 2018;27(5):1357–1367. [DOI] [PubMed] [Google Scholar]
  • 18.Ryan GW, Bernard HR. Techniques to identify themes. Field Methods. 2003;15(1):85–109. [Google Scholar]
  • 19.Spiegel BMR, Hays RD, Bolus R, et al. Development of the NIH Patient-Reported Outcomes Measurement Information System (PROMIS) Gastrointestinal Symptom Scales. The American Journal Of Gastroenterology. 2014;109:1804–1814. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Maida CA, Marcus M, Hays RD, et al. Child and adolescent perceptions of oral health over the life course. Quality of Life Research. 2015;24(11):2739–2751. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Fairbank J, Couper J, Davies J, O’brien J. The Oswestry Low Back Pain Disability Questionnaire. Physiotherapy. 1980;66(8):271–273. [PubMed] [Google Scholar]
  • 22.Fairbank JC, Pynsent PB. The Oswestry Disability Index. Spine. 2000;25(22):2940–2953. [DOI] [PubMed] [Google Scholar]
  • 23.Vernon H, Mior S. The Neck Disability Index: A study of reliability and validity. Journal of Manipulative and Physiological Therapeutics. 1991;14(7):409–415. [PubMed] [Google Scholar]
  • 24.National Institutes of Health (NIH). Patient-Reported Outcomes Measurement Information Systems (PROMIS®). 2017; https://commonfund.nih.gov/promis/index. Accessed Nov 19, 2018.
  • 25.Jensen MP, Karoly P, Braver S. The measurement of clinical pain intensity: A comparison of six methods. Pain. 1986;27(1):117–126. [DOI] [PubMed] [Google Scholar]
  • 26.Jensen MP, Turner JA, Romano JM, Lawler BK. Relationship of pain-specific beliefs to chronic pain adjustment. Pain. 1994;57(3):301–309. [DOI] [PubMed] [Google Scholar]
  • 27.Anderson KO, Dowds BN, Pelletz RE, Edwards WT, Peeters-Asdourian C. Development and initial validation of a scale to measure self-efficacy beliefs in patients with chronic pain. Pain. 1995;63(1):77–83. [DOI] [PubMed] [Google Scholar]
  • 28.Morone NE, Greco CM, Moore CG, et al. A mind-body program for older adults with chronic low back pain: A randomized clinical trial. JAMA Internal Medicine. 2016;176(3):329–337. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Shpaner M, Kelly C, Lieberman G, et al. Unlearning chronic pain: A randomized controlled trial to investigate changes in intrinsic brain connectivity following cognitive behavioral therapy. NeuroImage: Clinical. 2014;5:365–376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Williams KA, Petronis J, Smith D, et al. Effect of Iyengar yoga therapy for chronic low back pain. Pain. 2005;115(1–2):107–117. [DOI] [PubMed] [Google Scholar]
  • 31.Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. Journal of Behavior Therapy and Experimental Psychiatry. 2000;31(2):73–86. [DOI] [PubMed] [Google Scholar]
  • 32.The Center for Epidemiologic Studies Depression Scale Revised. http://cesd-r.com/.Accessed Oct 23, 2018.
  • 33.Hays RD, Weech-Maldonado R, Teresi JA, Wallace SP, Stewart AL. Commentary: copyright restrictions versus open access to survey instruments. Medical Care. 2018;56(2):107–110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Fowler FJ. Improving Survey Questions: Design and Evaluation. SAGE Publications; 1995. [Google Scholar]
  • 35.Johnson RL, Morgan GB. Survey Scales: A Guide to Development, Analysis, and Reporting. Guilford Publications; 2016. [Google Scholar]
  • 36.Willis GB, Artino AR Jr. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. Journal of Graduate Medical Education. 2013;5(3):353–356. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Barry LC, Kerns RD, Guo Z, Duong BD, Iannone LP, Reid MC. Identification of strategies used to cope with chronic pain in older persons receiving primary care from a Veterans Affairs Medical Center. Journal of the American Geriatrics Society. 2004;52(6):950–956. [DOI] [PubMed] [Google Scholar]
  • 38.Fleishman JA, Sherbourne CD, Cleary PD, Wu AW, Crystal S, Hays RD. Patterns of coping among persons with HIV infection: Configurations, correlates, and change. American Journal of Community Psychology. 2003;32(1–2):187–204. [DOI] [PubMed] [Google Scholar]
  • 39.Deyo RA, Dworkin SF, Amtmann D, et al. Report of the NIH Task Force on research standards for chronic low back pain. Physical Therapy. 2015;95(2):e1–e18. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Singleton R Jr, Straits BC. Approaches to Social Research. 5th ed: Oxford University Press; 2010. [Google Scholar]
  • 41.Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–334. [Google Scholar]
  • 42.Krosnick JA, Alwin DF. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly. 1987;51(2):201–219. [Google Scholar]
  • 43.Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A Method for the Detailed Assessment of the Appropriateness of Medical Technologies. International Journal of Technology Assessment in Health Care. 2009;2(1):53–63. [DOI] [PubMed] [Google Scholar]
  • 44.World Health Organization; Regional Office for Europe, Appropriateness in Health Care Services: Report on a WHO Workshop, 23–25 March 2000, Koblenz, Germany, 2000. [Google Scholar]
  • 45.Von Korff M, Ormel J, Keefe FJ, Dworkin SF. Grading the severity of chronic pain. Pain. 1992;50(2):133–149. [DOI] [PubMed] [Google Scholar]
  • 46.Linton SJ, Halldén K. Can we screen for problematic back pain? A screening questionnaire for predicting outcome in acute and subacute back pain. The Clinical Journal of Pain. 1998;14(3):209–215. [DOI] [PubMed] [Google Scholar]
  • 47.Lorig K, Chastain RL, Ung E, Shoor S, Holman HR. Development and evaluation of a scale to measure perceived self‐efficacy in people with arthritis. Arthritis & Rheumatism. 1989;32(1): 37–44. [DOI] [PubMed] [Google Scholar]
  • 48.Symonds T, Burton AK, Tillotson KM, Main CJ. Do attitudes and beliefs influence work loss due to low back trouble? Occupational Medicine. 1996;46(1):25–32. [DOI] [PubMed] [Google Scholar]
  • 49.Calin A, Garrett S, Whitelock H, et al. A new approach to defining functional ability in ankylosing spondylitis: the development of the Bath Ankylosing Spondylitis Functional Index. The Journal of Rheumatology. 1994;21(12):2281–2285. [PubMed] [Google Scholar]
  • 50.Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: Psychometric properties. Journal of Sonsulting and Clinical Psychology. 1988;56(6):893. [DOI] [PubMed] [Google Scholar]
  • 51.Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Archives of General Psychiatry. 1961;4(6):561–571. [DOI] [PubMed] [Google Scholar]
  • 52.Borg G Perceived exertion as an indicator of somatic stress. Scandinavian Journal of Rehabilitation Medicine. 1970;2(2):92–98. [PubMed] [Google Scholar]
  • 53.Bolton JE, Breen AC. The Bournemouth Questionnaire: A short-form comprehensive outcome measure. I. Psychometric properties in back pain patients. Journal of Manipulative and Physiological Therapeutics. 1999;22(8):503–510. [DOI] [PubMed] [Google Scholar]
  • 54.Radloff LS. The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement. 1977;1(3):385–401. [Google Scholar]
  • 55.McCracken LM, Vowles KE, Eccleston C. Acceptance of chronic pain: Component analysis and a revised assessment method. Pain. 2004;107(1–2):159–166. [DOI] [PubMed] [Google Scholar]
  • 56.Jensen MP, Turner JA, Romano JM, Strom SE. The chronic pain coping inventory: Development and preliminary validation. Pain. 1995;60(2):203–216. [DOI] [PubMed] [Google Scholar]
  • 57.Rosenstiel AK, Keefe FJ. The use of coping strategies in chronic low back pain patients: relationship to patient characteristics and current adjustment. Pain. 1983;17(1):33–44. [DOI] [PubMed] [Google Scholar]
  • 58.Hudak PL, Amadio PC, Bombardier C, et al. Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder, and head). American Journal of Industrial Medicine. 1996;29(6):602–608. [DOI] [PubMed] [Google Scholar]
  • 59.Goldberg DP, Hillier VF. A scaled version of the General Health Questionnaire. Psychological Medicine. 1979;9(1):139–145. [DOI] [PubMed] [Google Scholar]
  • 60.Schwarzer R, Jerusalem M. Generalized Self-Efficacy scale In: Weinman SW J, & Johnston M, ed. Measures in Health Psychology: A User’s Portfolio. Causal and Control Beliefs Windsor, UK: NFER-NELSON; 1995:35–37. [Google Scholar]
  • 61.Fischer D, Stewart AL, Bloch DA, Lorig K, Laurent D, Holman H. Capturing the patient’s view of change as a clinical outcome measure. JAMA. 1999;282(12):1157–1162. [DOI] [PubMed] [Google Scholar]
  • 62.Jaeschke R, Singer J, Guyatt GH. Measurement of health status: Ascertaining the minimal clinically important difference. Controlled Clinical Trials. 1989;10(4):407–415. [DOI] [PubMed] [Google Scholar]
  • 63.Karsdorp PA, Vlaeyen JWS. Goals matter: Both achievement and pain-avoidance goals are associated with pain severity and disability in patients with low back and upper extremity pain. Pain. 2011;152(6):1382–1390. [DOI] [PubMed] [Google Scholar]
  • 64.Newschaffer CJ. Validation of Behavioral Risk Factor Surveillance System (BRFSS) HRQOL measures in a statewide sample. Atlanta, GA: Centers for Disease Control and Prevention; 1998. [Google Scholar]
  • 65.Sullivan MJ, Adams H, Horan S, Maher D, Boland D, Gross R. The role of perceived injustice in the experience of chronic pain and disability: Scale development and validation. Journal of Occupational Rehabilitation. 2008;18(3):249–261. [DOI] [PubMed] [Google Scholar]
  • 66.Beattie PF, Nelson R, Murphy DR. Development and preliminary validation of the MedRisk instrument to measure patient satisfaction with chiropractic care. Journal of Manipulative and Physiological Therapeutics. 2011;34(1):23–29. [DOI] [PubMed] [Google Scholar]
  • 67.Wallston BS, Wallston KA, Kaplan GD, Maides SA. Development and validation of the health locus of control (HLC) scale. Journal of Consulting and Clinical Psychology. 1976;44(4):580. [DOI] [PubMed] [Google Scholar]
  • 68.Leak A, Cooper J, Dyer S, Williams K, Turner-Stokes L, Frank A. The Northwick Park Neck Pain Questionnaire, devised to measure neck pain and disability. Rheumatology. 1994;33(5):469–474. [DOI] [PubMed] [Google Scholar]
  • 69.Sullivan MJ, Bishop SR, Pivik J. The pain Catastrophizing Scale: Development and validation. Psychological Assessment. 1995;7(4):524. [Google Scholar]
  • 70.Chibnall JT, Tait RC. The Pain Disability Index: Factor structure and normative data. Archives of Physical Medicine and Rehabilitation. 1994;75(10):1082–1086. [DOI] [PubMed] [Google Scholar]
  • 71.Nicholas MK. The pain self-efficacy questionnaire: Taking pain into account. European Journal of Pain. 2007;11(2):153–163. [DOI] [PubMed] [Google Scholar]
  • 72.Shaw WS, Woiszwillo MJ, Krupat E. Further validation of the Patient-Practitioner Orientation Scale (PPOS) from recorded visits for back pain. Patient Education and Counseling. 2012;89(2):288–291. [DOI] [PubMed] [Google Scholar]
  • 73.Kim HS. Collaborative decision making in nursing practice: A theoretical framework In: Chinn P, ed. Advances in Nursing Theory Development. Rockville, MD: Aspen Systems; 1983:271–283. [Google Scholar]
  • 74.Stratford P, Gill C, Westaway M, Binkley J. Assessing disability and change on individual patients: A report of a patient specific measure. Physiotherapy Canada. 1995;47(4):258–263. [Google Scholar]
  • 75.Leeuw M, Goossens ME, van Breukelen GJ, Boersma K, Vlaeyen JW. Measuring perceived harmfulness of physical activities in patients with chronic low back pain: The Photograph Series of Daily Activities—short electronic version. The Journal of Pain. 2007;8(11):840–849. [DOI] [PubMed] [Google Scholar]
  • 76.Wicksell RK, Lekander M, Sorjonen K, Olsson GL. The Psychological Inflexibility in Pain Scale (PIPS)–Statistical properties and model fit of an instrument to assess change processes in pain related disability. European Journal of Pain. 2010;14(7):771. e771–771. e714. [DOI] [PubMed] [Google Scholar]
  • 77.Kopec JA, Esdaile JM, Abrahamowicz M, et al. The Quebec Back Pain Disability Scale. Measurement properties. Spine. 1995;20(3):341–352. [DOI] [PubMed] [Google Scholar]
  • 78.Roter D, Larson S. The Roter interaction analysis system (RIAS): Utility and flexibility for analysis of medical interactions. Patient Education and Counseling. 2002;46(4):243–251. [DOI] [PubMed] [Google Scholar]
  • 79.Altmaier EM, Russell DW, Kao CF, Lehmann TR, Weinstein JN. Role of self-efficacy in rehabilitation outcome among chronic low back pain patients. Journal of Counseling Psychology. 1993;40(3):335. [Google Scholar]
  • 80.Antonovsky A The structure and properties of the sense of coherence scale. Social Science & Medicine. 1993;36(6):725–733. [DOI] [PubMed] [Google Scholar]
  • 81.Brazier J, Usherwood T, Harper R, Thomas K. Deriving a preference-based single index from the UK SF-36 Health Survey. Journal of Clinical Epidemiology. 1998;51(11):1115–1128. [DOI] [PubMed] [Google Scholar]
  • 82.Roach KE, Budiman‐Mak E, Songsiridej N, Lertratanakul Y. Development of a shoulder pain and disability index. Arthritis & Rheumatism.1991;4(4):143–149. [PubMed] [Google Scholar]
  • 83.Brewer B, Van Raalte J, Petitpas A, Sklar J, Ditmar T. A brief measure of adherence during sport injury rehabilitation sessions. Journal of Applied Sport Psychology. 1995;7(Suppl):S44. [Google Scholar]
  • 84.Vlaeyen JWS, Kole-Snijders AMJ, Boeren RGB, van Eek H. Fear of movement/(re)injury in chronic low back pain and its relation to behavioral performance. Pain. 1995;62(3):363–372. [DOI] [PubMed] [Google Scholar]
  • 85.Brown GK, Nicassio PM. Development of a questionnaire for the assessment of active and passive coping strategies in chronic pain patients. Pain. 1987;31(1):53–64. [DOI] [PubMed] [Google Scholar]
  • 86.Bellamy N, Buchanan WW, Goldsmith CH, Campbell J, Stitt LW. Validation study of WOMAC: A health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. The Journal of Rheumatology. 1988;15(12):1833–1840. [PubMed] [Google Scholar]
  • 87.Horvath AO, Greenberg LS. Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology. 1989;36(2):223. [Google Scholar]
  • 88.Zung WW. A self-rating depression scale. Archives of General Psychiatry. 1965;12(1):63–70. [DOI] [PubMed] [Google Scholar]

RESOURCES