Abstract
Purpose
The field of patient-centered outcomes research (PCOR) continues to develop. Patient-reported outcomes, and in particular, the Patient-Reported Outcomes Measurement Information System (PROMIS) contribute complementary data to clinician-derived outcomes traditionally used in health decision-making. However, there has been little work to understand how PROMIS measures may inform or be integrated into PCOR or clinical applications.
Methods
Lead investigators from four pilot projects funded by the Patient-Centered Outcomes Research Institute (PCORI) collaborated to discuss lessons learned about the use of PROMIS in PCOR studies via virtual and in-person meetings. In addition, a qualitative data collection tool was used to assess the pilot projects’ experiences.
Results
Lessons learned from the pilot projects centered on practical elements of research design, such as choosing the right outcomes to study, considering the advantages and limitations of the PROMIS short forms and computer adaptive technology versions, planning ahead for a feasible data collection process, maintaining the focus on patients by ensuring that the research is truly patient-centered, and helping patients and providers make the most of PROMIS in care.
Conclusions
The PCORI Pilot Projects demonstrated that PROMIS can be successfully used to conduct research that will help patients make decisions about their care. Interest in PCOR continues to grow and the lessons learned from these projects about the use of PROMIS will be helpful to investigators. Given the numerous benefits of PROMIS, implementing this tool in research and care will hopefully lead to significant progress in measuring health outcomes that are meaningful and relevant to all stakeholders.
Keywords: PROMIS, patient-centered outcomes, assessment center, rheumatoid arthritis, vasculitis, chronic disease, substance use
INTRODUCTION
As efforts continue to better integrate patients and their perspectives into decision-making at all levels of the health care system, patient-centered outcomes research (PCOR) is an increasingly important field of investigation. One valuable tool for identifying and incorporating the patient voice in PCOR is the collection of patient-reported outcomes (PROs), which are provided directly by patients without interpretation by providers.(1) A number of groups have advocated for collection of PROs, and recommendations and guidelines for their development, validation, and use in various settings have been published.(2–4) The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed with support from the National Institutes of Health to enhance PRO collection. PROMIS had three broad objectives: i) creating a large group of item banks measuring PROs; ii) developing a computerized adaptive testing (CAT) system that allows for efficient, psychometrically robust assessment of PROs; and iii) producing a publicly available online data collection system (Assessment Center, www.assessmentcenter.net) that allows clinical researchers to access the item banks and CATs (www.nihpromis.org).
There has been little work to understand how PROMIS measures may inform or be integrated into PCOR projects. Four Pilot Projects supported by the Patient-Centered Outcomes Research Institute (PCORI) initiated studies in 2012 that included PROMIS measures, thus providing an opportunity to evaluate the feasibility and utility of this outcome instrument in different research and care settings.
PCOR and the Role of PROs
PCOR, defined by the Patient-Centered Outcomes Research Institute (PCORI) as “research that addresses the questions and concerns most relevant to patient’, has roots in both community-based participatory research that includes collaboration with the end-users of research, and comparative effectiveness research with its focus on identifying the best option for patients among many choices.(5, 6) Above all, in PCOR studies the perspectives of patients and other stakeholders are resonant, a significant departure from the dominant research paradigm of the last century.(7)
PROs are an important aspect of PCOR and supplement other types of health data for a variety of purposes, including clinical care, quality improvement, research, population surveillance, observational research, and interventional clinical trials. Because patients directly record these outcomes, the meaning of PROs is less likely to be misinterpreted or modified. Therefore, PROs contribute important, useful, meaningful, and complementary data to augment the clinician-derived outcomes traditionally used in health decision-making.(8, 9)
PROMIS
The Patient Reported Outcomes Measurement Information System (PROMIS) represents a significant development in the science of measuring PROs for several reasons. PROMIS measures are non-proprietary and available without licensing or royalty fees. These measures have been tested and validated in large and diverse populations, and thus can be leveraged for research on many chronic conditions. PROMIS instruments are available across a variety of physical, mental, and social health domains, allowing investigators to select those of interest in their particular study.(10) PROMIS measures use item response theory (IRT) methods to create item banks. PROMIS measures can be administered through paper-based or computerized fixed-length profiles and short forms, or using computer adaptive testing (CAT) software algorithms that select items to match the person’s trait level (these versions of PROMIS instruments are referred to as “CATs”). CATs can improve precision and decrease patient burden.
Because of the broad spectrum of concepts and domains represented within the growing PROMIS framework there is a potential for wide applicability of these measures across health conditions, including rare diseases. Within the areas of physical, mental, and social health, an increasing number of discrete domains or “concepts of measurement” have been identified, each with item banks that cover the spectrum of that particular concept or symptom. For example, when considering the measurement of pain, separate item banks have been developed for pain intensity, interference, behavior, and quality, each reflecting a different aspect of the pain experience. Some PRO instruments have been created exclusively by researchers or clinicians using preexisting instruments, but PROMIS item banks were refined with extensive input from patients via focus groups and cognitive interviews, ensuring that the measures include their perspective and are comprehensible. The Assessment Center, the electronic testing platform for PROMIS, also allows researchers to customize surveys and items to include the content that is most relevant to their target population. PROMIS instruments have been developed for elementary school reading levels and have been translated and validated in several languages, with ongoing efforts to assess cross-cultural longitudinal validity.(11)
Because PROMIS measures have been calibrated in the general population, results are presented as T-scores in reference to population-based norms allowing comparisons across diseases. The PROMIS instruments also avoid the floor and ceiling effects present in many currently-used PROs, permitting use regardless of where along the symptom continuum a patient resides. This instrument attribute may be particularly important in developing individual treatment plans in multiple chronic conditions, across the lifespan, and throughout the disease course.(12)
While PROMIS was initially developed for clinical research, there is considerable interest in how PROMIS measures might be use as tools in individualizing clinical care. Providing PROs in general or PROMIS results in particular to patients and providers at the point of care is emerging as a strategy to help patients measure their own disease progress and may help to facilitate patient-provider communication and enhance shared health decision-making.(4, 13–15)
METHODS
The PCORI Pilot Projects Learning Network (PPPLN) was initiated by PCORI and facilitated by AcademyHealth, a professional organization for health services research, with the goal of fostering cross-project collaboration and harvesting lessons learned about conducting PCOR in the Pilot Projects. The four projects covered a spectrum of medical conditions: rheumatoid arthritis (RA), vasculitis, chronic diseases, and substance use (Table 1). Through the coordination by AcademyHealth staff, a working group of representatives from the four projects using PROMIS met virtually several times to discuss their collective experiences, and convened at a face-to-face meeting of the PCORI Pilot Project investigators in March 2014.
Table 1.
Summary of the Use of PROMIS in Four PCORI Pilot Projects
| Disease Area | Pilot Project | Research Aim | Methods |
|---|---|---|---|
| Rheumatoid Arthritis | Integrating Patient- Centered Outcomes in Arthritis Clinical Care Principal Investigator: Clifton Bingham, MD Johns Hopkins University |
To examine the feasibility and impact of integrating expanded PRO assessments, including PROMIS measures, into the clinical care of patients with rheumatoid arthritis |
|
| Vasculitis | Patient-Reported Outcomes for Vasculitis Principal Investigator: Peter A. Merkel, MD, MPH University of Pennsylvania |
To investigate the feasibility and validity of using PROMIS instruments to measure disease activity in vasculitis |
|
| Chronic disease | Creation of the Person- Centered Wellness HomeTM Across the Life Course Principal Investigator: Thelma Mielenz, PhD Columbia University |
To create a new public health framework called “the person- centered wellness home”TM |
|
| Substance Use | Evaluating PROMIS Instruments and Methods for PCOR: Substance Use Treatment Principal Investigator: Paul A. Pilkonis, PhD University of Pittsburgh |
To demonstrate the methodological advantages, ease of use, and value and efficiency of PROMIS in comparative effectiveness research and clinical care regarding substance use |
|
A data collection instrument was developed to assess their experiences in using PROMIS in their projects, which contained questions about why PROMIS was chosen for use, how it was implemented, and what challenges the projects encountered. This instrument was collected at a single point in time. This information was compiled and analyzed to identify common themes, challenges, and successes and to summarize these in terms of lessons learned. Information obtained through this initial survey was augmented with ongoing and iterative discussions between investigators regarding their experiences and has been assimilated for this report.
RESULTS
The Pilot Projects that used PROMIS encountered various challenges and successes in their studies that can be used to understand and improve this process in the future. This section outlines their lessons learned and provides guidance for investigators looking to use PROMIS in research and care.
Choose the right outcomes
Pilot Project investigators emphasized the importance of measuring outcomes that are meaningful and relevant to all stakeholders, including both patients and the research community. This means that choosing the “right” outcomes may be different for every study. However, there are a number of factors to consider when making this decision, including the scope of the project and its aims, context of use (e.g. observational research, clinical trials, and clinical care), previous work on PROs in the condition of interest, and the known limitations of instruments. The selection of an outcome is a two-step process: first to consider what should be measured, and then to consider how to measure it.
When selecting which outcomes or domains to include, it may be helpful to refer to past research on patients’ preferences for symptom measurement. This is an ideal step of the research process in which patient considerations can be brought into the research process, a hallmark of PCORI-funded research.(16) In the RA group, PROMIS domains were selected based on previous symptom prioritization exercises conducted with patients with RA and included fatigue, depression, and social participation, in addition to standard assessments of pain and fatigue.(17) The substance abuse project domain selection was guided by “core” domains included in PROMIS profile measures and further expanded to include additional measures of emotional distress, social support, cognitive concerns, sexual interest and others that patients and other stakeholders had prioritized. (18) The final set of PROMIS domains chosen for the vasculitis project was the result of a multistage and iterative process involving analysis of prior qualitative data on patient-reported concerns, prioritization exercises conducted through the OMERACT (Outcome Measures in Rheumatology) process, examination of deficiencies of already-utilized outcome tools use in the field, and extensive discussion within the project Steering Committee which prominently included Patient-Partners as full members.(19)
After domains for a study are selected, it is important to then consider which instruments are available and appropriate to use. When choosing instruments it is important to evaluate available measures that cover the domains of interest for their reliability, precision, and responsiveness. The projects described here all selected PROMIS measures, but all four collected additional PROs. PROMIS offers a large variety of outcomes, researchers should consider that they may need to supplement with additional outcomes depending on patients’ and researchers’ needs. PROMIS was originally developed to cover domains that were “universal” and broadly applicable across health conditions. As such PROMIS instruments mostly address concepts at the level of functioning (according to the Wilson Cleary model), or activity limitations/participation (according to the ICF model), or general symptoms like pain and fatigue. Thus PROs to cover domains that may be more restricted to a particular condition or body location may also be needed. In the Pilot Projects, the RA team programmed additional questions into Assessment Center concerning “flare” and stiffness. The Chronic Disease team added non-PROMIS physical activity and self-efficacy measures (these are forthcoming in PROMIS). The substance use project created a comprehensive health profile that covered 9 PROMIS domains to allow examination of the global impacts of substance use on patient health. The vasculitis group combined collection of data from multiple PROMIS domains with other general and disease-specific PROs.
Consider the advantages and limitations of the PROMIS short forms and CATs
The choice to use PROMIS short forms or CATs or a combination of the two formats has implications for the feasibility, precision, and patient-centeredness of data collection, as well as technology and infrastructure needs. Pilot Project investigators’ experiences reflect the importance of weighing the advantages and disadvantages of each format in advance of the study, while still maintaining flexibility if challenges arise.
Depending on the context of use, considerations of instruments such as measurement error may be important. For PROMIS instruments, CATs have the potential advantage of improved precision compared to a fixed item short form. This may be especially important, for example, if one wanted to detect individual patient-level changes over time in clinical practice that could inform health decision making. Whereas, fixed item short forms may be adequate for group level comparisons in observational studies, if these had associated larger measurement error, they may not be sufficient in a clinical care setting.
In choosing which type of instrument to use, investigators should also keep in mind the experience of patients who will complete the instruments. In some cases, CATs were the best option since they helped to reduce the burden of data collection by requiring fewer responses from patients. Because the substance use project team wanted to measure nine domains across twenty item banks, CATs were the preferred choice for reducing responder burden. Patients may also prefer one tool over another because of ease of use. In qualitative interviews, patients with RA reported a preference for the computer administration over traditional paper forms. All the projects that used CATs reported that patients, even those with limited computer experience, were able to complete the questionnaires without significant challenges.
However, several projects also demonstrated that using the technology for CATs and Assessment Center can present challenges. Study designs that go beyond the basic functions of Assessment Center may require support from Assessment Center staff. Some examples of modifications to the software that presented challenges to the pilot projects are included in Table 2. While these staff provided support in setting up the questionnaires, investigators on some projects still experienced additional challenges throughout their studies.
Table 2.
Modifications Made to PROMIS by the Pilot Projects that Required Support from Assessment Center Staff
|
The Chronic Disease project used Assessment Center Lite (the offline version of the platform) to administer PROMIS short forms due to requirements for on-site data storage and management at the specific researcher’s institution. While the specified hardware for laptops and software was time intensive to install, Assessment Center staff helped to overcome these barriers. Assessment Center Lite is being replaced by an Assessment Center Application Programming Interface (API) that should help to facilitate future offline use. The Chronic Disease study data collection began with hard copy short forms because of these barriers, and then transitioned to Assessment Center Lite in the follow-up assessments. Several Pilot Project investigators also experienced challenges using Assessment Center with certain browsers and devices that required, in some cases, custom solutions to technical challenges.
One important advantage of short forms is that they may measure domains for which there is no corresponding item bank and CAT. The pilot projects that used short forms did so at times because they offered health domains that lacked an item bank (e.g., pain intensity).
Future investigators may opt to use a combination of CATs and short forms for the reasons stated above, or to compare their feasibility, reliability, and responsiveness. The RA project incorporated fixed-length short forms and profiles for some domains specifically to compare responses with concomitantly administered CATs.(20) Similarly, the vasculitis study used CAT and short form instruments for patients in an already-established longitudinal cohort, who were accustomed to completing PROs and were seen at a small number of selected study sites. The vasculitis study is also using short forms for separate sets of patients enrolled in ongoing randomized controlled interventional clinical trials because of the ease of administration of short forms compared to the requirements for programming other general and disease-specific PROs and administering a CAT algorithm via Assessment Center over across a large number of trial sites.
Plan ahead for a feasible data collection process
A successful data collection process is essential for the success of any investigation that uses PROs to ensure that these data can be collected efficiently, and using PROMIS is no different. Among the advantages of using Assessment Center for the PROMIS instruments and corresponding CAT item banks is that the data are entered directly by patients into a computer database allowing subsequent merger with another database both easy and accurate. Assessment Center scores all PROMIS instruments prior to downloading. However, use of PROMIS may also present particular challenges in operationalization and implementation that require forethought and creative solutions. Despite some initial obstacles, the Pilot Projects showed it is possible, with appropriate planning and considerations, to make PROMIS data collection both feasible and patient-centered.
Based on their experiences, the principal investigators recommend that future investigators ensure they will have adequate technology for data collection, such as dedicated computers or mobile technology and Internet access. For example, if data collection will occur in a clinic waiting room, these resources may not be readily available. One project recommended using tablets instead of computers because they require no dedicated space and set up equipment; they do, however, require wireless internet access and security considerations. PROMIS is currently working on expanding data collections platforms away from Assessment Center to existing institutional EHR and research platforms such as EPIC and RedCAP. Another project noted that desktop computers are an option but should be dedicated for use in the study only.
Researchers who conduct their studies in a clinic setting, especially those who integrate data collection into the patient visit, should expect some challenges because this process can put added pressure on limited staff and patient time. The Pilot Projects implemented several strategies to address these issues. For instance, some projects focused on reducing the time needed for data collection. To test a larger number of PROMIS instruments in a short time, patients in the vasculitis study were randomized to different sets of PROMIS instruments with some common to all participants and others tested in only subgroups.
Other teams focused on collaborating closely with clinic staff. The substance use project team found it essential to have research staff housed within the substance use clinic. By being on site, staff were able to accommodate unexpected changes to patient and clinician schedules, to ensure prompt and complete collections of data, and to generate PRO data on health status that were used in interviews with patients and clinicians. Several teams regularly collected feedback from clinic staff to improve the process. The RA project used questionnaires, qualitative interviews, and focus groups to understand how their research affected the clinic workflow. They recommend that others should conduct these kinds of assessments early and often in the research process. In cases where this is not possible, seeking informal feedback can still be valuable, a lesson learned by the substance use team.
Ensure that the research is truly patient-centered
All the Pilot Projects emphasized the importance of conducting research that respects patients’ preferences and needs. Investigators found that, by using what they knew about patients from past research, as well as what they learned through engaging patients, the data collection process could be optimized by collecting more relevant information and more efficient through the use of Assessment Center and CATs, thus respecting patient desires and time easier. They also have good reason to think that patient feedback will contribute to research that is both more meaningful and actionable.
Several pilot project teams learned that engaging patients helped them to understand how to better accommodate patients’ needs. The substance use project faced a challenge in scheduling data collection visits given participants’ frequent recidivism. However, because they were familiar with their patient population, the investigators were ready to address this issue, and an effort was made to shorten as much as possible the interval between intake screening and completion of the set of PROMIS domains. In addition, repeated attempts were made to ensure the completion of longitudinal assessments, even if such assessments occurred outside the preferred one-month and three-month follow-up time frames due to challenges such as relapse or incarceration.
In some cases, the changes made based on patient needs seemed simple but were essential to project success. The vasculitis and RA projects learned that styluses helped many participants who struggle to use the touch-screen tablets for PROMIS data collection. This small change also made data collection in in-patient settings possible since desktop computers were not an option.
Engaging patients in data collection was also an effective strategy for making the process patient-centered: by seeking out patients’ input early and often, investigators were able to reduce the length of surveys. Patient research partners in the vasculitis project were also essential in framing and writing much of the text for instructions for participants. In the RA project, patient stakeholders had the opportunity to pilot the surveys and offer suggestions. A recent publication has summarized the engagement strategies used across these and the other PCORI Pilot Projects.(16)
Help patients and providers to make the most of PROs and PROMIS in care
PROs have great potential to serve as valuable tools to help patients and providers make collaborative decisions about care. (4, 12, 15) Because PROMIS was originally primarily developed as a research tool, the process for integrating PROMIS into health care decision-making is not always straightforward. The pilot projects demonstrated that certain strategies can greatly enhance the value of PROMIS in this context.
The current output of PROMIS data, while appropriate for researchers, can be hard for patients (and clinicians) to understand. Future researchers should explore ways to make PROMIS results reports more accessible by developing alternative outputs and visual displays and seek the input of the end-users (patients seen in clinical practice and their providers). In the substance use project, content experts, and PROMIS research staff created an initial version of a graphical output to present to patients and their clinicians. After a round of interviews with patients, several edits were made, including renaming some PROMIS measures in the output for clarity (e.g., “cognitive concerns” was changed to “concentration issues”), choosing a standard order of assessments (versus a hierarchical order from highest to lowest score), and adding color to emphasize when the scores were higher and potentially more concerning. While this process took time, it paid off: at the conclusion of the studies, patients reported that seeing their PROMIS results was motivating, improved communication with their provider, and helped them make decisions about care.
Providers who are unfamiliar with PROMIS can also benefit from guidance on how to use the results in clinical care. The RA team found that training sessions with providers on the T-score metrics and temperature maps were helpful. Likewise, the substance use project created a “frequently asked questions” document about interpreting PROMIS results to aid clinicians.
CONCLUSION
As interest in PCOR continues to grow, more and more investigators will use PROs in research and integrate this information into clinical care. Many of the principles of PRO selection and implementation discussed in this paper can be generalized. As more investigators consider using PROMIS measures, several additional considerations become relevant. In order to best utilize PROMIS, investigators will need to have the knowledge and resources necessary to conduct these studies effectively.
The PCORI Pilot Projects demonstrated that PROMIS measures can successfully be used to conduct research that will help patients make decisions about their care. These projects set out to answer disparate research questions and had different experiences; however, the project teams also found solutions to some common challenges they encountered. In some cases, their experiences reflect the importance of planning ahead as much as possible. Nonetheless, some problems cannot be anticipated and instead require creative solutions that leverage existing resources, knowledge, and partnerships.
Although the Pilot Projects represent only a small handful of experiences in using PROMIS to conduct PCOR and to move PROs into clinical care, the findings reported here exemplify the growing understanding of how PROMIS can be used and some of the potential advantages and challenges in their use. More work is needed to identify best practices for the expanding field of PRO research in general, and to conduct additional validation studies and identify best practices for implementation of PROMIS as PROs.
The Pilot Project investigative teams believe that their common experiences can be leveraged to improve both the process of conducting research and the utility of its results. Future researchers are advised to consider the lessons learned during the conduct of these projects when carrying out their own research and to continue to report on their own experiences to guide others in the use of PROMIS in PCOR and other settings.
Acknowledgments
This work was supported through Patient Centered Outcomes Research Institute (PCORI) Awards: Contract Numbers IP2-PI000737 (Bingham PI), IP2- PI000603 (Merkel PI), IP2-PI000797 (Mielenz PI), IP2-PI000189 (Pilkonis PI), and 2000-20-10-10 CMO 0001 (AcademyHealth). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the PCORI, its Board of Governors or Methodology Committee. Dr. Merkel’s work on this project was also supported by a grant from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (R01-AR064153).
Footnotes
Disclaimers
At the time of conceptualization and initial writing of the paper, Ms. Edmundson and Ms. Moore were employees of AcademyHealth.
References
- 1.Food and Drug Administration. Guidance for Industry--Patient-reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Washington, DC: Department of Health and Human Services; 2009. [Accessed May 26, 2014]. available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf. [Google Scholar]
- 2.Reeve BB, Wyrwich KW, Wu AW, Velikova G, Terwee CB, Snyder CF, Schwartz C, Revicki DA, Moinpour CM, McLeod LD, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 2013;22(8):1889–1905. doi: 10.1007/s11136-012-0344-y. [DOI] [PubMed] [Google Scholar]
- 3.Zbrozek A, Hebert J, Gogates G, Thorell R, Dell C, Molsen E, Craig G, Grice K, Kern S, Hines S. Validation of electronic systems to collect patient-reported outcome (PRO) data-recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force. Value Health. 2013;16(4):480–489. doi: 10.1016/j.jval.2013.04.002. [DOI] [PubMed] [Google Scholar]
- 4.Bingham CO, 3rd, Auger C, Feldman D, Noonan V, Wolfe D, Ahmed S, Bartlett SJ. From Research to Care: Clinical Applications of Patient Reported Outcomes. J Clin Epidemiol. 2016 doi: 10.1016/j.jclinepi.2017.04.014. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Patient Centered Outcomes Research Institute. [Accessed September 14, 2015];PCORI’s definition of “Patient-Centered Outcomes Research”. Available at: http://www.pcori.org/content/patient-centered-outcomes-research.
- 6.Patient Centered Outcomes Research Institute. [Accessed September 14, 2015];About PCORI. Available at: http://www.pcori.org/about-us.
- 7.Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–1584. doi: 10.1001/jama.2012.500. [DOI] [PubMed] [Google Scholar]
- 8.Snyder CF, Jensen RE, Segal JB, Wu AW. Patient-reported outcomes (PROs): putting the patient perspective in patient-centered outcomes research. Med Care. 2013;51(8 Suppl 3):S73–79. doi: 10.1097/MLR.0b013e31829b1d84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wu AW, Snyder C, Clancy CM, Steinwachs DM. Adding the patient perspective to comparative effectiveness research. Health Aff (Millwood) 2010;29(10):1863–1871. doi: 10.1377/hlthaff.2010.0660. [DOI] [PubMed] [Google Scholar]
- 10.Patient Reported Outcomes Measurement Information System. [Accessed September 14, 2015];PROMIS Instrument Overview. Available at: http://www.nihpromis.org/measures/instrumentoverview.
- 11.Alonso J, Bartlett SJ, Rose M, Aaronson NK, Chaplin JE, Efficace F, Leplege A, Lu A, Tulsky DS, Raat H, et al. The case for an international patient-reported outcomes measurement information system (PROMIS(R)) initiative. Health Qual Life Outcomes. 2013 Dec 20;11:210. doi: 10.1186/1477-7525-11-210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Wagner LI, Schink J, Bass M, Patel S, Diaz MV, Rothrock N, Pearman T, Gershon R, Penedo FJ, Rosen S, et al. Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer. 2015;121(6):927–934. doi: 10.1002/cncr.29104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Brundage MD, Smith KC, Little EA, Bantug ET, Snyder CF Board PRODPSA. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res. 2015;24(10):2457–2472. doi: 10.1007/s11136-015-0974-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Rose M, Bjorner JB, Gandek B, Bruce B, Fries JF, Ware JE., Jr The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency. J Clin Epidemiol. 2014;67(5):516–526. doi: 10.1016/j.jclinepi.2013.10.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Snyder CF, Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY, Hess R, Miller DM, Reeve BB, Santana M. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21(8):1305–1314. doi: 10.1007/s11136-011-0054-x. [DOI] [PubMed] [Google Scholar]
- 16.Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, Frank L. Patient and Stakeholder Engagement in the PCORI Pilot Projects: Description and Lessons Learned. J Gen Intern Med. 2016 Jan;31(1):13–21. doi: 10.1007/s11606-015-3450-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Bartlett SJ, Hewlett S, Bingham CO, 3rd, Woodworth TG, Alten R, Pohl C, Choy EH, Sanderson T, Boonen A, Bykerk V, et al. Identifying core domains to assess flare in rheumatoid arthritis: an OMERACT international patient and provider combined Delphi consensus. Ann Rheum Dis. 2012;71(11):1855–1860. doi: 10.1136/annrheumdis-2011-201201. [DOI] [PubMed] [Google Scholar]
- 18.Johnston KL, Lawrence SM, Dodds NE, Yu L, Daley DC, Pilkonis PA. Evaluating PROMIS instruments and methods for patient-centered outcomes research: Patient and provider voices in a substance use treatment setting. Qual Life Res. 2015 Sep 9; doi: 10.1007/s11136-015-1131-3. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Robson JC, Milman N, Tomasson G, Dawson J, Cronholm PF, Kellom K, Shea J, Ashdown S, Boers M, Boonen A, et al. Exploration, Development, and Validation of Patient-reported Outcomes in Antineutrophil Cytoplasmic Antibody-associated Vasculitis Using the OMERACT Process. J Rheumatol. 2015;42(11):2204–2209. doi: 10.3899/jrheum.141143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bartlett SJ, Orbai AM, Duncan T, DeLeon E, Ruffing V, Clegg-Smith K, Bingham CO., 3rd Reliability and Validity of Selected PROMIS Measures in People with Rheumatoid Arthritis. PloS One. 2015;10(9):e0138543. doi: 10.1371/journal.pone.0138543. [DOI] [PMC free article] [PubMed] [Google Scholar]