A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

Dawn M Guthrie; Nicole Williams; Cheryl Beach; Emma Buzath; Joachim Cohen; Anja Declercq; Kathryn Fisher; Brant E Fries; Donna Goodridge; Kirsten Hermans; John P Hirdes; Hsien Seow; Maria Silveira; Aynharan Sinnarajah; Susan Stevens; Peter Tanuseputro; Deanne Taylor; Christina Vadeboncoeur; Tracy Lyn Wityk Martin

doi:10.1371/journal.pone.0266569

. 2022 Apr 7;17(4):e0266569. doi: 10.1371/journal.pone.0266569

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

Dawn M Guthrie ^1,^2,^*,^#, Nicole Williams ^1,^#, Cheryl Beach ^3,^‡, Emma Buzath ^4,^‡, Joachim Cohen ^5,^‡, Anja Declercq ^6,^7,^‡, Kathryn Fisher ^8,^‡, Brant E Fries ^9,^‡, Donna Goodridge ^10,^‡, Kirsten Hermans ^6,^‡, John P Hirdes ^11,^‡, Hsien Seow ^12,^‡, Maria Silveira ^13,^‡, Aynharan Sinnarajah ^14,^‡, Susan Stevens ^15,^‡, Peter Tanuseputro ^16,^‡, Deanne Taylor ^17,^18,^‡, Christina Vadeboncoeur ^19,^20,^21,^‡, Tracy Lyn Wityk Martin ^4,^‡

Editor: Jason Scott²²

¹Department of Kinesiology and Physical Education, Wilfrid Laurier University, Waterloo, Ontario, Canada

²Department of Health Sciences, Wilfrid Laurier University, Waterloo, Ontario, Canada

³Integrated Community Services, Fraser Health, Surrey, British Columbia, Canada

⁴Provincial Palliative and-End-of-Life Care, Seniors Health and Continuing Care, Alberta Health Services, Calgary, Alberta, Canada

⁵End-of-Life Care Research Group, Vrije Universiteit Brussel, Brussels, Belgium

⁶LUCAS – Center for Care Research and Consultancy, KU Leuven, Leuven, Belgium

⁷CESO – Center for Sociological Research, KU Leuven, Leuven, Belgium

⁸School of Nursing, McMaster University, Hamilton, Ontario, Canada

⁹Department of Health Management and Policy and Department of Geriatric and Palliative Medicine, University of Michigan, Ann Arbor, Michigan, United States of America

¹⁰College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada

¹¹School of Public Health Sciences, University of Waterloo, Waterloo, Ontario, Canada

¹²Department of Oncology, McMaster University, Hamilton, Ontario, Canada

¹³Division of Geriatric and Palliative Medicine, Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, United States of America

¹⁴Department of Medicine, Queen’s University, Kingston, Ontario, Canada

¹⁵Nova Scotia Health, Halifax, Halifax, Nova Scotia, Canada

¹⁶Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada

¹⁷Research Department, Interior Health Authority, Kelowna, British Columbia, Canada

¹⁸Rural Coordination Centre of BC, Penticton, British Columbia, Canada

¹⁹Department of Pediatrics, University of Ottawa, Ottawa, Ontario, Canada

²⁰Children’s Hospital of Eastern Ontario Ottawa, Ontario, Canada

²¹Roger Neilson House, Ottawa, Ontario, Canada

²²Northumbria University, UNITED KINGDOM

Competing Interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: Dr. A. Sinnarajah has received research grants (last 5 years) on palliative care (as principal investigator or co-investigator), from the Canadian Institutes of Health Research, MSI Foundation, Canadian Cancer Society, Applied Research in Cancer Control, College of Family Physicians of Canada, Choosing Wisely Alberta, Alberta Innovates Health Research, Alberta Cancer Foundation, Alberta Health Services, University of Calgary, Canadian Frailty Network, Alberta Health and Campus Alberta Health Outcomes and Public Health. He has an academic appointment for palliative care research with Queen’s University and Lakeridge Health (currently), and University of Calgary (last 5 years). Lastly, he is/has paid medical administrative positions with Alberta Health Services and Lakeridge Health. The remaining authors declare that no competing interests exist. We confirm that Dr. Sinnarajah has some competing interests to report (previously detailed in the online submission) and we confirm that this does not alter our adherence to PLOS ONE policies on sharing data and materials.

‡ These authors also contributed equally to this work.

^✉

* E-mail: dguthrie@wlu.ca

Contributed equally.

Roles

Dawn M Guthrie: Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Nicole Williams: Data curation, Formal analysis, Methodology, Project administration, Software, Validation, Writing – review & editing

Cheryl Beach: Writing – review & editing

Emma Buzath: Writing – review & editing

Joachim Cohen: Writing – review & editing

Anja Declercq: Writing – review & editing

Kathryn Fisher: Writing – review & editing

Brant E Fries: Writing – review & editing

Donna Goodridge: Writing – review & editing

Kirsten Hermans: Writing – review & editing

John P Hirdes: Writing – review & editing

Hsien Seow: Writing – review & editing

Maria Silveira: Writing – review & editing

Aynharan Sinnarajah: Writing – review & editing

Susan Stevens: Writing – review & editing

Peter Tanuseputro: Writing – review & editing

Deanne Taylor: Writing – review & editing

Christina Vadeboncoeur: Writing – review & editing

Tracy Lyn Wityk Martin: Writing – review & editing

Jason Scott: Editor

PMCID: PMC8989210 PMID: 35390091

Abstract

Background

Individuals receiving palliative care (PC) are generally thought to prefer to receive care and die in their homes, yet little research has assessed the quality of home- and community-based PC. This project developed a set of valid and reliable quality indicators (QIs) that can be generated using data that are already gathered with interRAI assessments—an internationally validated set of tools commonly used in North America for home care clients. The QIs can serve as decision-support measures to assist providers and decision makers in delivering optimal care to individuals and their families.

Methods

The development efforts took part in multiple stages, between 2017–2021, including a workshop with clinicians and decision-makers working in PC, qualitative interviews with individuals receiving PC, families and decision makers and a modified Delphi panel, based on the RAND/ULCA appropriateness method.

Results

Based on the workshop results, and qualitative interviews, a set of 27 candidate QIs were defined. They capture issues such as caregiver burden, pain, breathlessness, falls, constipation, nausea/vomiting and loneliness. These QIs were further evaluated by clinicians/decision makers working in PC, through the modified Delphi panel, and five were removed from further consideration, resulting in 22 QIs.

Conclusions

Through in-depth and multiple-stakeholder consultations we developed a set of QIs generated with data already collected with interRAI assessments. These indicators provide a feasible basis for quality benchmarking and improvement systems for care providers aiming to optimize PC to individuals and their families.

Introduction

The goal of palliative care (PC) is to improve quality of life for individuals and their families facing the problems associated with a life-limiting illness, and to provide care that promotes dignity, respect, and comfort [1, 2]. Palliative and end-of-life care spans the disease process from early diagnosis to end-of-life, inclusive of bereavement. A majority of Canadians surveyed in 2016 supported resources being available for PC at home [3]. Recent data from Ontario—one of the sites of this study and the largest province in Canada with more than 14 million residents–found that among the approximately 50% of decedents who received PC, 43% (roughly 54,000 over the course of one year), had palliative home care services [4]. Despite this, little research has assessed the quality of home-based palliative services [5–7], instead focusing on utilization, organization, and cost-effectiveness of these services [8, 9]. Existing QIs in the literature tend to focus on hospital use [5, 10–12]. Currently, Canada does not have a standard set of quality indicators (QIs) for community-based PC. Having valid and reliable QIs, grounded in a community-based perspective, is essential to identify and monitor areas for improvement [13], and ultimately, contribute to the delivery of optimal care to individuals receiving PC. QIs have enormous potential for improving care by providing important information to a variety of users, such as care providers, consumers, accreditation organizations and researchers [14].

QI rates are often used to compare providers, at a systems level, a process known as benchmarking [15]. Benchmarking can also be thought of as a starting point to understand which factors contribute to quality improvement, to promote discussions among health care providers to encourage organizational change within the organizations being compared, and to learn from each other’s improvement strategies [16].

Although QIs have been proposed for those receiving PC, or individuals with life-limiting illnesses [17–23], they have several limitations. The majority of existing QIs tend to rely on administrative data to capture hospital admissions/emergency department (ED) visits and focus on the process or structure of care rather than on outcomes that matter most to individuals receiving PC [20, 24–26]. They also tend to be focused predominantly on individuals with cancer [17, 27, 28], despite the fact that cancer is the cause of death of less a third of decedents, and despite PC being increasingly available to those with other life-limiting conditions such as organ failure and dementia [29]. Therefore, QIs are needed that encompass a wide range of individuals who are receiving PC or who could benefit from such care, including those for whom prognosis may be less predictable [30, 31].

We outline the development of a set of proposed QIs that were explicitly defined based on data elements within the interRAI assessments. interRAI is a not-for-profit network of researchers, clinicians, and policy experts from just over 30 countries who develop and test standardized, clinical assessments for use in a variety of health and social services settings (e.g., home care, nursing homes/long-term care, inpatient mental health) [32–34]. These assessments are extensively used in multiple parts of Canada, and around the world, and the data are available to interRAI researchers and their trainees. For example, the interRAI Home Care assessment (interRAI HC) is currently used in 21 countries [35, 36] and for all long-term home care clients in Ontario. The assessments are performed by trained assessors (typically registered nurses) using all sources of information, including the person receiving care, informal caregivers (natural support persons) and families, clinical care providers and the medical record. The assessment is completed in Ontario for all home care clients who require home care services for a minimum of 60 days.

Utilizing the existing, routinely-collected, and population-based interRAI data to generate QIs is a very efficient use of this information and avoids the tremendous time and energy required using traditional approaches to quality improvement, such as using detailed chart reviews [37]. In fact, the interRAI data represents one of the only databases in Canada with a sufficient number of assessments and unique individuals to allow for the creation and testing of QIs. There is also a long history of QI development and testing by interRAI researchers [14, 38–40], with the earliest QIs proposed in the mid-1990s [41]. Earlier work by our team, using the interRAI assessment for home care, identified a set of preliminary QIs for community-based PC that could be generated with these assessments [42], and also explored the rates of these preliminary QIs by province/territory [43].

The main goal of this paper is to report on the first steps, of a larger study, to develop, test and validate a set of QIs for community-based PC. In this paper, we describe the creation of the QIs that were developed through in-depth consultations with multiple stakeholder groups and evaluation by a panel of experts.

Materials and methods

QI development took place over approximately four years (2017–2021) and is still ongoing (Fig 1). The research team included 18 researchers, clinicians or decisions makers from Canada, the US and Belgium with expertise in palliative medicine, health services research, epidemiology, and knowledge translation. The team was actively involved in all stages of the development of the QIs, which included three sequential phases. The study protocol was reviewed and approved by the Wilfrid Laurier University Research Ethics Board (REB #5683).

Phase I: Qualitative input from key stakeholders

The first activity, in this phase, involved input from decision makers from across Ontario, Canada’s largest province. A group of 30 PC experts participated in a one-day workshop, the results of which have been published previously [44]. Participants included clinical leaders, researchers, front-line care providers as well as health and information system administrators. At the time of the workshop, the province was divided geographically into 14 Local Health Integration Networks (LHINs) and 12 of the 14 LHINs were represented. Other participants represented key organizations such as Hospice Palliative Care Ontario, the Ontario Ministry of Health and Long-Term Care, Ontario Palliative Care Network and the Canadian Institute for Health Information.

In summary, the qualitative analysis of the information provided by the workshop participants identified six key themes important for measuring quality in community-based PC: access to care, patient care, caregiver support, symptom management, spiritual care, and home as the preferred place of death. Where possible, QIs were created that directly related to these six themes (as described below in “Phase II”). For example, since participants discussed symptom management as an important area to assess, we created QIs to measure pain, shortness of breath, fatigue, and other troublesome physical symptoms. The indicator concepts, structure, and definitions were derived from the validated interRAI data elements.

Following the workshop, project Knowledge Users (KUs) helped the research team to recruit individuals receiving PC, their family members and decision makers from across Canada. It was considered inappropriate, and potentially a violation of research ethics, for the research team to directly contact individuals receiving PC and their families. As such, the KUs assisted the team in recruiting potential study participants. KUs represent individuals who are likely to be able to use research results to make informed decisions about health policies, programs and/or practices. The eight KUs on our team included individuals from five different provinces/territories. A series of interviews and focus groups were held with families (n = 9) and decision makers (n = 11), including one individual who was actively receiving PC. The primary goal of these interviews and focus groups was to understand participants’ experiences within the health care system and associate these experiences to measurable QIs that could be developed with existing interRAI data. For example, it was apparent from the interviews that caregivers often felt overwhelmed in their caregiving role and expressed that there was a clear disconnect between what the system could provide and what caregivers expected from the system. Palliative care typically strives to provide access to on-call practitioners during and after office hours. Despite this, some care caregivers felt that during a crisis, they had to rely on ambulances and the use of the emergency department (ED). Caregivers cited emotional and psychological needs as well as loneliness among those receiving PC [45]. This rich qualitative data was used to guide QI development in the next phase of the project.

Phase II: Defining potential QIs

The research team used the feedback from Phase I, along with current literature [46, 47], to define potential QIs that captured the important domains related to PC quality. For example, based on the workshop results, a QI was drafted to capture the rate of caregiver distress, and other QIs were created to capture issues related to symptom management (e.g., QIs related to pain, breathlessness, falls, constipation, nausea/vomiting). Since the family caregivers and decision makers also discussed issues of accessing the ED, QIs were also developed related to ED visits and hospital admissions. QIs were defined to capture issues such as negative mood, anxiety and loneliness since these were discussed by caregivers and are supported as important for quality assessment in the literature [48, 49].

To generate these QIs, we focused on two interRAI assessment systems, namely the interRAI Home Care instrument (interRAI HC) and the interRAI Palliative Care (interRAI PC) tool, since these are currently used in care planning for home care clients, and palliative clients, in various parts of Canada. The interRAI HC, for example, is completed fully across Ontario, Newfoundland and Labrador and Yukon Territory, with partial coverage in British Columbia and Alberta [50], resulting in roughly 250,000 assessments annually. Research on the interRAI instruments supports the validity and reliability of these data and concludes that the overall quality can be trusted when used to support decision-making [51].

Both of these assessments provide several of the same validated health index scales, which are generated automatically once the assessment is completed. These scales include the Depression Rating Scale (DRS) [52], the Pain Scale [53], the Cognitive Performance Scale [54], the Caregiver Risk Evaluation [55], and the Pressure Ulcer Risk Scale [56]. Since these scales have confirmed validation, the scores on these scales were used in the QI definitions when possible. For example, for the QI on the prevalence of negative mood, the QI definition uses the DRS score of 3+, a cut-point with established predictive validity [52]. Other QIs are based on individual assessment items. Each QI has a distinct numerator and denominator (S1 Table). The QIs fall into two broad categories, namely, “follow-up prevalence” QIs and “failure to improve” QIs (Table 1). The first group captures the prevalence of the issue on re-assessment. For these QIs, the rate was based on re-assessments and admission assessments were excluded from the calculation. This is necessary since admission assessments would not truly reflect quality of PC at the time of the assessment. The “failure to improve” QIs assess the lack of improvement on the issue over two points in time. This is important to capture since individuals who come into contact with PC generally have an indication bias of high health needs. While the baseline function of these individuals is important to reflect those needs, future change in those needs (i.e., as addressed by PC) is also important to measure. In total, 27 preliminary QIs were created for further evaluation in Phase III. Of this list, 20 (74.1%) can be generated with both interRAI instruments, five (18.5%) others can be generated only with the interRAI PC data, and the remaining two QIs, can only be calculated with the interRAI HC data.

Table 1. List of the 27 preliminary QIs reviewed by the expert panel, how they relate to the 6 themes identified in Phase I and which interRAI assessment can be used to generate the QI.

Name and brief description of each theme and QIs that relate to that theme	Failure to Improve	Follow-up prevalence
1. Access to care: coordination/continuity of care, access to care providers, access to services at the appropriate time
Prevalence of emergency department visits^a,^b		X
Prevalence of hospital admissions^a,^b		X
2. Patient care: Discussion of preferred setting of death across the illness trajectory, advanced goals/care planning
Prevalence of clients feeling that progress is not being made regarding completion of personal goals^b		X
Prevalence of no advance directives^b		X
Prevalence of clients feeling a lack of completion of financial, legal and other formal responsibilities^b		X
3. Caregiver support: How to cope with distress/burden/loneliness, education for caregivers/knowledge to keep client at home, caregiver supports (networks, respite)
Prevalence of caregiver distress^a,^b		X
4. Symptom management: Treating symptoms and also having a patient-centred approach to care
Prevalence of falls^a,^b		X
Prevalence of severe or excruciating daily pain^a,^b		X
Prevalence of pain that is not controlled by medications^a,^b		X
Failure for pain to improve^a,^b	X
Prevalence of constipation^a,^b		X
Prevalence of shortness of breath at rest^a,^b		X
Prevalence of shortness of breath upon exertion^a,^b		X
Failure for shortness of breath to improve^a,^b	X
Prevalence of stasis/pressure ulcers^a,^b		X
Prevalence of a delirium-like syndrome^a,^b		X
Prevalence of nausea or vomiting^a,^b		X
Prevalence of fatigue^a,^b		X
Prevalence of sleep problems^a,^b		X
Prevalence of poor self-reported health^a,^b		X
Prevalence of negative mood^a,^b		X
Failure for negative mood to improve^a,^b	X
Prevalence of declining social activities that causes distress^a		X
Prevalence of loneliness^a		X
Prevalence of anxious complaints^a,^b		X
5. Spiritual care: Patient and their family should be provided resources and have access to spiritual care
Prevalence of struggling with the meaning of life^b		X
Prevalence of wanting to die now^b		X
6. Home as the preferred place of death: no QIs could be created to address this theme

Open in a new tab

^a indicates a QI that can be generated with the interRAI HC data

^b indicates a QI that can be generated with the interRAI PC data

Phase III: Modified Delphi panel to evaluate preliminary QIs

The third phase utilized a modified Delphi panel, based on the RAND/ULCA appropriateness method [57]. The main goal of the Delphi panel was to assess the level of agreement among a group of PC clinicians, researchers, and decision makers with respect to keeping or dropping any of the preliminary QIs. The Delphi method has been widely used in PC research [58]. Prior to the Delphi panel receiving the QI definitions and evaluation criteria, an Excel spreadsheet containing each of the QIs and evaluation criteria (along with an information letter/consent form) was shared with two researchers on our team, and a group of graduate students, for their feedback. We first consulted with two researchers with expertise in the Delphi process and a strong understanding of the goals of our study and a small group of graduate students (roughly 15–20), representing “non-experts,” to ensure that the materials were clear and the instructions were easy to follow. The students represented a mix of master’s and PhD level students who were all completing degrees within the School of Public Health Sciences at the University of Waterloo.

Decision makers who took part in an interview or focus group, during Phase I, were eligible to participate in the Delphi panel. Panel participants provided written informed consent prior to their participation. They were asked to evaluate each QI on four criteria: 1) Importance- the extent to which the indicator reflects an important outcome or issue for those receiving PC or their caregivers; 2) Validity- the degree to which the indicator truly captures some aspect of the quality of care (at a population level, not for an individual); 3) Evidence of improved outcomes- evidence that improvement in the indicator can have a positive effect on the individual; and 4) Usability- the extent to which the QI can be readily interpreted and used to improve care delivery. These criteria were based on previous research [59] and were rated on a scale of 1 to 9 (1 = low; 9 = high) as per the RAND/UCLA method. The evaluation spreadsheet also asked for input on the definition of the numerator/denominator. Finally, participants were given the opportunity to provide open-ended feedback on each QI, as well as space to suggest any additional QIs that the Delphi panelists felt were missing from the list, regardless of whether they could be measured with existing interRAI data.

Participants were given six weeks to complete the documents. If documents were not returned two weeks after the initial deadline, an individual reminder e-mail was sent, asking them to be returned within two weeks. After that time, if they still were not returned, a final reminder phone call was made. Any documents not returned then were considered a non-response and no further contact was made with the participant.

Determining consensus among raters

As outlined in the RAND/UCLA manual [57], the process to determine the level of agreement among the panel members involves multiple steps.

Step 1: focused on determining if there was “disagreement” or “agreement,” for each of the four criteria. This involved several calculations in order to arrive at the value of the interpercentile range adjusted for asymmetry (IPRAS). This method is ideal for panel sizes larger than 15, and therefore appropriate for our panel (n = 21) [57]. An example has been provided (S1 File) which outlines how each of the four criteria were determined to have agreement/disagreement.
Step 2: involved assessing the value of the median, for each of the four criteria, in conjunction with whether there was “agreement” or not from Step 1. Each of the four criteria were assigned into one of three mutually exclusive groups, namely “discard,” “retain,” or “review,” based on the work of Nakazawa et al. [60]. “Discard” was defined when the median value was between 1–3 AND there was agreement in Step 1. “Retain” was defined when the median was between 7–9 AND there was agreement from Step 1. “Review” was defined when the median was between 4–6 OR the median was another value AND there was disagreement in Step 1.
Step 3: In this step, each QI was assigned into one of three groups, namely “discard,” “retain,” or “review”, based on a review of the rating of the individual QI’s four criteria. For example, if any of the four criteria were rated “discard” in Step 2, then the QI would be discarded. If three or four of the criteria were considered “retain” then that QI was kept. Two scenarios were used to decide if a QI should be “reviewed.” First, if two of the criteria were considered “retain,” then we retained the QI for further review. Second, if two or more of the criteria were considered “review” then the QI would also fall into the “review” category. It should be noted that the research team utilized the Delphi results as a guide, to support decision-making, but the team also used their discretion when making the final decisions about whether or not to keep a QI for further consideration.

Results

A total of 33 individuals were invited to take part in the Delphi panel. They represented members of our research team (n = 12) and other experts who they suggested that we approach (n = 21). Of the 33 evaluation spreadsheets sent to the Delphi panel members, 21 were completed for a response rate of 63.6%. This group of 21 participants included three individuals who also provided input during Phase I. Among those who did not respond, one person felt that they did not have the necessary expertise to complete the evaluations, and the remainder (n = 11), did not respond after repeated emails. There were six individuals who consented, completed the demographic questionnaire, but then ultimately did not respond. They were very similar to respondents in terms of age (mean = 53.8; sd = 5.9), gender (female = 66.7%) and years of experience working in PC (>10 years = 66.7%). These individuals came from a variety of backgrounds including research (n = 3), nursing (2), and medicine (1).

The Delphi participants were mostly female (71.4%), with a mean age of 46.3 years (sd = 7.3) and the majority had been working in the area of PC for more than 10 years (66.7%; Table 2). The largest proportion (38.1%) came from Ontario, but there were also representatives from British Columbia, Alberta, Nova Scotia and Yukon Territory and two experts from outside of Canada. The majority had a clinical background in nursing or medicine (61.9%), with the remainder involved in PC research or working in the field in the role of a health care administrator or policy maker.

Table 2. Demographic characteristics of individuals who participated in the expert panel.

	Total sample (n = 21)
Mean age in years (standard deviation)	46.3 (7.3)
% (n)
Gender
Female	71.4 (15)
Male	28.6 (6)
Professional background ^a
Director/Senior Director/Project Lead	28.6 (6)
Physician	23.8 (5)
Registered practical nurse	19.0 (4)
Researcher	19.0 (4)
Other	14.3 (3)
Years of experience working in palliative care
<1 year	4.8 (1)
1–10 years	28.6 (6)
>10 years	66.7 (14)
Highest degree of education completed
University—graduate degree	90.5 (19)
College/Undergraduate university degree	9.5 (2)
Geographic location
Ontario	38.1 (8)
British Columbia	19.0 (4)
Nova Scotia	14.3 (3)
Alberta	9.5 (2)
Yukon Territory	9.5 (2)
Outside of Canada	9.5 (2)

Open in a new tab

^aThese groups are not mutually exclusive as participants were able to select all that applied

Of the 27 preliminary QIs that were evaluated, 20 (74.1%) were classified as “retain” and the remainder, as “review” (Table 3). None of the proposed QIs had scores that would put them into the “discard” category. The team decided to keep all QIs where the Delphi panel suggested that the QI be retained. Since there were only seven QIs classified as “review,” a second Delphi panel was deemed unnecessary. Instead, an online meeting was held with the research team, who provided feedback on these QIs. The final decision was made to drop five of these QIs from further consideration, mainly based on the concern about whether these QIs were truly capturing quality of PC services. Panel members also provided suggestions for new QIs that the team should consider. These included topics such as the timely access to PC services, satisfaction with care, the place of death/preferred place of death, and details around the treatment for certain issues (e.g., for depression, for anxiety). Since there were no interRAI data elements to capture these suggestions, no additional QIs were developed.

Table 3. Summary of scores from the expert panel and final decision for each of the proposed QIs.

Quality indicator^a	Median score from the expert panel 1 = low and 9 = high (and results from Step 1)^b				Panel Decision	Final Decision
	Importance	Validity	Evidence of improved outcomes	Usability
Prevalence of falls	8	6	8	7	retain	KEEP
Prevalence of severe or excruciating daily pain	9	8	9	8	retain	KEEP
Prevalence of pain that is not controlled by medications	9	7	8	8	retain	KEEP
Failure for pain to improve	8	7	7	7	retain	KEEP
Prevalence of constipation	8	7	8	8	retain	KEEP
Prevalence of shortness of breath at rest	9	7.5	8	8	retain	KEEP
Prevalence of shortness of breath upon exertion	7	6	8	7	retain	KEEP
Failure for shortness of breath to improve	8	7	9	7	retain	KEEP
Prevalence of stasis/pressure ulcers	8	8	8	8	retain	KEEP
Prevalence of a delirium-like syndrome	9	7	8	7	retain	KEEP
Prevalence of nausea or vomiting	8	7	8	7.5	retain	KEEP
Prevalence of fatigue	7	6	8	7	retain	KEEP
Prevalence of sleep problems	8	7	8	7	retain	KEEP
Prevalence of poor self-reported health	6	4	6	5	review	DROP
Prevalence of negative mood	8	6	8	7	retain	KEEP
Failure for negative mood to improve	8	7	8	8	retain	KEEP
Prevalence of declining social activities that causes distress	7	6	7	6	review	KEEP
Prevalence of loneliness	8	6 (D)	8	7	retain	KEEP
Prevalence of caregiver distress	9	8	8	8	retain	KEEP
Prevalence of anxious complaints	8	6	8	6	review	KEEP
Prevalence of struggling with the meaning of life	6.5	5 (D)	6	5.5	review	DROP
Prevalence of clients feeling a lack of completion of financial, legal and other formal responsibilities	7	5.5	7	6	review	DROP
Prevalence of clients feeling that progress is not being made regarding completion of personal goals	7	5.5	6	6	review	DROP
Prevalence of wanting to die now	8	5	5.5 (D)	5 (D)	review	DROP
Prevalence of emergency department visit	7	7	7	7	retain	KEEP
Prevalence of hospital admissions	7	7	7	7	retain	KEEP
Prevalence of no advance directives	8	7.5	7.5	8	retain	KEEP

Open in a new tab

^a The final list of 22 quality indicators that were kept are shown in italicised font

^b All criteria had “agreement” following step 1 except those marked with “D” to indicate disagreement

The modified Delphi panel resulted in 22 QIs kept for further testing and validation. Within the QIs capturing clinical issues, the indicators with the highest scores related to importance were those related to pain, shortness of breath and delirium. Those with the highest importance scores in the “psychosocial” area included QIs capturing caregiver distress, mood and loneliness. Three other QIs were kept related to hospital or ED use and advance directives.

Discussion

To our knowledge, this is the first project to recommend a set of standardized QIs for community-based PC using existing interRAI data. The proposed set of 22 QIs was developed through a rigorous and multi-year process involving many stakeholders and researchers from across Canada and in two other countries. The QIs explicitly capture the issues cited as important by those receiving PC, their families, and those working in the field. The proposed QIs can be measured with existing interRAI instruments, currently used in more than 30 countries around the world. This allows for cross-country comparisons, which have previously been completed using the QIs for nursing homes [61]. Using the existing interRAI data is an efficient and cost-effective use of this information and avoids the additional effort that would be required if quality was assessed by using detailed chart reviews [37] or additional surveying of staff, individuals receiving PC, and families.

While it is important to understand the validity of individual indicators, it is also important to evaluate the content validity of the set of QIs. Several criteria have been proposed with which to carry out this evaluation [62]. For example, it is important that the QIs adequately cover the depth and breadth of the content of interest. One way to assess this criterion is to compare our QI set to the six key themes which were discussed during Phase I. In five of these themes, we were able to develop at least one QI in each theme. However, we could not create any QIs related to theme six, namely, the home as the preferred place of death, as we lack the data elements in the interRAI tools to capture this.

A second criterion of content validity relates to proportional representation, or the number of QIs in each domain that matches the importance of that domain in the assessment of PC quality. We treated each of the six themes as equally relevant since we had no other information with which to judge the importance of these themes. However, it is clear that given the type of clinical data captured within the interRAI assessments, it was easier to create QIs to address theme four, symptom management, versus the other themes. Another criterion is the costs of measurement which relates to the burden of data collection on providers. In this regard, these preliminary QIs have a very low cost since they are calculated using existing data and no additional data collection is required. The QIs were rated, during the Delphi panel, in terms of importance, providing insight into the criterion that captures the priority or ranking of the QIs. We feel that the QI list does not include redundant QIs, another important criterion, since none of the QIs were suggested to be dropped from the original list during the Delphi process. This provides evidence that all of the 27 preliminary indicators were rated high enough to warrant further consideration. Finally, the QI development did a very good job in addressing stakeholder involvement. The QIs were developed with input from multiple stakeholders which improves the confidence that the QIs have content validity. We were, however, limited to input from only one individual, with lived experience, who was receiving PC, despite nearly a year of effort in attempting to recruit other care recipients from across Canada.

Although the interRAI data represent a very rich source of information, the study team was limited to creating QIs that could be measured using existing items within two interRAI assessments and were unable to create QIs to reflect some salient issues mentioned during the qualitative interviews, many of which have also been cited in the literature as important aspects when assessing the quality of PC. For example, we were unable to create QIs to assess issues related to communication between the person, their family and members of the health care team [63], the use of hospice/PC services [64], the extent to which the person’s wishes were met [64], and access to resources and services. We were also unable to determine what specific treatments were received for certain clinical issues like depression and anxiety [18] and how satisfied people were with the PC services they received [65]. This information is important to assess as part of ongoing quality improvement efforts, although it was not directly germane to the clinical rationale for the interRAI assessments, which is care planning. As a result, this type of information would have to be captured using alternative means.

The proposed QIs provide an efficient means to capture key quality issues using existing interRAI data. Since the interRAI assessments are used widely in Canada, and in multiple other countries, these data provide a cost-effective source of information for testing and validating QIs. Although their main function is to assess the overall care needs of the individual, in order to drive care planning, interRAI assessments are useful for case-mix measurement [66–69] and quality assessment [39, 70–72]. In Canada, public reporting on select QIs already exists for long-term care homes [73], and those receiving PC and their families deserve a similar level of transparency. The proposed QIs can contribute to improvements in quality by providing detailed information to individual care providers (e.g., home care agencies) to drive internal continuous quality improvement efforts. The QIs can also provide a solid basis for quality benchmarking and learning, when organizations are compared at a systems-level. Finally, the QI data can be used to educate consumers and to guide health care policy.

The next steps in this project will involve analyzing the existing interRAI data to understand the properties of these QIs (e.g., magnitude of the issue, level of variation between geographic regions) as part of the ongoing validation process to decide which ones should be kept in the final list. Our team has access to approximately 3.7 million home care assessments from five provinces and one territory [50]. In addition, the study team has access to approximately 110,000 interRAI PC assessments, which are completed with palliative home care clients in Ontario only. We also plan to create client-level risk adjusters for each of the proposed QIs to account for differences in risk factors across patient populations [74–76]. This step is very important since complex illnesses, multiple co-existing conditions and case-mix differences can influence the QI measures, irrespective of quality. Organizations that provide care to more impaired individuals will tend to have higher unadjusted rates, regardless of the quality of care they provide [74]. Risk adjustment methods are therefore needed to maximize the ability to make fair comparisons between providers [77].

As a result of this work, we have identified a set of 22 validated palliative care QIs capturing multiple issues that are important to individuals receiving PC, families and decision makers. This work fills an important gap as many other sectors of the health care system in Canada have access to interRAI-based QIs to assist in decision support and quality improvement [70, 71, 78, 79], but this has been lacking in the PC sector. Once the QIs are finalized, they can be readily embedded into existing software systems for use by Canadian provinces and health authorities who are using the interRAI assessments. They can also be calculated in other countries using these interRAI tools (e.g., the 21 countries using the home care instrument). The final set of QIs will be useful for the purposes of benchmarking performance across different subpopulations of interest, such as health planning/funding regions. The QIs will also provide community-based PC providers, and health system and policy decision makers, with real-time data to support them in targeting their quality improvement efforts and evaluating client outcomes.

Supporting information

S1 Table. Operational definitions for each of the 27 quality indicators (QIs).

(DOCX)

Click here for additional data file.^{(32.4KB, docx)}

S1 File. An example of applying the interpercentile range adjusted for asymmetry (IPRAS).

(DOCX)

Click here for additional data file.^{(23.6KB, docx)}

Acknowledgments

The authors gratefully acknowledge Kate Fillmore, Laurel Gillespie, Christina Lawand and Michelle Peterson Fraser for their contributions to this project.

Data Availability

The data can be found at the DOI: https://doi.org/10.5683/SP3/8U8B83.

Funding Statement

DG received an award from the Canadian Institutes of Health Research (CIHR; grant #PJT-156359). CIHR web site:https://cihr-irsc.gc.ca/e/193.html. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

1.Meier DE. Increased access to palliative care and hospice services: opportunities to improve value in health care. The Milbank Quarterly. 2011;89(3):343–80. doi: 10.1111/j.1468-0009.2011.00632.x [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Alberta Health Services. Palliative and End-of-life Care: Alberta Provincial Framework. Calgary, AB: 2014.
3.Roulston E. Canadians’ View of Palliative Care: National Online Survey. Toronto, ON: Canadian Partnership Against Cancer/Ipsos Public Affairs, 2016. [Google Scholar]
4.Health Quality Ontario. Palliative care at the end of life. Toronto: Health Quality Ontario, 2016.
5.Gagnon B, Nadeau L, Scott S, Dumont S, MacDonald N, Aubin M, et al. The association between home palliative care services and quality of end-of-life care indicators in the province of Quebec. Journal of Pain and Symptom Management. 2015;50(1):48–58. doi: 10.1016/j.jpainsymman.2014.12.012 . [DOI] [PubMed] [Google Scholar]
6.Seow H, Sutradhar R, McGrail K, Fassbender K, Pataky R, Lawson B, et al. End-of-life cancer care: Temporal association between homecare nursing and hospitalizations. Journal of Palliative Medicine. 2016;19(3):263–70. doi: 10.1089/jpm.2015.0229 . [DOI] [PubMed] [Google Scholar]
7.Cohen J, Hermans K, Dupont C, Van den Block L, Deliens L, Leemans K. Nationwide evaluation of palliative care (Q-PAC study) provided by specialized palliative care teams using quality indicators: Large variations in quality of care. Palliative Medicine. 2021;35(8):1525–41. Epub 2021/06/01. doi: 10.1177/02692163211019881 . [DOI] [PubMed] [Google Scholar]
8.Walker H, Anderson M, Farahati F, Howell D, Librach SL, Husain A, et al. Resource use and cost of end-of-life/palliative care: Ontario adult cancer patients dying during 2002 and 2003. Journal of Palliative Care. 2011;27(2):79–88. [PubMed] [Google Scholar]
9.Sussman J, Barbera L, Bainbridge D, Howell D, Yang J, Husain A, et al. Health system characteristics of quality care delivery: a comparative case study examination of palliative care for cancer patients in four regions in Ontario, Canada. Palliative Medicine. 2012;26(4):322–35. doi: 10.1177/0269216311416697 . [DOI] [PubMed] [Google Scholar]
10.Barbera L, Paszat L, Chartier C. Indicators of poor quality end-of-life cancer care in Ontario. Journal of Palliative Care. 2006;22(1):12–7. [PubMed] [Google Scholar]
11.Sussman J, Barbera L, Bainbridge D, Howell D, Yang J, Husain A, et al. Health system characteristics of quality care delivery: a comparative case study examination of palliative care for cancer patients in four regions in Ontario, Canada. Palliative Medicine. 2011;0(00):1–14. doi: 10.1177/0269216311416697 [DOI] [PubMed] [Google Scholar]
12.Grunfeld E, Lethbridge L, Dewar R, Lawson B, Paszat L, Johnston G, et al. Towards using administrative databases to measure population-based indicators of quality of end-of-life care: Testing the methodology. Palliative Medicine. 2006;20:769–77. doi: 10.1177/0269216306072553 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Cohen J, Leemans K. How can you prove that you are delivering good care? Monitoring the quality of palliative care using quality indicators. European Journal of Palliative Care. 2014;21(5):228–31. [Google Scholar]
14.Zimmerman DR. Improving nursing home quality of care through outcomes data: the MDS quality indicators. Int J Geriatr Psychiatry. 2003;18(3):250–7. Epub 2003/03/19. doi: 10.1002/gps.820 . [DOI] [PubMed] [Google Scholar]
15.Currow DC, Allingham S, Yates P, Johnson C, Clark K, Eagar K. Improving national hospice/palliative care service symptom outcomes systematically through point-of-care data collection, structured feedback and benchmarking. Supportive Care in Cancer. 2015;23(2):307–15. Epub 2014/07/27. doi: 10.1007/s00520-014-2351-8 . [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Ettorchi-Tardy A, Levif M, Michel P. Benchmarking: A method for continuous quality improvement in health. Healthcare Policy. 2012;7:e101–e19. [PMC free article] [PubMed] [Google Scholar]
17.Raijmakers N, Galushko M, Domeisen F, Beccaro M, Hagelin CL, Lindqvist O, et al. Quality indicators for care of cancer patients in their last days of life: literature update and experts’ evaluation. Journal of Palliative Medicine. 2012;15(3):308–16. doi: 10.1089/jpm.2011.0393 [DOI] [PubMed] [Google Scholar]
18.Claessen SJJ, Francke AL, Belarbi HE, Pasman RW, van der Putten MJA, Deliens L. A new set of quality indicators for palliative care: process and results of the development trajectory. Journal of Pain and Symptom Management. 2011;42(2):169–82. doi: 10.1016/j.jpainsymman.2010.10.267 [DOI] [PubMed] [Google Scholar]
19.Leemans K, Cohen J, Francke AL, Stichele RV, Claessen SJJ, Van den Block L, et al. Towards a standardized method of developing quality indicators for palliative care: protocol of the Quality indicators for Palliative Care (Q-PAC) study. BMC Palliative Care. 2013;12(6). doi: 10.1186/1472-684X-12-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Lorenz KA, Rosenfeld K, Wenger N. Quality indicators for palliative and end-of-life care in vulnerable elders. Journal of the American Geriatrics Society. 2007;55(S2):S318–S26. doi: 10.1111/j.1532-5415.2007.01338.x [DOI] [PubMed] [Google Scholar]
21.Dy SM, Kiley K, Ast K, Lupu D, Norton S, Mcmillan SC, et al. Measuring what matters: top-ranked quality indicators for hospice and palliative care from the American Academy of Hospice and Palliative Medicine and Palliative Nurses Association. Journal of Pain and Symptom Management. 2015;49(4):773–81. doi: 10.1016/j.jpainsymman.2015.01.012 [DOI] [PubMed] [Google Scholar]
22.De Schreye R, Houttekier D, Deliens L, Cohen J. Developing indicators of appropriate and inappropriate end-of-life care in people with Alzheimer’s disease, cancer or chronic obstructive pulmonary disease for population-level administrative databases: A RAND/UCLA appropriateness study. Palliative Medicine. 2017;31(10):932–45. doi: 10.1177/0269216317705099 . [DOI] [PubMed] [Google Scholar]
23.van Riet Paap J, Vernooij-Dassen M, Droes R, Radbruch L, Vissers K, Engels Y. Consensus on quality indicators to assess the organisation of palliative cancer and dementia care applicable across national healthcare systems and selected by international experts. BMC Health Services Research. 2014;14:396. doi: 10.1186/1472-6963-14-396 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.D’Angelo D, Mastroianni C, Vellone E, Alvaro R, Casale G, Latina R, et al. Palliative care quality indicators in Italy. What do we evaluate? Supportive Care in Cancer. 2012;20:1983–9. doi: 10.1007/s00520-011-1301-y [DOI] [PubMed] [Google Scholar]
25.Woitha K, Van Beek K, Ahmed N, Hasselaar J, Mollard JM, Colombet I, et al. Development of a set of process and structure indicators for palliative care: the Europall project. BMC Health Services Research. 2012;12(1):381. doi: 10.1186/1472-6963-12-381 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Bone AE, Evans CJ, Etkind SN, Sleeman KE, Gomes B, Aldridge M, et al. Factors associated with older people’s emergency department attendance towards the end of life: a systematic review. European Journal of Public Health. 2019;29(1):67–74. doi: 10.1093/eurpub/cky241 . [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Earle CC, Park ER, Lai B, Weeks JC, Ayanian JZ, Block S. Identifying potential indicators of the quality of end-of-life cancer care from administrative data. Journal of Clinical Oncology. 2003;21(6):1133–8. doi: 10.1200/JCO.2003.03.059 [DOI] [PubMed] [Google Scholar]
28.Barbera L, Seow H, Sutradhar R, Chu A, Burge F, Fassbender K, et al. Quality of end-of-life cancer care in Canada: a retrospective four-province study using administrative health care data. Current Oncology. 2015;22(5):341–55. doi: 10.3747/co.22.2636 . [DOI] [PMC free article] [PubMed] [Google Scholar]
29.World Health Organization. Global Atlas of Palliative Care at the End of Life. Geneva, Switzerland: World Health Organization, 2014. [Google Scholar]
30.Lynn J. Living long in fragile health: the new demographics shape end of life care. Hastings Center Report. 2005;35(7):S14–S8. doi: 10.1353/hcr.2005.0096 [DOI] [PubMed] [Google Scholar]
31.Rocker G, Downar J, Morrison S. Palliative care for chronic illness: driving change. Canadian Medical Association Journal. 2016;188:e493–e8. doi: 10.1503/cmaj.151454 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Carpenter GI, Hirdes JP. Using interRAI assessment systems to measure and maintain quality in long term care. A Good Life in Old Age? Monitoring and Improving Quality in Long-term Care. Paris, France: OECD/European Commission; 2013. p. 93–139. [Google Scholar]
33.Steel K, Jonsson P, DuPasquier JN, Gilgen R, Hirdes JP, Schroll M, et al. Systems of care for frail older persons. Transactions of the American Clinical and Climatological Association. 1999;110:30–7. [PMC free article] [PubMed] [Google Scholar]
34.Heckman G, Gray LC, Hirdes JP. Addressing health care needs for frail seniors in Canada: The role of interRAI instruments. Canadian Geriatrics Society Journal of CME. 2013;3. [Google Scholar]
35.van der Roest HG, van Eenoo L, van Lier LI, Onder G, Garms-Homolova V, Smit JH, et al. Development of a novel benchmark method to identify and characterize best practices in home care across six European countries: design, baseline, and rationale of the IBenC project. BMC Health Services Research. 2019;19(1):310. Epub 2019/05/17. doi: 10.1186/s12913-019-4109-y . [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Foebel AD, van Hout HP, van der Roest HG, Topinkova E, Garms-Homolova V, Frijters D, et al. Quality of care in European home care programs using the second generation interRAI Home Care Quality Indicators (HCQIs). BMC Geriatrics. 2015;15:148. doi: 10.1186/s12877-015-0146-5 . [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Chartier LB, Ovens H, Hayes E, Davis B, Calder L, Schull M, et al. Improving quality of care through a mandatory provincial audit program: Ontario’s emergency department return visit quality program. Annals of Emergency Medicine. 2021;77(2):193–202. Epub 2020/11/18. doi: 10.1016/j.annemergmed.2020.09.449 . [DOI] [PubMed] [Google Scholar]
38.Mor V, Angelelli J, Jones R, Roy J, Moore T, Morris JN. Inter-rater reliability of nursing home quality indicators in the US. BMC Health Services Research. 2003;3:20. doi: 10.1186/1472-6963-3-20 [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Berg K, Mor V, Morris J, Murphy K, Moore T, Harris Y. Identification and evaluation of existing nursing homes quality indicators. Health Care Financing Review. 2002;23(4):19–36. [PMC free article] [PubMed] [Google Scholar]
40.Karon SL, Sainfort F, Zimmerman D. Stability of nursing home quality indicators over time. Medical Care. 2000;37(6):570–9. [DOI] [PubMed] [Google Scholar]
41.Zimmerman DR, Karon SL, Arling G, Ryther Clark B, Collins T, Ross R, et al. Development and testing of nursing home quality indicators. Health Care Financing Review. 1995;16(4):104–27. [PMC free article] [PubMed] [Google Scholar]
42.Harman LE, Guthrie DM, Cohen J, Declercq A, Fisher K, Goodridge D, et al. Potential quality indicators for seriously ill home care clients: a cross-sectional analysis using Resident Assessment Instrument for Home Care (RAI-HC) data for Ontario. BMC Palliative Care. 2019;18:3. doi: 10.1186/s12904-018-0389-y [DOI] [PMC free article] [PubMed] [Google Scholar]
43.Guthrie DM, Harman LE, Barbera L, Burge F, Lawson B, McGrail K, et al. Quality indicator rates for seriously ill home care clients: Analysis of Resident Assessment Instrument for Home Care Data in six Canadian provinces. Journal of Palliative Medicine. 2019;22(11):1346–56. Epub 2019/05/17. doi: 10.1089/jpm.2019.0022 . [DOI] [PubMed] [Google Scholar]
44.Williams N, Boumans N, Luymes N, White NE, Lemonde M, Guthrie DM. What should be measured to assess the quality of community-based palliative care? Results from a collaborative expert workshop. Palliative and Supportive Care. 2021;1–7. doi: 10.1017/S1478951521000791 [DOI] [PubMed] [Google Scholar]
45.Luymes N, Williams N, Garrison L, Goodridge D, Silveira M, Guthrie DM. “The system is well intentioned, but complicated and fallible”: Interviews with caregivers and decision makers about palliative care in Canada. BMC Palliative Care. 2021;20:149. doi: 10.1186/s12904-021-00843-x [DOI] [PMC free article] [PubMed] [Google Scholar]
46.Ahluwalia SC, Chen C, Raaen L, Motala A, Walling AM, Chamberlin M, et al. A Systematic Review in Support of the National Consensus Project Clinical Practice Guidelines for Quality Palliative Care, Fourth Edition. Journal of Pain and Symptom Managament. 2018;56(6):831–70. doi: 10.1016/j.jpainsymman.2018.09.008 . [DOI] [PubMed] [Google Scholar]
47.Pasman HRW, Brandt HE, Deliens L, Francke AL. Quality indicators for palliative care: a systematic review. Journal of Pain and Symptom Management. 2009;38(1):145–56. doi: 10.1016/j.jpainsymman.2008.07.008 [DOI] [PubMed] [Google Scholar]
48.Cohen SR, Leis R. What determines the quality of life for terminally-ill cancer patients from their own perspective? Journal of Palliative Care. 2002;18(1):48–58. [PubMed] [Google Scholar]
49.National Consensus Project for Quality Palliative Care. Clinical practice guidelines for quality palliative care, third edition. Pittsburgh, PA: 2013 2013. Report No.
50.Canadian Institute for Health Information. Profile of Clients in Home Care, 2019–20202021 Mar.8, 2021. https://www.cihi.ca/en/quick-stats?utm_medium=email&utm_source=ebulletin&utm_campaign=hcc-ebulletin-march-2021&utm_content=quick-stats-en&emktg_lang=en&emktg_order=2.
51.Hogeveen SE, Chen J, Hirdes JP. Evaluation of data quality of interRAI assessments in home and community care. BMC Medical Informatics and Decision Making. 2017;17(1):150. doi: 10.1186/s12911-017-0547-9 . [DOI] [PMC free article] [PubMed] [Google Scholar]
52.Martin L, Poss JW, Hirdes JP, Jones RN, Stones MJ, Fries BE. Predictors of a new depression diagnosis among older adults admitted to complex continuing care: implications for the Depression Rating Scale (DRS). Age and Ageing. 2008;37(1):51–6. doi: 10.1093/ageing/afm162 [DOI] [PubMed] [Google Scholar]
53.Fries BE, Simon SE, Morris JN, Flodstrom C, Bookstein FL. Pain in US nursing homes: validating a pain scale for the Minimum Data Set. The Gerontologist. 2001;41(2):173–9. doi: 10.1093/geront/41.2.173 [DOI] [PubMed] [Google Scholar]
54.Morris JN, Fries BE, Mehr DR, Hawes C, Mor V, Lipsitz L. MDS Cognitive Performance Scale. Journals of GerontologySeries A, Biological Sciences and Medical Sciences. 1994;49(4):M174–M82. doi: 10.1093/geronj/49.4.m174 [DOI] [PubMed] [Google Scholar]
55.Guthrie DM, Williams N, Beach C, Maxwell CJ, Mills D, Mitchell L, et al. Development and validation of Caregiver Risk Evaluation (CaRE): A new algorithm to screen for caregiver burden. Journal of applied gerontology: the official journal of the Southern Gerontological Society. 2021;40:731–41. doi: 10.1177/0733464820920102 [DOI] [PubMed] [Google Scholar]
56.Poss J, Murphy KM, Woodbury MG, Orsted H, Stevenson K, Williams G, et al. Development of the interRAI Pressure Ulcer Risk Scale (PURS) for use in long-term care and home care settings. BMC Geriatrics. 2010;10(1):67. doi: 10.1186/1471-2318-10-67 [DOI] [PMC free article] [PubMed] [Google Scholar]
57.Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA appropriateness method user’s manual. Santa Monica, CA: 2001. [Google Scholar]
58.Junger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. Epub 2017/02/14. doi: 10.1177/0269216317690685 . [DOI] [PubMed] [Google Scholar]
59.Seow H, Snyder CF, Mularski RA, Shugarman LR, Kutner JS, Lorenz KA, et al. A framework for assessing quality indicators for cancer care at the end of life. Journal of Pain and Symptom Management. 2009;38(6):903–12. doi: 10.1016/j.jpainsymman.2009.04.024 [DOI] [PubMed] [Google Scholar]
60.Nakazawa Y, Kato M, Yoshida S, Miyashita M, Morita T, Kizawa Y. Population-based quality indicators for palliative care programs for cancer patients in Japan: A Delphi study. Journal of Pain and Symptom Managament. 2016;51(4):652–61. Epub 2015/12/18. doi: 10.1016/j.jpainsymman.2015.11.011 . [DOI] [PubMed] [Google Scholar]
61.Jensdottir AB, Rants M, Hjaltadottir I, Guomundsdottir H, Rook M, Grando V. International comparison of quality indicators in United States, Icelandic and Canadian nursing facilities. International Nursing Review. 2003;50:79–84. doi: 10.1046/j.1466-7657.2003.00163.x [DOI] [PubMed] [Google Scholar]
62.Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review of criteria. International Journal for Quality in Health Care. 2021;33(3). Epub 2021/07/21. doi: 10.1093/intqhc/mzab107 . [DOI] [PMC free article] [PubMed] [Google Scholar]
63.Yabroff KR, Mandelbaltt JS, Ingham J. The quality of medical care at the end-of-life in the USA: Existing barriers and examples of process and outcome measures. Palliative Medicine. 2004;18:202–16. doi: 10.1191/0269216304pm880oa [DOI] [PubMed] [Google Scholar]
64.Schenck AP, Rokoske FS, Durham D, Cagle JG, Hanson LC. Quality measures for hospice and palliative care: Piloting the PEACE measures. Journal of Palliative Medicine. 2014;17:769–75. doi: 10.1089/jpm.2013.0652 [DOI] [PMC free article] [PubMed] [Google Scholar]
65.Seow H, Bainbridge D, Bryant D, Guthrie D, Urowitz S, Zwicker V, et al. The CaregiverVoice survey: A pilot study surveying bereaved caregivers to measure the caregiver and patient experience at end of life. Journal of Palliative Medicine. 2016;19(7):712–9. doi: 10.1089/jpm.2015.0366 . [DOI] [PubMed] [Google Scholar]
66.Bjorkgren MA, Fries BE, Shugarman LR. Testing a RUG-III based case-mix system for home care. Canadian Journal on Aging. 2000;19 (suppl 2):106–25. [Google Scholar]
67.Fries BE, James ML, Martin L, Head MJ, Park PS. A Case-Mix System for Adults with Developmental Disabilities. Health Serv Insights. 2019;12:1178632919856011. Epub 2019/07/03. doi: 10.1177/1178632919856011 . [DOI] [PMC free article] [PubMed] [Google Scholar]
68.Fries BE, Schneider D, Foley WJ, Gavazzi M, Burke R, Cornelius E. Refining a case-mix measure for nursing homes: Resource Utilization Groups (RUG-III). Medical Care. 2001;32(7):668–85. [DOI] [PubMed] [Google Scholar]
69.Guthrie DM, Poss JW. Development of a case-mix funding system for adults with combined vision and hearing loss. BMC Health Services Research. 2013;13:137. doi: 10.1186/1472-6963-13-137 [DOI] [PMC free article] [PubMed] [Google Scholar]
70.Hirdes JP, Fries BE, Morris JN, Ikegami N, Zimmerman D, Dalby DM, et al. Home care quality indicators (HCQIs) based on the MDS-HC. The Gerontologist. 2004;44(5):665–79. doi: 10.1093/geront/44.5.665 [DOI] [PubMed] [Google Scholar]
71.Perlman C, Hirdes JP, Barbaree H, Fries BE, McKillop I, Morris JN, et al. Development of mental health quality indicators (MHQIs) for inpatient psychiatry based on the interRAI mental health assessment. BMC Health Services Research. 2013;13(15):1–12. doi: 10.1186/1472-6963-13-15 [DOI] [PMC free article] [PubMed] [Google Scholar]
72.Morris JN, Berg K, Topinkova E, Gray LC, Schachter E. Developing quality indicators for in-patient post-acute care. BMC Geriatrics. 2018;18(1):161. doi: 10.1186/s12877-018-0842-z . [DOI] [PMC free article] [PubMed] [Google Scholar]
73.Canadian Institute for Health Information. Your Health System: Potentially Inappropriate Use of Antipsychotics in Long-Term Care: Canadian Institute for Health Information; 2019 [Sept. 29, 2021]. https://yourhealthsystem.cihi.ca/hsp/inbrief;jsessionid=Zkd8zMpVT0R77iPDVK9sDlmX.yhs?lang=en#!/indicators/008/potentially-inappropriate-use-of-antipsychotics-in-long-term-care/;mapC1;mapLevel2;/.
74.Jones RN, Hirdes JP, Poss JW, Kelly M, Berg K, Fries BE, et al. Adjustment of nursing home quality indicators. BMC Health Services Research. 2010;10(96):1–8. doi: 10.1186/1472-6963-10-96 [DOI] [PMC free article] [PubMed] [Google Scholar]
75.Dalby DM, Hirdes JP, Fries BE. Risk adjustment methods for home care quality indicators (HCQIs) based on the minimum data set for home care. BMC Health Serv Res. 2005;5:7. doi: 10.1186/1472-6963-5-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
76.Dalby DM, Hirdes JP. The relationship between agency characteristics and quality of home care. Home Health Care Services Quarterly. 2008;27(1):59–74. doi: 10.1300/J027v27n01_04 [DOI] [PubMed] [Google Scholar]
77.Iezzoni LI. Risk adjusting rehabilitation outcomes: an overview of methodologic issues. American Journal of Physical Medicine and Rehabilitation. 2004;83(4):316–26. Epub 2004/03/17. doi: 10.1097/01.phm.0000118041.17739.bb . [DOI] [PubMed] [Google Scholar]
78.Hirdes JP, van Everdingen C, Ferris J, Franco-Martin M, Fries BE, Heikkilä J, et al. The interRAI suite of mental health assessment instruments: An integrated system for the continuum of care. Frontiers in Psychiatry. 2020;10. doi: 10.3389/fpsyt.2019.00926 [DOI] [PMC free article] [PubMed] [Google Scholar]
79.Norton PG, Murray M, Doupe MB, Cummings GG, Poss JW, Squires JE, et al. Facility versus unit level reporting of quality indicators in nursing homes when performance monitoring is the goal. BMJ open. 2014;4(2):e004488. Epub 2014/02/14. doi: 10.1136/bmjopen-2013-004488 . [DOI] [PMC free article] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0266569.r001

Decision Letter 0

Jason Scott

7 Jan 2022

PONE-D-21-34279A multi-stage process to develop quality indicators for community-based palliative care using interRAI dataPLOS ONE

Dear Dr. Guthrie,

In addition to responding to the reviewer comments, I do have some significant concerns of my own that need addressing that link into data sharing and participant consent / ethics. 1. In your data sharing statement you state that the secondary data are available by placing a request with the Canadian Institute for Health Information, and that the data do not belong to the researchers. Can you please clarify exactly what data that you used in this study is held with this organisation? It's currently unclear how you collected data from the Delphi panel participants, which you appeared to convene for the purpose of this study, without having control of the data. It's also unclear why this anonymised data could not be included with the submitted paper as per the journal's policy on data sharing. 2. Linked to this is your statement that consent was not obtained from participants because you only used secondary anonymised data. Please explain how were data collected for a very specific research project such as this, where you describe recruiting Delphi panel participants, without collecting primary data. To me this appears to be primary data collection that would require participant consent. 3. Finally, you need to provide further details about recruitment methods in the manuscript. Reviewers 2 and 3 both mention the lack of clarity relating to recruitment to some extent, and this needs significantly strengthening beyond what the reviewers have asked for. What were the characteristics of this 'group of graduate students'? Eg how many were there, what was their expertise, and how did they relate to the project / research team beyond this exercise? For the Delphi panel participants, how were they identified? Who recruited them? How many people were approached and declined to participate or did not respond to an invitation (you provide a response rate, but that's based on total panel members not the total number of people approached to be panel members)? Was there any potential for bias (based on their characteristics)?

Please submit your revised manuscript by Feb 21 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Jason Scott

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

3. Thank you for stating the following in the Competing Interests section:

“I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. A. Sinnarajah has received research grants (last 5 years) on palliative care (as principal investigator or co-investigator), from the Canadian Institutes of Health Research, MSI Foundation, Canadian Cancer Society, Applied Research in Cancer Control, College of Family Physicians of Canada, Choosing Wisely Alberta, Alberta Innovates Health Research, Alberta Cancer Foundation, Alberta Health Services, University of Calgary, Canadian Frailty Network, Alberta Health and Campus Alberta Health Outcomes and Public Health. He has an academic appointment for palliative care research with Queen’s University and Lakeridge Health (currently), and University of Calgary (last 5 years). Lastly, he is/has paid medical administrative positions with Alberta Health Services and Lakeridge Health. The remaining authors declare that no competing interests exist.”

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I found this article very interesting and I am sure that other readers in the field will also find it so. Observations: The Q.Is were explicit although no real comment was made about the lack of expert by experience or service user input here. What is apparent is the nature of the Q.Is that were dropped. Although these were dropped because of Q.I scores they were focused on psycho social elements. The others were all medical which could be expected however, there are close links between both these elements. I am sure you agree. I understand that this would be a focus of the next stage of research? However, having a rational for this and ensuring that this is a key consideration would have reflected your statement of intention to rectify this. As you say, there was only one family member involved and this could be considered a gap.

Reviewer #2: This manuscript is about a set of quality indicators for community-based palliative care by using interRAI data. Although the author successfully described total 22 QIs developed by panel evaluations.

Major

1. The criteria the author used was slightly different from the criteria in the previous literature the author cited (Importance, Scientific acceptability, Usability, and Feasibility). Furthermore, generalizability of the indicator is also one topic of the criteria for QI. How about the generalizability of these indicators outside the interRAI network? Please discuss.

2. Modified Delphi approach usually includes two or more rounds excluding item generation phase. Why did the author perform only one round? Please discuss.

3. How to determine the domain in this study group? That information would also be useful for readers to capture the process of consensus development.

4. The description about panel members were slightly complex to capture. The relationship between 33 Delphi panel members and 30 PCs and other stakeholders was unclear. Who were these 33 members?

5. Disagreement and panel decision about QI seemed to be complex to understand.

5-i. First, according to the Step 2 in this manuscript, three groups (“discard”, “retain”, or “review”) seemed not to be collectively exhaustive. How to deal with the indicator when the median value was between 7-9 and no agreement in STEP1?

5-ii. Next, in STEP3, if the criteria were considered exact two “review” and two “retain”, which category should the QI be in?

5-iii. Third, how to determine “DROP” or “KEEP” in the Table3? This process was not described in Method section.

5-iv. Finally, Table3 only included appropriateness score for each QI. Disagreement/agreement of each category is necessary to understand the panel decision of each QI precisely. It seemed that the words (“review”, “retain”, and “discard”) were used both for the four criteria and for the individual QI, which could make readers confuse.

6. There is no data about the numerator and denominator of each QI. Furthermore, the detail explanation of each QI should be described.

Minor

1. In supplemental file, “Interpercentile range (IPR)= Upper IPR – Lower IPR” is correct? Upper limit IPR?

Reviewer #3: 1. Line 344: The authors write that “the proposed QIs provide a relatively comprehensive picture of the issues that are 345 widely accepted as key metrics when assessing the quality of PC services[61].” What does “comprehensive” mean in this context? I strongly suggest that you critically discuss the content validity not only of single indicators, but also of the indicator set as a whole (see e.g. Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review of criteria. Int J Qual Health Care. 2021 Jul 31;33(3):mzab107. doi: 10.1093/intqhc/mzab107. PMID: 34282841; PMCID: PMC8325455.)

Specifically, it would be helpful to explicitly report on the degree to which the proposed QI cover the 6 themes of palliative care that you mention at the beginning of your paper, the gaps in your indicator set and whether the proposed QI represent a balanced view of what is important for palliative care patients, e.g. with respect to the 6 themes.

1. Table 1: Why is the distinction between “clinical” and “psychosocial” indicators relevant? I would expect more content-oriented distinctions between indicators, e.g. with respect to the 6 themes of palliative care you mention.

2. While a Delphi study can help to capture judgements by experts, the qualitative reasons put forward for quantitative ratings remain a “black box”. Also, what does the following sentence mean: “It should be noted that the research team utilized the Delphi results as a guide, to support decision-making, but not as a constraint” (line 272 f.)? To enhance transparency of your findings, I strongly suggest reporting for each QI why you dropped or kept them, and how you made that decision.

3. Why did you decide to drop important indicators only because no data was available? If these indicators were important, wouldn’t it be valuable to develop the required data, e.g. expand on existing interRAI assessments?

4. It remains unclear whose quality is to be measured by the indicators – are specific providers accountable for the features measured? Or a regional health system as a whole? Please specify.

5. Please specify what RN (line 114) means – research nurse?

6. What exactly are “knowledge users” (line 160) and why is it important that they did the recruitment of patients and family members?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Atsushi Mizuno

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: 2022-01-03_QI-palliative-care.docx

Click here for additional data file.^{(13.9KB, docx)}

PLoS One. 2022 Apr 7;17(4):e0266569. doi: 10.1371/journal.pone.0266569.r002

Author response to Decision Letter 0

27 Jan 2022

Our responses to the reviewers' comments have been uploaded with our cover letter.

Attachment

Submitted filename: response to reviewers_jan2622.docx

Click here for additional data file.^{(32.3KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0266569.r003

Decision Letter 1

Jason Scott

23 Feb 2022

PONE-D-21-34279R1A multi-stage process to develop quality indicators for community-based palliative care using interRAI dataPLOS ONE

Dear Dr. Guthrie,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Whilst many of the reviewer comments have been addressed, there are some further comments requiring attention. These relate specifically to the processes around collecting, analysing and interpreting the data.

Please submit your revised manuscript by Apr 09 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Jason Scott

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Reviewer #1: Following amendments reviewer concerns have been addressed. It is evident that manuscript authors have considered all comments to some extent. Although accepted, I would still be interested in seeing a clearer statement about ethics. Line 160 identifies KUs recruited those in receipt of PC and their families. Line 161 because it was inappropriate for the research team to do so. Line 165 identifies 8 KUs on the research team? I imagine KUs have stringent ethical probity however, this is not declared and lines 160, 161 and 165 appear to contradict each other.

Reviewer #2: Although I appreciate efforts about point-by-point responses and the comments about the uniqueness and potential usefulness of this indicators for routine and clinical practice, there are still several concerns about the process in this manuscript.

1. As I described, the generalizability of this results should be discussed. First of all, the author did not discuss about actual dataset of interRAI but interRAI data elements (e.g. data components) only. Thus, these indicators by authors are just only agreement of these elements could be useful for palliative care through consensus strategy. Therefore, the author should describe not only advantage of these indicators but also disadvantages such as lack of information compared with the previous literatures. The advantages of these indicators could be more contrasted. For example, “Prevalence of falls” is unique as indicators for palliative care, especially under sub-theme of symptom management.

2. As far as I am correct, the panel members for this unique Delphi manuscript, especially for Phase III seemed to be completely different from Phase I and II. This should be clarified not only in Results section but also in Method section. As it has not still been improved, it could make readers confuse. The selection bias of panel members should also be described in limitation of this study.

3. The word about “domain” and “theme” could be used carefully not to make readers confuse in this manuscript. It would be better to describe how to determine these domains and themes by this study team (including all Phase I, II and III).

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

PLoS One. 2022 Apr 7;17(4):e0266569. doi: 10.1371/journal.pone.0266569.r004

Author response to Decision Letter 1

14 Mar 2022

Reviewer #1:

Following amendments reviewer concerns have been addressed. It is evident that manuscript authors have considered all comments to some extent. Although accepted, I would still be interested in seeing a clearer statement about ethics. Line 160 identifies KUs recruited those in receipt of PC and their families. Line 161 because it was inappropriate for the research team to do so. Line 165 identifies 8 KUs on the research team? I imagine KUs have stringent ethical probity however, this is not declared and lines 160, 161 and 165 appear to contradict each other.

We apologize if this was confusing. The 8 KUs on our team acted as a “conduit” to families and individuals receiving palliative care and helped to link us to these individuals. The KUs and our team felt that it would be inappropriate for us, as a research team, to reach out directly to families/care recipients. Instead, the KUs helped to share our project with potential participants who could then contact us IF they were interested. For example, the KUs helped us by posting flyers on their web pages or by emailing study information to potential participants. In this sense, they were not participants themselves, but were simply helping us to identify individuals who were willing to take part in the research. We have modified line 161 and added one new sentence (line 164-165) to help clarify this process.

Reviewer #2:

To clarify, the “interRAI dataset” is made up of the data elements within the assessment and nothing else, so the two things are identical. To be more explicit about which QIs have been cited in the literature, but which we could not measure, we have added references to the literature for those specific QIs where we mention them on lines 376-382 (please note that these added references do not show up in the “track changes” in the word document, but include ref #18, 63, 64 and 65).We feel that we have already described the unique advantages of the QIs derived using the interRAI dataset in the discussion section and feel it would make the discussion somewhat repetitious to add further comment on this.

It was possible for decision makers participating in the Delphi panel to have also taken part in an interview/focus group during Phase I. We did not explicitly exclude them from the Delphi panel. We have modified lines 242-243 in the methods, and lines 291-292 in the results, to make this more explicit.

We do not feel that there was selection bias operating in the panel members. We had 6 individuals who consented, provided a demographic questionnaire but then did not complete the Delphi exercise. They were very similar to Delphi panel members in terms of age, gender, years of experience and clinical background. We have added additional text to describe this group (lines 294-298).

We see these as two distinct concepts. We have used the term “domain” when discussing the QIs and the extent to which the indicators capture important information to assess quality. We found one instance where the term “domain” likely was inappropriate (line 329) and switched it to the word “area” to be more clear.

On the other hand, we have used the term “theme” specifically as it relates to the qualitative analysis of the workshop results, the focus groups, and interviews. We have added some text (line 153) to make it more clear with respect to the analysis of the workshop data. A quick search confirms that the term “theme” has always been used in this context in the manuscript. This is the accepted wording used in qualitative research to highlight a topic of discussion that was raised by multiple participants. It reflects the way in which the key issues raised by participants “emerge” during the qualitative analysis. As such, we feel that these two terms are indeed unique and have been used appropriately throughout the paper.

Attachment

Submitted filename: response to reviewers_mar322.docx

Click here for additional data file.^{(22.2KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0266569.r005

Decision Letter 2

Jason Scott

23 Mar 2022

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

PONE-D-21-34279R2

Dear Dr. Guthrie,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jason Scott

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #2: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Reviewer #2: The authors have answered all the comments and improved manuscripts for readers appropriately. As I asked previously, I would recommend the authors to describe how to make "domain" of these quality indicators.

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

PLoS One. doi: 10.1371/journal.pone.0266569.r006

Acceptance letter

Jason Scott

29 Mar 2022

PONE-D-21-34279R2

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

Dear Dr. Guthrie:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jason Scott

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Table. Operational definitions for each of the 27 quality indicators (QIs).

(DOCX)

Click here for additional data file.^{(32.4KB, docx)}

S1 File. An example of applying the interpercentile range adjusted for asymmetry (IPRAS).

(DOCX)

Click here for additional data file.^{(23.6KB, docx)}

Attachment

Submitted filename: 2022-01-03_QI-palliative-care.docx

Click here for additional data file.^{(13.9KB, docx)}

Attachment

Submitted filename: response to reviewers_jan2622.docx

Click here for additional data file.^{(32.3KB, docx)}

Attachment

Submitted filename: response to reviewers_mar322.docx

Click here for additional data file.^{(22.2KB, docx)}

Data Availability Statement

The data can be found at the DOI: https://doi.org/10.5683/SP3/8U8B83.

[pone.0266569.ref001] 1.Meier DE. Increased access to palliative care and hospice services: opportunities to improve value in health care. The Milbank Quarterly. 2011;89(3):343–80. doi: 10.1111/j.1468-0009.2011.00632.x [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref002] 2.Alberta Health Services. Palliative and End-of-life Care: Alberta Provincial Framework. Calgary, AB: 2014.

[pone.0266569.ref003] 3.Roulston E. Canadians’ View of Palliative Care: National Online Survey. Toronto, ON: Canadian Partnership Against Cancer/Ipsos Public Affairs, 2016. [Google Scholar]

[pone.0266569.ref004] 4.Health Quality Ontario. Palliative care at the end of life. Toronto: Health Quality Ontario, 2016.

[pone.0266569.ref005] 5.Gagnon B, Nadeau L, Scott S, Dumont S, MacDonald N, Aubin M, et al. The association between home palliative care services and quality of end-of-life care indicators in the province of Quebec. Journal of Pain and Symptom Management. 2015;50(1):48–58. doi: 10.1016/j.jpainsymman.2014.12.012 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref006] 6.Seow H, Sutradhar R, McGrail K, Fassbender K, Pataky R, Lawson B, et al. End-of-life cancer care: Temporal association between homecare nursing and hospitalizations. Journal of Palliative Medicine. 2016;19(3):263–70. doi: 10.1089/jpm.2015.0229 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref007] 7.Cohen J, Hermans K, Dupont C, Van den Block L, Deliens L, Leemans K. Nationwide evaluation of palliative care (Q-PAC study) provided by specialized palliative care teams using quality indicators: Large variations in quality of care. Palliative Medicine. 2021;35(8):1525–41. Epub 2021/06/01. doi: 10.1177/02692163211019881 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref008] 8.Walker H, Anderson M, Farahati F, Howell D, Librach SL, Husain A, et al. Resource use and cost of end-of-life/palliative care: Ontario adult cancer patients dying during 2002 and 2003. Journal of Palliative Care. 2011;27(2):79–88. [PubMed] [Google Scholar]

[pone.0266569.ref009] 9.Sussman J, Barbera L, Bainbridge D, Howell D, Yang J, Husain A, et al. Health system characteristics of quality care delivery: a comparative case study examination of palliative care for cancer patients in four regions in Ontario, Canada. Palliative Medicine. 2012;26(4):322–35. doi: 10.1177/0269216311416697 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref010] 10.Barbera L, Paszat L, Chartier C. Indicators of poor quality end-of-life cancer care in Ontario. Journal of Palliative Care. 2006;22(1):12–7. [PubMed] [Google Scholar]

[pone.0266569.ref011] 11.Sussman J, Barbera L, Bainbridge D, Howell D, Yang J, Husain A, et al. Health system characteristics of quality care delivery: a comparative case study examination of palliative care for cancer patients in four regions in Ontario, Canada. Palliative Medicine. 2011;0(00):1–14. doi: 10.1177/0269216311416697 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref012] 12.Grunfeld E, Lethbridge L, Dewar R, Lawson B, Paszat L, Johnston G, et al. Towards using administrative databases to measure population-based indicators of quality of end-of-life care: Testing the methodology. Palliative Medicine. 2006;20:769–77. doi: 10.1177/0269216306072553 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref013] 13.Cohen J, Leemans K. How can you prove that you are delivering good care? Monitoring the quality of palliative care using quality indicators. European Journal of Palliative Care. 2014;21(5):228–31. [Google Scholar]

[pone.0266569.ref014] 14.Zimmerman DR. Improving nursing home quality of care through outcomes data: the MDS quality indicators. Int J Geriatr Psychiatry. 2003;18(3):250–7. Epub 2003/03/19. doi: 10.1002/gps.820 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref015] 15.Currow DC, Allingham S, Yates P, Johnson C, Clark K, Eagar K. Improving national hospice/palliative care service symptom outcomes systematically through point-of-care data collection, structured feedback and benchmarking. Supportive Care in Cancer. 2015;23(2):307–15. Epub 2014/07/27. doi: 10.1007/s00520-014-2351-8 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref016] 16.Ettorchi-Tardy A, Levif M, Michel P. Benchmarking: A method for continuous quality improvement in health. Healthcare Policy. 2012;7:e101–e19. [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref017] 17.Raijmakers N, Galushko M, Domeisen F, Beccaro M, Hagelin CL, Lindqvist O, et al. Quality indicators for care of cancer patients in their last days of life: literature update and experts’ evaluation. Journal of Palliative Medicine. 2012;15(3):308–16. doi: 10.1089/jpm.2011.0393 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref018] 18.Claessen SJJ, Francke AL, Belarbi HE, Pasman RW, van der Putten MJA, Deliens L. A new set of quality indicators for palliative care: process and results of the development trajectory. Journal of Pain and Symptom Management. 2011;42(2):169–82. doi: 10.1016/j.jpainsymman.2010.10.267 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref019] 19.Leemans K, Cohen J, Francke AL, Stichele RV, Claessen SJJ, Van den Block L, et al. Towards a standardized method of developing quality indicators for palliative care: protocol of the Quality indicators for Palliative Care (Q-PAC) study. BMC Palliative Care. 2013;12(6). doi: 10.1186/1472-684X-12-6 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref020] 20.Lorenz KA, Rosenfeld K, Wenger N. Quality indicators for palliative and end-of-life care in vulnerable elders. Journal of the American Geriatrics Society. 2007;55(S2):S318–S26. doi: 10.1111/j.1532-5415.2007.01338.x [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref021] 21.Dy SM, Kiley K, Ast K, Lupu D, Norton S, Mcmillan SC, et al. Measuring what matters: top-ranked quality indicators for hospice and palliative care from the American Academy of Hospice and Palliative Medicine and Palliative Nurses Association. Journal of Pain and Symptom Management. 2015;49(4):773–81. doi: 10.1016/j.jpainsymman.2015.01.012 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref022] 22.De Schreye R, Houttekier D, Deliens L, Cohen J. Developing indicators of appropriate and inappropriate end-of-life care in people with Alzheimer’s disease, cancer or chronic obstructive pulmonary disease for population-level administrative databases: A RAND/UCLA appropriateness study. Palliative Medicine. 2017;31(10):932–45. doi: 10.1177/0269216317705099 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref023] 23.van Riet Paap J, Vernooij-Dassen M, Droes R, Radbruch L, Vissers K, Engels Y. Consensus on quality indicators to assess the organisation of palliative cancer and dementia care applicable across national healthcare systems and selected by international experts. BMC Health Services Research. 2014;14:396. doi: 10.1186/1472-6963-14-396 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref024] 24.D’Angelo D, Mastroianni C, Vellone E, Alvaro R, Casale G, Latina R, et al. Palliative care quality indicators in Italy. What do we evaluate? Supportive Care in Cancer. 2012;20:1983–9. doi: 10.1007/s00520-011-1301-y [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref025] 25.Woitha K, Van Beek K, Ahmed N, Hasselaar J, Mollard JM, Colombet I, et al. Development of a set of process and structure indicators for palliative care: the Europall project. BMC Health Services Research. 2012;12(1):381. doi: 10.1186/1472-6963-12-381 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref026] 26.Bone AE, Evans CJ, Etkind SN, Sleeman KE, Gomes B, Aldridge M, et al. Factors associated with older people’s emergency department attendance towards the end of life: a systematic review. European Journal of Public Health. 2019;29(1):67–74. doi: 10.1093/eurpub/cky241 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref027] 27.Earle CC, Park ER, Lai B, Weeks JC, Ayanian JZ, Block S. Identifying potential indicators of the quality of end-of-life cancer care from administrative data. Journal of Clinical Oncology. 2003;21(6):1133–8. doi: 10.1200/JCO.2003.03.059 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref028] 28.Barbera L, Seow H, Sutradhar R, Chu A, Burge F, Fassbender K, et al. Quality of end-of-life cancer care in Canada: a retrospective four-province study using administrative health care data. Current Oncology. 2015;22(5):341–55. doi: 10.3747/co.22.2636 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref029] 29.World Health Organization. Global Atlas of Palliative Care at the End of Life. Geneva, Switzerland: World Health Organization, 2014. [Google Scholar]

[pone.0266569.ref030] 30.Lynn J. Living long in fragile health: the new demographics shape end of life care. Hastings Center Report. 2005;35(7):S14–S8. doi: 10.1353/hcr.2005.0096 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref031] 31.Rocker G, Downar J, Morrison S. Palliative care for chronic illness: driving change. Canadian Medical Association Journal. 2016;188:e493–e8. doi: 10.1503/cmaj.151454 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref032] 32.Carpenter GI, Hirdes JP. Using interRAI assessment systems to measure and maintain quality in long term care. A Good Life in Old Age? Monitoring and Improving Quality in Long-term Care. Paris, France: OECD/European Commission; 2013. p. 93–139. [Google Scholar]

[pone.0266569.ref033] 33.Steel K, Jonsson P, DuPasquier JN, Gilgen R, Hirdes JP, Schroll M, et al. Systems of care for frail older persons. Transactions of the American Clinical and Climatological Association. 1999;110:30–7. [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref034] 34.Heckman G, Gray LC, Hirdes JP. Addressing health care needs for frail seniors in Canada: The role of interRAI instruments. Canadian Geriatrics Society Journal of CME. 2013;3. [Google Scholar]

[pone.0266569.ref035] 35.van der Roest HG, van Eenoo L, van Lier LI, Onder G, Garms-Homolova V, Smit JH, et al. Development of a novel benchmark method to identify and characterize best practices in home care across six European countries: design, baseline, and rationale of the IBenC project. BMC Health Services Research. 2019;19(1):310. Epub 2019/05/17. doi: 10.1186/s12913-019-4109-y . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref036] 36.Foebel AD, van Hout HP, van der Roest HG, Topinkova E, Garms-Homolova V, Frijters D, et al. Quality of care in European home care programs using the second generation interRAI Home Care Quality Indicators (HCQIs). BMC Geriatrics. 2015;15:148. doi: 10.1186/s12877-015-0146-5 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref037] 37.Chartier LB, Ovens H, Hayes E, Davis B, Calder L, Schull M, et al. Improving quality of care through a mandatory provincial audit program: Ontario’s emergency department return visit quality program. Annals of Emergency Medicine. 2021;77(2):193–202. Epub 2020/11/18. doi: 10.1016/j.annemergmed.2020.09.449 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref038] 38.Mor V, Angelelli J, Jones R, Roy J, Moore T, Morris JN. Inter-rater reliability of nursing home quality indicators in the US. BMC Health Services Research. 2003;3:20. doi: 10.1186/1472-6963-3-20 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref039] 39.Berg K, Mor V, Morris J, Murphy K, Moore T, Harris Y. Identification and evaluation of existing nursing homes quality indicators. Health Care Financing Review. 2002;23(4):19–36. [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref040] 40.Karon SL, Sainfort F, Zimmerman D. Stability of nursing home quality indicators over time. Medical Care. 2000;37(6):570–9. [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref041] 41.Zimmerman DR, Karon SL, Arling G, Ryther Clark B, Collins T, Ross R, et al. Development and testing of nursing home quality indicators. Health Care Financing Review. 1995;16(4):104–27. [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref042] 42.Harman LE, Guthrie DM, Cohen J, Declercq A, Fisher K, Goodridge D, et al. Potential quality indicators for seriously ill home care clients: a cross-sectional analysis using Resident Assessment Instrument for Home Care (RAI-HC) data for Ontario. BMC Palliative Care. 2019;18:3. doi: 10.1186/s12904-018-0389-y [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref043] 43.Guthrie DM, Harman LE, Barbera L, Burge F, Lawson B, McGrail K, et al. Quality indicator rates for seriously ill home care clients: Analysis of Resident Assessment Instrument for Home Care Data in six Canadian provinces. Journal of Palliative Medicine. 2019;22(11):1346–56. Epub 2019/05/17. doi: 10.1089/jpm.2019.0022 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref044] 44.Williams N, Boumans N, Luymes N, White NE, Lemonde M, Guthrie DM. What should be measured to assess the quality of community-based palliative care? Results from a collaborative expert workshop. Palliative and Supportive Care. 2021;1–7. doi: 10.1017/S1478951521000791 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref045] 45.Luymes N, Williams N, Garrison L, Goodridge D, Silveira M, Guthrie DM. “The system is well intentioned, but complicated and fallible”: Interviews with caregivers and decision makers about palliative care in Canada. BMC Palliative Care. 2021;20:149. doi: 10.1186/s12904-021-00843-x [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref046] 46.Ahluwalia SC, Chen C, Raaen L, Motala A, Walling AM, Chamberlin M, et al. A Systematic Review in Support of the National Consensus Project Clinical Practice Guidelines for Quality Palliative Care, Fourth Edition. Journal of Pain and Symptom Managament. 2018;56(6):831–70. doi: 10.1016/j.jpainsymman.2018.09.008 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref047] 47.Pasman HRW, Brandt HE, Deliens L, Francke AL. Quality indicators for palliative care: a systematic review. Journal of Pain and Symptom Management. 2009;38(1):145–56. doi: 10.1016/j.jpainsymman.2008.07.008 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref048] 48.Cohen SR, Leis R. What determines the quality of life for terminally-ill cancer patients from their own perspective? Journal of Palliative Care. 2002;18(1):48–58. [PubMed] [Google Scholar]

[pone.0266569.ref049] 49.National Consensus Project for Quality Palliative Care. Clinical practice guidelines for quality palliative care, third edition. Pittsburgh, PA: 2013 2013. Report No.

[pone.0266569.ref050] 50.Canadian Institute for Health Information. Profile of Clients in Home Care, 2019–20202021 Mar.8, 2021. https://www.cihi.ca/en/quick-stats?utm_medium=email&utm_source=ebulletin&utm_campaign=hcc-ebulletin-march-2021&utm_content=quick-stats-en&emktg_lang=en&emktg_order=2.

[pone.0266569.ref051] 51.Hogeveen SE, Chen J, Hirdes JP. Evaluation of data quality of interRAI assessments in home and community care. BMC Medical Informatics and Decision Making. 2017;17(1):150. doi: 10.1186/s12911-017-0547-9 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref052] 52.Martin L, Poss JW, Hirdes JP, Jones RN, Stones MJ, Fries BE. Predictors of a new depression diagnosis among older adults admitted to complex continuing care: implications for the Depression Rating Scale (DRS). Age and Ageing. 2008;37(1):51–6. doi: 10.1093/ageing/afm162 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref053] 53.Fries BE, Simon SE, Morris JN, Flodstrom C, Bookstein FL. Pain in US nursing homes: validating a pain scale for the Minimum Data Set. The Gerontologist. 2001;41(2):173–9. doi: 10.1093/geront/41.2.173 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref054] 54.Morris JN, Fries BE, Mehr DR, Hawes C, Mor V, Lipsitz L. MDS Cognitive Performance Scale. Journals of GerontologySeries A, Biological Sciences and Medical Sciences. 1994;49(4):M174–M82. doi: 10.1093/geronj/49.4.m174 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref055] 55.Guthrie DM, Williams N, Beach C, Maxwell CJ, Mills D, Mitchell L, et al. Development and validation of Caregiver Risk Evaluation (CaRE): A new algorithm to screen for caregiver burden. Journal of applied gerontology: the official journal of the Southern Gerontological Society. 2021;40:731–41. doi: 10.1177/0733464820920102 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref056] 56.Poss J, Murphy KM, Woodbury MG, Orsted H, Stevenson K, Williams G, et al. Development of the interRAI Pressure Ulcer Risk Scale (PURS) for use in long-term care and home care settings. BMC Geriatrics. 2010;10(1):67. doi: 10.1186/1471-2318-10-67 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref057] 57.Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA appropriateness method user’s manual. Santa Monica, CA: 2001. [Google Scholar]

[pone.0266569.ref058] 58.Junger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. Epub 2017/02/14. doi: 10.1177/0269216317690685 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref059] 59.Seow H, Snyder CF, Mularski RA, Shugarman LR, Kutner JS, Lorenz KA, et al. A framework for assessing quality indicators for cancer care at the end of life. Journal of Pain and Symptom Management. 2009;38(6):903–12. doi: 10.1016/j.jpainsymman.2009.04.024 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref060] 60.Nakazawa Y, Kato M, Yoshida S, Miyashita M, Morita T, Kizawa Y. Population-based quality indicators for palliative care programs for cancer patients in Japan: A Delphi study. Journal of Pain and Symptom Managament. 2016;51(4):652–61. Epub 2015/12/18. doi: 10.1016/j.jpainsymman.2015.11.011 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref061] 61.Jensdottir AB, Rants M, Hjaltadottir I, Guomundsdottir H, Rook M, Grando V. International comparison of quality indicators in United States, Icelandic and Canadian nursing facilities. International Nursing Review. 2003;50:79–84. doi: 10.1046/j.1466-7657.2003.00163.x [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref062] 62.Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review of criteria. International Journal for Quality in Health Care. 2021;33(3). Epub 2021/07/21. doi: 10.1093/intqhc/mzab107 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref063] 63.Yabroff KR, Mandelbaltt JS, Ingham J. The quality of medical care at the end-of-life in the USA: Existing barriers and examples of process and outcome measures. Palliative Medicine. 2004;18:202–16. doi: 10.1191/0269216304pm880oa [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref064] 64.Schenck AP, Rokoske FS, Durham D, Cagle JG, Hanson LC. Quality measures for hospice and palliative care: Piloting the PEACE measures. Journal of Palliative Medicine. 2014;17:769–75. doi: 10.1089/jpm.2013.0652 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref065] 65.Seow H, Bainbridge D, Bryant D, Guthrie D, Urowitz S, Zwicker V, et al. The CaregiverVoice survey: A pilot study surveying bereaved caregivers to measure the caregiver and patient experience at end of life. Journal of Palliative Medicine. 2016;19(7):712–9. doi: 10.1089/jpm.2015.0366 . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref066] 66.Bjorkgren MA, Fries BE, Shugarman LR. Testing a RUG-III based case-mix system for home care. Canadian Journal on Aging. 2000;19 (suppl 2):106–25. [Google Scholar]

[pone.0266569.ref067] 67.Fries BE, James ML, Martin L, Head MJ, Park PS. A Case-Mix System for Adults with Developmental Disabilities. Health Serv Insights. 2019;12:1178632919856011. Epub 2019/07/03. doi: 10.1177/1178632919856011 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref068] 68.Fries BE, Schneider D, Foley WJ, Gavazzi M, Burke R, Cornelius E. Refining a case-mix measure for nursing homes: Resource Utilization Groups (RUG-III). Medical Care. 2001;32(7):668–85. [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref069] 69.Guthrie DM, Poss JW. Development of a case-mix funding system for adults with combined vision and hearing loss. BMC Health Services Research. 2013;13:137. doi: 10.1186/1472-6963-13-137 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref070] 70.Hirdes JP, Fries BE, Morris JN, Ikegami N, Zimmerman D, Dalby DM, et al. Home care quality indicators (HCQIs) based on the MDS-HC. The Gerontologist. 2004;44(5):665–79. doi: 10.1093/geront/44.5.665 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref071] 71.Perlman C, Hirdes JP, Barbaree H, Fries BE, McKillop I, Morris JN, et al. Development of mental health quality indicators (MHQIs) for inpatient psychiatry based on the interRAI mental health assessment. BMC Health Services Research. 2013;13(15):1–12. doi: 10.1186/1472-6963-13-15 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref072] 72.Morris JN, Berg K, Topinkova E, Gray LC, Schachter E. Developing quality indicators for in-patient post-acute care. BMC Geriatrics. 2018;18(1):161. doi: 10.1186/s12877-018-0842-z . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref073] 73.Canadian Institute for Health Information. Your Health System: Potentially Inappropriate Use of Antipsychotics in Long-Term Care: Canadian Institute for Health Information; 2019 [Sept. 29, 2021]. https://yourhealthsystem.cihi.ca/hsp/inbrief;jsessionid=Zkd8zMpVT0R77iPDVK9sDlmX.yhs?lang=en#!/indicators/008/potentially-inappropriate-use-of-antipsychotics-in-long-term-care/;mapC1;mapLevel2;/.

[pone.0266569.ref074] 74.Jones RN, Hirdes JP, Poss JW, Kelly M, Berg K, Fries BE, et al. Adjustment of nursing home quality indicators. BMC Health Services Research. 2010;10(96):1–8. doi: 10.1186/1472-6963-10-96 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref075] 75.Dalby DM, Hirdes JP, Fries BE. Risk adjustment methods for home care quality indicators (HCQIs) based on the minimum data set for home care. BMC Health Serv Res. 2005;5:7. doi: 10.1186/1472-6963-5-7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref076] 76.Dalby DM, Hirdes JP. The relationship between agency characteristics and quality of home care. Home Health Care Services Quarterly. 2008;27(1):59–74. doi: 10.1300/J027v27n01_04 [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref077] 77.Iezzoni LI. Risk adjusting rehabilitation outcomes: an overview of methodologic issues. American Journal of Physical Medicine and Rehabilitation. 2004;83(4):316–26. Epub 2004/03/17. doi: 10.1097/01.phm.0000118041.17739.bb . [DOI] [PubMed] [Google Scholar]

[pone.0266569.ref078] 78.Hirdes JP, van Everdingen C, Ferris J, Franco-Martin M, Fries BE, Heikkilä J, et al. The interRAI suite of mental health assessment instruments: An integrated system for the continuum of care. Frontiers in Psychiatry. 2020;10. doi: 10.3389/fpsyt.2019.00926 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0266569.ref079] 79.Norton PG, Murray M, Doupe MB, Cummings GG, Poss JW, Squires JE, et al. Facility versus unit level reporting of quality indicators in nursing homes when performance monitoring is the goal. BMJ open. 2014;4(2):e004488. Epub 2014/02/14. doi: 10.1136/bmjopen-2013-004488 . [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

Dawn M Guthrie

Nicole Williams

Cheryl Beach

Emma Buzath

Joachim Cohen

Anja Declercq

Kathryn Fisher

Brant E Fries

Donna Goodridge

Kirsten Hermans

John P Hirdes

Hsien Seow

Maria Silveira

Aynharan Sinnarajah

Susan Stevens

Peter Tanuseputro

Deanne Taylor

Christina Vadeboncoeur

Tracy Lyn Wityk Martin

Roles

Abstract

Background

Methods

Results

Conclusions

Introduction

Materials and methods

Fig 1. Summary of the steps in the QI development process.

Phase I: Qualitative input from key stakeholders

Phase II: Defining potential QIs

Table 1. List of the 27 preliminary QIs reviewed by the expert panel, how they relate to the 6 themes identified in Phase I and which interRAI assessment can be used to generate the QI.

Phase III: Modified Delphi panel to evaluate preliminary QIs

Determining consensus among raters

Results

Table 2. Demographic characteristics of individuals who participated in the expert panel.

Table 3. Summary of scores from the expert panel and final decision for each of the proposed QIs.

Discussion

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Jason Scott

Roles

Author response to Decision Letter 0

Decision Letter 1

Jason Scott

Roles

Author response to Decision Letter 1

Decision Letter 2

Jason Scott

Roles

Acceptance letter

Jason Scott

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases