Relevance of assessment items in community paramedicine home visit programmes: results of a modified Delphi study

Abstract

Objective

Guidelines for a structured assessment in community paramedicine home visit programmes have not been established and evidence to inform their creation is lacking. We sought to investigate the relevance of assessment items to the practice of community paramedics according to a pre-established clarity-utility matrix.

Design

We designed a modified-Delphi study consisting of predetermined thresholds for achieving consensus, number of rounds of for scoring items, a defined meeting and discussion process, and a sample of participants that was purposefully representative.

Setting and participants

We established a panel of 26 community paramedics representing 20 municipal paramedic services in Ontario, Canada. The sample represented a majority of paramedic services within the province that were operating a community paramedicine home visit programme.

Measures

Drawing from a bank of standardised assessment items grouped according to domains aligned with the International Classification on Functioning, Disability, and Health taxonomy, 64 previously pilot-tested assessment items were scored according to their clarity (being free from ambiguity and easy to understand) and utility (being valued in care planning or case management activities). Assessment items covered a broad range of health, social and environmental domains. To conclude scoring rounds, assessment items that did not achieve consensus for relevance to assessment practices were discussed among participants with opportunities to modify assessment items for subsequent rounds of scoring.

Results

Resulting from the first round of scoring, 54 assessment items were identified as being relevant to assessment practices and 3 assessment items were removed from subsequent rounds. The remaining 7 assessment items were modified, with some parts removed from the final items that achieved consensus in the final rounds of scoring.

Conclusion

A broadly representative panel of community paramedics identified consensus for 61 assessment items that could be included in a structured, multidomain, assessment instrument for guiding practice in community paramedicine home visit programmes.

Trail registration number

ISRCTN58273216.

Strengths and limitations of this study.

• A broadly representative panel of frontline community paramedics participated in a multiround process to find consensus.

• Community paramedics were able to use a utility-clarity matrix to determine the relevance of assessment items included in a standardise assessment instrument designed for home visiting programmes.

• The modified Delphi methodology enabled frontline community paramedics to navigate the tension that exists between standardisation and adaptation to local and contextual criteria.

• By investigating the relevance of assessment items in community paramedicine home visit programmes, the findings can contribute evidence towards clinical utility and validation of a standardised assessment instrument that is fit for purpose.

Introduction

Paramedics, as mobile healthcare providers with limited access to diagnostic tools, use social and environmental observation, physical examination and oral history taking to understand patient condition and make treatment decisions.¹ In high-acuity emergencies, paramedics must quickly identify and treat threats to life and limb.¹ In lower acuity situations, paramedics must consider multiple pathologies that may be contributing to a patient’s condition through a more comprehensive and detailed problem-based approach.² In all cases, paramedics must assess patients and the surrounding environment thoroughly to deliver appropriate patient care and maintain safety.^{1 3}

A structured process for patient assessment is common in paramedicine and other emergency settings.^{1 4} Structured assessment processes have been identified as important to guiding practice, reducing errors or adverse events, and contributing to accuracy requirements that can improve patient care in many settings.^{1 5–8} Structured frameworks for assessments might include mnemonics or other tools or prompts to help ensure completeness and that findings are relevant to clinical practice.^{5 8} Common terminology and standardised documentation are helpful when communicating assessment findings with other members of the care team.⁷

Community paramedicine is an emerging area of paramedic practice where paramedics with broadened skillsets provide low-acuity and preventative care, often collaborating with other members of patients’ care teams in community settings.^{9 10} In community paramedicine home visit programmes, paramedics visit patients at home to identify, treat and conduct referrals for emerging health and social needs.^{10 11} This represents an extension of low acuity paramedic practice, with new aspects of patient assessment required for improved care integration, care planning and case management.^{10 12} While consistent, structured processes for patient assessment in paramedicine have long been in place,¹ how they have been redirected or altered for application in community paramedicine settings is not clear. Broad guidelines for structuring patient assessment in community paramedicine settings have not been established and concerns have been identified about potentially inconsistent assessment practices within and across regional jurisdictions.^{11 12}

The purpose of the Community Paramedicine Assessment Matters study was to explore consensus on the most relevant assessment items that should be included in structured, multidimensional, comprehensive, patient assessment practices for community paramedicine home visit programmes. Such assessment practices should capture the health, social and environmental considerations needed to direct community paramedic care planning and case management activities. In the absence of other sources of evidence, we expected that expert opinion would provide the best source of information¹³ needed to identify assessment items that might provide clarity and utility in clinical practice and determine what matters during an assessment conducted by a community paramedic in a community paramedicine home visit programme.

Methods

Study design

A modified Delphi process was used consisting of multiple iterations of online questionnaires and web-based discussions with an expert panel of community paramedics from one Canadian province. The questionnaires asked participants to evaluate individual assessment items for relevance to practice.

Preceding instrumentation

Assessment items (as grouped according to assessment domains) were derived from an instrument that had been pilot tested in multiple sites through the Common Assessments for Repeated Paramedic Encounters (CARPE) study (ISRCTN 58273216). Results of the CARPE study have been published elsewhere.¹⁴

Derivation of the CARPE assessment instrument included a literature review, preliminary modified Delphi study, and environmental scan of existing practices.^{10 11 15} The CARPE assessment instrument was constructed in accordance with other standardised assessment instruments that have been created by interRAI—an international group of researchers and clinicians.⁷ interRAI instruments are designed as an integrated assessment system to cover the continuum of care settings, have been implemented in over 30 countries, and align with the taxonomies established by the WHO’s International Classification on Functioning, Disability, and Health.7 11 16–18 The CARPE assessment instrument included 64 assessment items grouped according to 14 assessment domains covering an array of health, social and environmental factors. Between each iteration of questionnaires, web-based discussions were hosted to discuss results.

Patient and public involvement

Patients and members of the public were not involved in this study.

Recruitment and study orientation

All paramedic services in Ontario providing home visit programmes, identified in a 2019 provincial report on community paramedicine,¹⁹ were invited to participate in the study. Each paramedic service was allowed a maximum of three participants. A minimum of 24 participants with representation from at least 50% of Ontario paramedic services with home visit programmes was considered to be representative. We could allow for a maximum of 36 participants due to logistics and budget. Recruitment of participants was facilitated by the Ontario Community Paramedicine Secretariat. Selection of participants (within the minimum and maximum number) was based on maximising the number of representative services.

To participate, community paramedics needed to be certified as critical, advanced or primary care and be working in a community paramedicine home visit programme that included patient assessment as part of their regular clinical practice. Exclusion criteria were defined for paramedics who had an organisational rank of commander or higher unless they could demonstrate that patient assessment was a regular component of their assigned duties. Paramedics in acting or temporary administrative roles, or those who assume those roles over the course of the study were not be excluded.

Interested participants were invited to participate in an information and orientation session where they were presented with an outline of the aims and structure for the study. Prior to beginning the first round of scoring, participants provided written consent. Participants received gift cards of increasing value for each round that they participated in.

Finding consensus

We investigated two dimensions of relevance—clarity and utility—during each round of the modified Delphi process. Clarity of an item described the ease to which the information provided through an assessment item could be understood by the community paramedic and was free from ambiguity. Utility of an assessment item reflected whether or not the item was considered to be useful to the community paramedic’s role in care planning or case management. The question of utility investigated whether or not actionable information would be generated by an assessment item. The rationale for including two dimensions to relevance was to establish a relationship between any individual assessment item included in an assessment instrument and the practice of assessment to inform care planning and case management activities. For example, if an assessment item is clearly understood (high clarity) but does not provide actionable information (low utility) then it is not likely contributing to patient assessment. Alternatively, if an assessment item cannot be clearly understood (low clarity), even if it is determined to be actionable (high utility), then how it is acted on may vary from one situation to another. If an item is neither clearly understood (low clarity) nor actionable (low utility), then it should not be considered as relevant to assessment practice. For any assessment item to be considered relevant, it would need to satisfy the conditions according to the two dimensions identified (illustrated in figure 1).

Figure 1.

Matrix of clarity and utility used to define relevance of assessment items.

For an item to reach consensus, two-thirds (66.7%) of responses needed to either fall in the relevant or not relevant portions of the matrix illustrated in figure 1. Assessment items were grouped according to assessment domains. If no items within a domain were identified as being relevant, the domain was removed from subsequent rounds. Secondary analysis was conducted to review assessment items where greater than one-third (33.3%) of responses were within the central variable of either of the individual dimensions (somewhat useful or moderately understandable).

Delphi rounds

To help prevent participant fatigue and ensure ongoing participation, it was decided at the outset of the study that a maximum of three rounds of scoring would be used. Each round began with an online questionnaire to determine the clarity and utility of each assessment item by the participating community paramedics. Participants reviewed each assessment item and scored it on two separate three-point Likert scales (as illustrated in figure 1) to determine its relevance. After the first round, participants also received the proportion of responses according to each dimension of relevance from the preceding round. Each questionnaire presented assessment items grouped in domains and ordered in the sequence as they appeared in the CARPE Assessment instrument. Questionnaires were pilot tested with a minimum of three participants before each round to determine the approximate length of time needed for completion, and to refine the questionnaire if necessary. Participants were sent a web-link to the questionnaire at the beginning of each round of scoring. Each round of scoring was open for 2 weeks with reminder emails sent between 48 hours and 72 hours prior to closing of each round.

Between each round of scoring a web-conferencing meeting was held to discuss results of the preceding round and introduce the subsequent round. Results were summarised for assessment items that were classified as either relevant, not relevant, or consensus not reached. In the cases where consensus was not reached on the relevance of assessment items, discussions included characteristics of the assessment items that were not actionable in care planning activities or that were not clear. If participants indicated that an assessment item was difficult to understand, discussions explored how it could be modified (condensed or expanded) according to the context of assessment practices, for the next round of scoring. For example, if an assessment item included multiple parts and multiple levels of response, it could be modified to separate the multiple parts into individual items or to reduce the multiple levels of response to dichotomous levels. Alternatively, if an assessment item included only dichotomous levels of response and participants felt that more granularity was required, the item could be modified to provide multiple levels of response. Any modified multipart assessment items were reorganised to gather subparts into new multipart assessment items where applicable. Each web conference was recorded and shared with participants who were not able to attend. At conclusion of the three rounds of scoring, assessment items would be classified as either relevant, not relevant, modified, or consensus not reached.

Results

Panel and participation

Twenty-six community paramedics from 20 paramedic services agreed to participate in the study. All 26 participated in the first survey. Sixteen (62%) participated in the first meeting (12 in real-time, 4 by viewing the recording). Twenty (77%) participated in the second survey. Eleven (42%) participants joined the second meeting. The final survey was completed by 24 (92%) participants. Table 1 provides a summary of participation.

Table 1.

Summary of participation rates across three rounds of the modified-Delphi study

Study stage	Participation (including watching recorded meeting)	%
Round 1 questionnaire	26	100
Round 1 meeting	16 (4)	62
Round 2 questionnaire	20	77
Round 2 meeting	11 (5)	42
Round 3 questionnaire	24	92

Rounds

The first round presented a total of 64 assessment items grouped according to 14 assessment domains (see figure 2 and table 2). No items had responses indicating that they were not relevant to practice but one domain (which included three items) did not yield any responses that achieved consensus for relevance. Fifty-four items from eight domains met criteria for relevance to practice. The remaining seven items were presented to participants for discussion at the meeting to concluded round one. Secondary analysis identified 25 assessment items where more than one-third of responses were within the central variable in one of the individual dimensions of relevance, clarity or utility.

Figure 2.

Illustration representing outcomes from each round of the study. Diamonds represent consensus for exclusion/removal of assessment items, ellipses represent consensus for relevance of assessment items.

Table 2.

The number of assessment items according to their respective assessment domains presented to participants for rating in each round

Assessment domain	Number of assessment items, round #1	Number of modified assessment items, round #2	Number of modified assessment items, round #3
Living arrangement	3
Cognition	4
Communication and vision	4
Mood	2
Psychosocial well-being and social isolation	13
Functional status	7	19	19
Continence	3
Disease diagnoses	1	5
Health conditions	9	22	3
Nutritional status	2
Medications	5
Treatments and procedures	6
Home environment	4
Personal goals	1
Total	64	46	22

To accommodate the time constraints necessary to discuss the number of assessment items, the discussion was focused specifically on the seven assessment items that did not achieve consensus. Given that many of these assessment items had multiple parts with multiple categories of potential findings, discussion included options for reducing item complexity; either by reducing categories for responses or by separating multipart items into single part items. The meeting participants suggested that multilevel responses were more important. As a result, the questionnaire for the second round reorganised the seven multipart items into 46 single-part items (see table 3 and online supplemental table 1 for examples).

Table 3.

Summarised presentation and modification of an assessment item across rounds

Assessment item presented in round #1	Modification of assessment item presented in round #2	Modification of assessment item presented in round #3	Final reorganisation following scoring
Assessment of ability to perform ADLs. Assessment item includes 11 specific ADLs to assess and provides 8 response categories for levels of dependence from fully independent to fully dependent.	Each specific ADL is presented separately to participants while maintaining the original response categories for levels of dependence from fully independent to fully dependent.	Each specific ADL is presented separately to participants with response categories simplified to either independent or not independent.	Those ADLs that were scored as being relevant were reorganised into one new assessment item including 10 ADLs with response categories of independent or not independent. One modified assessment item did not achieve consensus.
Presented as one assessment item.	Presented as 11 modified items.	Presented as 11 modified items	Presented as 1 reorganised item identified as being relevant and 1 modified item as not achieving consensus.

Further details on modifications are presented in online supplemental table 1.

ADLs, Activities of daily living.

Resulting from the second-round questionnaire, 22 of the modified assessment items achieved consensus on relevance (see figure 2). Secondary analysis identified that seven assessment items had one-third of responses within the central variable of an individual dimension of relevance. Discussion at the second-round meeting focused on of assessment items that could have simplified response categories. The outcome from the second-round meeting was the removal of two modified parts and the reorganisation of the remaining 22 assessment items into simplified response categories (see table 3). The modified assessment items from the second (and third) round were reorganised into seven items representing edited versions of the seven items that did not achieve consensus after the first round.

In the third and final round of scoring, one modified assessment item did not achieve consensus on relevance while the remaining 21 did (see figure 2). The outcome from the three rounds of scoring meant that 54 original assessment items and seven modified assessment items were identified as being relevant to assessments in community paramedicine home visit programmes. In the modification process, three parts of assessment items included in the original set of assessment items were removed.

Discussion

Community paramedicine home visit programmes represent a relatively new area of practice for paramedics that redirect their skills towards preventive and integrated patient care.²⁰ Where assessment practices and guidelines have been established for emergency response, assessment practices and guidelines for community paramedics are still being established.^{10 21 22} Through taking a consensus-based approach with a panel of community paramedics from a cross-section of paramedic services, this study provides new information towards the standardisation of assessment practices in community paramedicine home visit programmes.

Implications for clinical practice

The relevance of assessment items in domains such as home environment, functional status and psychosocial well-being expand on existing paramedic assessment practices such as physical examination and medical history-taking. This reflects the underlying values and purpose of community paramedicine as a patient-centred approach that equally prioritises the biological, psychological and social determinants of health.^{12 20 23} Such comprehensive assessment practices are enabled by the low-acuity and less time-sensitive conditions in which community paramedic home visit programmes operate, as opposed to the norms of emergency response paramedicine where assessment focuses on the most emergent short-term medical needs.¹

Paramedic assessment in emergency response is geared towards guiding immediate treatment decisions and relaying pertinent information to emergency department staff, both examples of short-term care planning and treatment.^{1 2} In contrast, community paramedicine assessments are likely to identify medium-term and long-term care needs. The breadth and depth of assessment items that the expert panel considered relevant to practice suggests that the community paramedics who participated recognise their ability to take action on a range of patient needs that would necessitate the involvement of other healthcare providers from disciplines such as family medicine, occupational therapy, social work, pharmacy and community nursing. Team-based delivery of care introduces a higher level of complexity and uncertainty to assessment practices. How a community paramedic’s assessment informs their own care planning in comparison to its utility to a larger care team is unclear. It is also unclear to what extent assessment may be duplicated by other care providers, and whether or not they would be in agreement with the community paramedic’s assessment. The degree of integration (functional and professional) between different members of a patient’s care team, which community paramedics are a part of, remains an ongoing area of research in integrated care.²⁴

Previously published studies investigating assessment practices by community paramedics have considered different components of the assessment process.^{10 11 15} Principles of patient assessment both in paramedicine and other health settings reflect how the assessment process is a guiding component of any patient care activity.^{1 6} Assessments should gather the clinical and social information about patient condition.1 6 7 25 Asking community paramedics about the relevance of assessment items reveals what parts of an assessment process inform the delivery of care in their practice setting and is informative to how practice has evolved from the emergency setting. The implications from this study can include identifying remaining barriers or inconsistencies to community paramedic practice still need to be addressed. Paramedics are well situated to identify these challenges.

Strengths

In the absence of evidence about community paramedic patient assessment practices, it serves well to identify what community paramedics identify as relevant to the care that they are delivering—particularly when the delivery of care is part of an expanded role or extended scope of practice. Asking an expert panel is consistent with best practice when a definite evidence base is lacking. The methodology followed through our investigation is consistent with recommendations for modified Delphi studies.²⁶ Panel selection was outlined in a reproducible way. Consensus was defined a priori. The number of rounds was specified. Criteria were established to guide discussions.

Criticism of modified Delphi studies is often centred around unclear processes, a biased sampling process for establishing participation, or not having clearly established goals.^{26 27} We established a panel that was broadly representative of practice in Ontario. The process that was outlined and followed suggests that the clarity-utility matrix we established provided a functional method to define relevance of assessment items to assessment practices. The clarity-utility matrix could be broadly applied to future studies exploring paramedic practice or assessment practices in other settings.

Limitations

This study was limited to the Ontario context. While participation levels were adequate across all rounds of scoring and options were available for participants to view recorded meeting proceedings, we did not exclude participants if they were unable to complete one of the scoring rounds or join one of the meetings. For example, it is likely that some participants were less informed entering the third round than others. The structure of the questionnaires and each paramedic’s familiarity with their individual assessment practices should have been adequate in such circumstances and still provide meaningful insight because individual community paramedic practices can vary widely across different health systems.^{9 23} Repeating this study in other jurisdictions may yield different or conflicting results. However, given that community paramedicine home visit programmes are becoming more ubiquitous, the results of this study can contribute to establishing assessment practice guidelines across a wider range of jurisdictions.

We did not examine how community paramedic assessment items compare with those used by other members of the patient care team, and our Delphi panel consisted only of paramedics. Given the multi-disciplinary nature of community paramedicine, other work has explored some of these questions.¹⁵ It will be useful to know to what extent community paramedic assessment items reflect best practice from other fields of health and social care.

Future work

Derivation of an evidence-informed standardised assessment instrument that is fit-for-purpose in community paramedicine programmes can draw from the results of this study, the CARPE Study, and the studies that informed development of the CARPE assessment instrument. It is expected that a refined Community Paramedicine Contact Assessment instrument will be published by interRAI in the near future. Future work that expands the evaluation of the instrument, lending more evidence of it’s reliability, validity, sensitivity and clinical utility will further contribute to wider efforts in a relatively nascent field with opportunity for expanded programme evaluation and development of quality indicators.

Patient-centred care includes reducing barriers to access and better care coordination, consistent with aims of community paramedicine programmes. Future studies should also examine the patient perspective on what they feel is relevant to be included in a structured assessment process.

As the evidence base grows for community paramedicine assessment practices it will lead to a level of standardisation and consistency across jurisdictions and programmes. Future work could then examine the efficacy of these assessment practices by examining process-based and outcome-based indicators such as access to care, service utilisation, and measures of patient health. The development of practice guidelines in community paramedicine will also help develop processes for quality improvement and performance measurement. Evaluating consistent assessment practices in community paramedicine home visit programmes presents the opportunity to measure changes in patient condition over time and further improve case management.

Conclusion

Uptake of assessment guidelines that are broadly applicable to differing community paramedicine programmes is an important step in the growth, evolution and emergence of community paramedicine. By establishing consensus on the relevance of specific assessment items to detect health and social factors that drive functional decline, social isolation, loss of independence and ultimately repeated emergency calls, we believe that guidelines for assessment in community paramedicine programmes will be strengthened, with improved case-finding and care-planning expected to follow.

Data availability statement

Data are available upon reasonable request. Summarized questionnaire responses are available on request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The Hamilton Integrated Research Ethics Board approved the study.