Developing indicators for evaluation of age-friendly communities in Canada: process and results

Abstract

Introduction:

In 2006, the World Health Organization launched the Global Age-Friendly Cities Project to support active aging. Canada has a large number of age-friendly initiatives; however, little is known about the effectiveness and outcomes of age-friendly community (AFC) initiatives. In addition, stakeholders report that they lack the capacity and tools to develop and conduct evaluations of their AFC initiatives. In order to address these gaps, the Public Health Agency of Canada developed indicators to support the evaluation of AFC initiatives relevant to a wide range of Canadian communities. These indicators meet the varied needs of communities, but are not designed to evaluate collective impact or enable cross-community comparisons.

Methods:

An evidence-based, iterative consultation approach was used to develop indicators for AFCs. This involved a literature review and an environmental scan. Two rounds of key expert and stakeholder consultations were conducted to rate potential indicators according to their importance, actionability and feasibility. A final list of indicators and potential measures were developed based on results from these consultations, as well as key policy considerations.

Results:

Thirty-nine indicators emerged across eight AFC domains plus four indicators related to long-term health and social outcomes. All meet the intended purpose of evaluating AFC initiatives at the community level. A user-friendly guide is available to support and share this work.

Conclusion:

The AFC indicators can help communities evaluate age-friendly initiatives, which is the final step in completing a cycle of the Pan-Canadian AFC milestones. Communities are encouraged to use the evaluation results to improve their AFC initiatives, there by benefiting a broad range of Canadians.

Highlights

• An increasing number of communities across Canada have implemented age-friendly community (AFC) initiatives. Many are ready to evaluate their activities.

• Through a comprehensive review, the Public Health Agency of Canada generated potential indicators of age- friendliness that could be used for evaluation purposes. A multiphase consultation process was used to reduce and refine a long list of candidate indicators.

• The final list includes 39 indicators across eight domains, as well as four indicators of health and social outcomes, that can support communities in their evaluation activities.

Introduction

In 2006, the World Health Organization (WHO) kick-started the Global Age- Friendly Cities Project by bringing together representatives from cities around the world who were interested in supporting healthy aging.¹ This consultation identified eight key areas of community life where communities could become more age-friendly: outdoor spaces and buildings, transportation, housing, social participation, respect and social inclusion, civic participation and employment, communication and information, and community support and health services. Canada was a key partner in developing this approach and four Canadian cities took part in consultations that shaped the development of the model. Noting that Canada’s context includes a significant number of rural and remote communities, the federal, provincial and territorial ministers responsible for seniors sponsored a companion project that resulted in a document entitled Age-Friendly Rural and Remote Communities: A Guide.²

As part of its national leadership role to promote the development of age-friendly communities, the Public Health Agency of Canada (PHAC) developed pan-Canadian age-friendly community (AFC) milestones in collaboration with key partners. These milestones describe the steps needed to apply the AFC model in Canada as follows:

• 1. Establish an advisory committee that includes the active engagement of older adults;

• 2. Secure a local municipal council resolution to actively support, promote and work towards becoming age-friendly;

• 3. Establish a robust and concrete plan of action that responds to the needs identified by older adults in the community;

• 4. Demonstrate commitment to action by publicly posting the action plan; and

• 5. Commit to measuring activities, reviewing action plan outcomes and reporting on them publicly.³

Over 900 Canadian communities are currently working to become age-friendly. A number of jurisdictions and non-governmental organizations have reported that they lacked the capacity and tools to complete the fifth milestone, which is to conduct effective evaluations. In 2009, PHAC convened a meeting with key stakeholders and researchers in age-friendly communities to discuss evaluation of the AFC initiative as a whole. During this meeting, the need for indicators and data to assist community stakeholders was also identified.

In 2011, in response to mounting interest from provinces, municipalities, non-governmental organizations and Canadian researchers, PHAC undertook a rigorous process to develop a set of AFC indicators. These indicators form a menu that communities can choose from according to their unique local issues and capacities.

In this article, we describe the process undertaken by PHAC, the rationale and principles underlying the AFC indicators project and the progress to date in establishing AFC indicators.

Methods and results

Indicator identification and prioritization process

The process PHAC used to develop this set of indicators was adapted from previous, accepted methods that have been used to develop other indicator frameworks in public health.^4,5,6 Generally, the development of indicators includes establishing the purpose of the indicators; designing the conceptual framework that can be based on theory, policy and/or data;⁴ and selecting or creating the indicators.⁵

The purpose of this indicator development process was to identify a menu of indicators that communities undertaking agefriendly initiatives could choose from in order to support their evaluation and monitoring activities. The overarching conceptual framework was the WHO’s Age-Friendly Cities framework, which identifies eight domains on which communities can focus to support the healthy aging of their residents.¹

An Age-Friendly Indicators Working Group was established to provide input into the development of the AFC Indicators project. This Working Group, which met regularly throughout the project, comprised officials from PHAC; representatives of provincial, territorial and municipal governments; researchers; members of non-governmental organizations and seniors.

Literature review and environmental scan

The first step was to identify potential indicators through a literature review. We searched the following databases for peerreviewed literature published between 1990 and 2012: Web of Science, AgeLine, Cochrane Injuries Group trials register, CINAHL Database, MEDLINE, Health Source and Social Sciences Citation Index. Only English and French publications related to adults aged 65 years and older were retained. The search included the following terms and their combinations: seniors, elderly, older persons, evaluation, age-friendly cities/communities/hospitals/ businesses, senior-friendly communities, elder-friendly communities, visitable housing, built environment, housing, housing modification, building codes, stairs, sidewalks, transportation, social environment/inclusion, respect, employment, volunteering, outcomes, improvement, indicators of success, falls, accidents, motor vehicle accidents, road traffic accidents, communication, community support and health services.

First, we screened titles for relevance. Articles that were clearly out of scope, for example, articles about clinical care for a specific disease, were rejected at this stage. We then reviewed the abstracts for the remaining articles and included those that reported on or implied information about indicators or measures to do with age-friendly communities. Most of these articles focussed on process evaluations or short-term outcomes of the work undertaken by communities. We found no reports of completed long-term outcome evaluations of any age-friendly initiatives.

From an initial list of over 2000 documents, 23 articles were identified as relevant.

We also conducted a scan of existing AFC evaluation activities in Canada and abroad. We emailed key stakeholders in provinces, territories and selected municipalities about their activities and asked respondents to provide copies of any evaluation tools. The keywords “age-friendly” and “evaluation” were used to identify programs abroad and other relevant grey literature via the Google search engine. We retained documents from this search if they included information about existing age-friendly initiatives and/or community-based evaluations as well as information to do with indicators or measures; we found five community reports through this Internet search.^7-11 A review of our files together with the online search resulted in 20 additional documents based on our criteria.

Overall, 43 documents and articles were selected for review: 12 from the United States,^12-23 three multi-country international, ^1,24,25 two from France,^26,27 two from Australia,^28,29 two from the Netherlands,^30,31 and one from the United Kingdom.³² Twenty-one documents originated in Canada: five documents were from British Columbia,^33-37 five from Quebec,^7,38-41 four from Ontario,^9,10,42,43 one from Manitoba,⁸ one from Saskatchewan¹¹ and five were national in scope.^44-48

Two qualitative researchers reviewed the articles and reports, and made detailed notes on potential indicators of age-friendliness based on article contents, within each of the eight WHO-identified domains.¹ This theory-driven, or deductive, approach to collecting qualitative data⁴⁹ generated an initial list of 241 potential indicators. The two qualitative researchers worked together to develop sub-themes within each domain by grouping similar indicators together and identifying codes (or labels) that captured the common underlying concepts (see Table 1). This is an example of inductive, or data-driven, coding.⁴⁹ The research team (made up of four members with expertise in qualitative or quantitative methods as well as content expertise in georontology and AFC) reviewed the list of potential indicators for duplication, relevance and clarity. Based on consensus, they grouped some similar concepts into a single indicator and removed concepts deemed out of scope, reducing the list to 194. A small number of indicators that originally populated the eight domains did not fit well within sub-themes, but were considered important from a healthy aging perspective; these were grouped into a new domain, “Health and Social Outcomes.” This structure was presented to the Age- Friendly Indicators Working Group for confirmation.

Table 1. Potential indicators for evaluating age-friendly communities in Canada and sub-themes.

We also developed a list of potential criteria based on reviewed literature and shared this with the Age-Friendly Indicators Working Group. The list of criteria included the following: evidence-based, reflecting burden, representative, available, amenable to change, understandable, repeatable, important, sound, viable, direct, objective, useful, attributable, practical and adequate. ^50-53 The research team identified important/relevant, actionable and feasible as the most important criteria for selecting indicators, and confirmed this with the Working Group.

Indicator prioritization through stakeholder engagement and consultation

The second phase of the indicator selection process included two targeted stakeholder consultations aimed at reducing the list of potential indicators. These consultations were conducted using an online survey platform developed using FluidSurvey.

Consultation #1: The first consultation targeted 789 known stakeholders, including provincial and territorial representatives, municipal representatives, members of non-governmental organizations, researchers and project staff or volunteers on agefriendly projects. We identified stakeholders through existing contact lists, by the Age- Friendly Reference Group and the Age- Friendly Indicators Working Group. Respondents rated the 194 potential indicators according to their importance for measuring the age-friendliness of communities on a scale of 1 to 4, where 1 was “unimportant,” 2 “of little importance,” 3 “important,” and 4 “very important.”

A total of 191 people responded to the first consultation survey (24% response rate). All provinces were represented, except for New Brunswick, although there were few respondents from Prince Edward Island and Saskatchewan. There were no respondents from the Territories. All of the intended stakeholder groups were represented. The majority of the respondents were female (70%), replied in English (61%) and were less than 55 years old (59%). Stakeholders lived in various sizes of community: 65% lived in centres of between 1000 and 29 999 inhabitants; 9% in centres of between 30 000 and 99 999 inhabitants; and 23% in centres of 100 000 inhabitants or more.

Table 2 shows the characteristics of respondents for the first and second consultations.

Table 2. Characteristics of respondents to consultation surveys 1 and 2 used in the process of selecting indicators for evaluating age-friendly communities in Canada.

The average importance rating for each indicator (on a scale of 1 to 4) ranged from a low of 2.58 to a high of 3.69. Of the top five rated indicators, three were related to enhanced health and community services and included the following:

• 1. Existence of programs for caregiver support (3.69 on a scale of 1 to 4)(Supportive health services)

• 2. Availability of low-cost food programs (3.67) (Community services)

• 3. Level of unmet home-care need (3.66)(Community services)

• 4. Existence of regulations and standards for nursing home (3.64) (Community services)

• 5. Number of affordable options for transportation (3.64) (Transportation options and public transit)

Based on rankings by mean importance score, the top 50% of indicators in each sub-domain were retained for further analysis. The data were also stratified by community size to examine them separately for small/medium and large communities. In the few instances where the top 50% of indicators by mean importance score were not the same in both small/medium and large communities, we added indicators to ensure inclusion of the top 50% for both of these groups. This process reduced the number of indicators to 129. A number of indicators with similar concepts were subsequently combined, reducing the list to 109.

Of the 191 respondents to the first consultation survey, 93 indicated that they would be interested in participating in a subsequent consultation survey.

Consultation #2: The aim of the second consultation survey was to further streamline the list of 109 potential indicators according to the criteria of “actionability” and “feasibility” as defined below.

For an indicator to be actionable, it can be influenced by the local or regional community, government or private sector and is likely to show change in response to action. This criterion was also used at the Age-Friendly Indicator Development Group Global Age-friendly Cities meeting⁵⁰ and by the Canadian Injury Indicators Development Team.⁵¹ It is also similar to Daniel’s⁵² idea of changeability.

For an indicator to be feasible, data for it is measured (e.g. from a survey or administrative data) or described (e.g. with a photo or story) in a realistic manner without obstacles to collection or use. Such data can also be used to add richness and bring a program’s results to life.⁵³ In addition, data collection methods are easy and realistic.⁵²

Because of the response fatigue observed in the first consultation survey, the second consultation survey response choices were simplified to “yes,” “no” and “don’t know or no opinion.” Two questions concerning the preferred data collection methods for evaluation and the design of a forthcoming PHAC-produced guide were also included in this consultation.

This second survey was sent via the online survey platform to the 93 respondents to the first survey who had indicated that they would participate in a subsequent round. Of these, 49 people responded (52% response rate). As shown in Table 2, 80% of respondents were female, and 92% responded in English. Most regions in Canada as well as a wide range of stakeholder groups were represented.

We calculated the proportion of respondents who considered a given indicator actionable or feasible. Three categories were created in order to group indicators based on actionability and feasibility. High agreement was assigned to indicators that scored 89.5% or higher for both criteria. Moderate agreement reflected indicators that scored at least 79.5% for both criteria but less than 89.5% for at least one. Low agreement was assigned to indicators that scored lower than 79.5% for at least one of the two criteria. These cut-offs created categories that approximated tertiles, giving equal weight to actionability and feasibility.

A total of 38 indicators achieved high agreement, 47 achieved moderate agreement and 24 achieved low agreement.

Indicators

Utilizing a combination of level of agreement, frequency of appearance of concept in peer-reviewed literature and grey literature, perceived concordance with the proposed WHO indicators, and whether the indicator was a measure of impact, we selected the final list of indicators (see Table 1).

All four long-term health and social outcomes indicators were retained due to their concordance with the framework proposed by WHO at the time. This resulted in a final list of 43 indicators, shown in Table 3 by domain and theme within each domain.

Table 3. List of indicators for age-friendly communities.

Respondents were also asked for suggestions on developing a tool or guide to evaluate their age-friendly initiatives. The following themes emerged from analyzing their input:

• The tool should have clear definitions and criteria for measuring the indicators; it should be flexible to account for unique features of a community and should have more behavioural measures.

• The findings should lead to action at the community level.

• Long-term outcomes should be included.

• The cost of evaluations should not be a barrier (which raises the question of human and financial capacity for evaluation in communities).

During the second consultation survey, respondents were asked for ideas about the kinds of data collection methods that would be most practical for use in evaluating age-friendly initiatives. Fewer than 60% of the respondents rated the following as “very practical” or “practical”: face-to-face interviews (54%), telephone interviews (51%), stories (58%) and photographs or videotaping (56%). All the other methods (online questionnaire, paper and pencil questionnaire, observations and audits, information gathered for other purposes, administrative data, use of secondary data analysis) were rated as very practical or practical by at least 75% or more of the respondents. The most practical methods were thought to be focus groups (100%), paper-and-pencil questionnaires (91%) and use of administrative data (87%).

Results of both consultations were summarized, translated into French and shared with all participants.

Based on feedback from the consultation surveys and guidance from the Agefriendly Communities Reference Group, a range of measures were identified as potential tools for communities wishing to implement age-friendly indicators in their evaluation activities. For each indicator, one or more of five potential approaches to measurement were identified: assessment tools, accessibility tools, existing data, program inventories and surveys. Qualitative and quantitative approaches were included. Tools and existing data were identified for communities wishing to evaluate their activities. Subsequently, a user-friendly guide, Using Indicators in Your Age-Friendly Community Initiative, (available from PHAC at http://www .phac-aspc.gc.ca/seniors-aines/indicators-indicateurs-eng.php/) was developed to share the indicators, proposed measurement approaches and useful tools with communities.

Discussion

Note that the initial indicator lists were based on the literature on AFCs available when the review was conducted in 2012. At that time it was clear that the peer-reviewed and grey literature on AFCs had the following limitations: insufficient attention to specialized populations, such as ethnic groups, First Nations, and Lesbian, Gay, Bisexual, Transgender and Questioning (LGBTQ) populations; the heavy use of focus groups to seek community input (which may disadvantage those with mobility issues); a lack of consideration of specialized settings, such as nursing homes or hospitals; and finally, little emphasis on family cohesion and integration as key elements in community age-friendliness.

Since then, the literature on AFCs has grown considerably. As noted in the introduction, WHO initiated a process to identify core indicators for use by the Global Network of Age-Friendly Cities and Communities in 2011, shortly after PHAC initiated the project described in this article; WHO published a guide in 2015.⁵⁴ There is considerable concordance between the indicators suggested by WHO and the menu proposed by PHAC, which is not surprising as both projects informed one another and were developed in parallel. Concepts in common include walkability; accessibility of public spaces and buildings, of public transportation and vehicles, and of public transportation stops; affordability of housing; engagement in volunteer activity, paid employment, and sociocultural activity; availability of information, health and social services; and quality of life. A recent realist review of the age-friendliness of cities in the European Healthy Cities Network clustered the eight WHO domains into three groups: physical environment, social environment and municipal services.⁵⁵ Using a realist synthesis, the researchers mapped contexts; interventions; short-, mid- and long-term outcomes; and the goal of age- friendly programs. Again, the concepts identified based on existing programs using information from European cities is highly concordant with that developed through the present process, confirming our results and demonstrating transfer-ability of our findings to other Western contexts.

Strengths and limitations

The process we describe in this article was developed to yield a set of indicators, relevant to the Canadian context, that would be acceptable and helpful for communities undertaking age-friendly initiatives. Because of this, a number of methods were adopted to ensure that the process was sound and of high quality. The initial stage, reviewing the literature and identifying potential indicators, was based on qualitative methods. The credibility, transferability, dependability and confirmability of this initial step were central to the process.⁵⁶

• The credibility of these analyses is supported by the qualitative expertise of the research team, the research team’s review of coding and the use of consensus to agree on grouping similar concepts into a single indicator.

• Transferability to countries similar to those described in the document review is likely. However, the identified indicators are not transferable to low- or middle-income or non-Western countries.

• Dependability is supported by describing our methods to identify indicator concepts using both deductive and inductive processes.

• Confirmability was supported through triangulation, by using multiple qualitative researchers during the document review process and by maintaining detailed notes during the process of identifying and reducing the initial lists of indicators.

Several factors limit the generalizability of the findings of the two consultation surveys. First, consultations were conducted with known stakeholders and may not be representative of a cross section of those with an interest in seniors and/or age- friendly work. For example, the general public was not consulted, and all stake-holders would have had some degree of familiarity with age-friendly initiatives. As a result, few respondents were aged over 75 years, and few identified themselves as belonging to a visible minority group. The most vulnerable and marginalized members of society, including those who were homeless, living at low income, living with dementia, living in institutions or without Internet access are missing from our sample.

Second, the response rates for each survey were fairly low and as a result, the final number of respondents was limited. Third, a pattern of response fatigue was identified because the proportion of missing data increased towards the end of both surveys, and a few participants commented that the surveys were excessively time consuming. Responses were visually screened for response sets (i.e. respondents entering the same response for all questions); no patterns of response set were identified.

The fact that the definitions for both criteria (actionable and feasible) included more than one concept may have confused some of those completing the second survey and caused uncertainty in interpreting the results. If a respondent answered “very actionable” to an indicator, it was not clear whether this was because they felt the appropriate level of government was involved or because the indicator was responsive to a change of policy or a new program or activity. Similarly, if an indicator was rated as “very feasible,” did the respondent mean that the data were both qualitative and quantitative or that data could be collected with ease? Future consultations should separate criteria so that only one issue is evaluated per question.

Conclusion

The number of communities engaging in age-friendly projects is increasing across Canada and many are ready to conduct evaluations of their activities. There is a strong interest in identifying shared ways to measure progress towards becoming age-friendly, the fifth pan-Canadian AFC milestone. In this article, we reported on PHAC’s process to identify and select potential AFC indicators. From an initial list of 241 indicators, we created a list of 43 indicators based on actionability and feasibility.

Acknowledgements

We would like to acknowledge the contributions of Danielle Maltais and Dawn Nickel to this project.