Abstract
Background/objective
Participation in the Active Healthy Kids Global Alliance (AHKGA) Global Matrix initiative represents a significant work and resource investment for Report Card teams. The objective of this paper was to evaluate the process and findings of the Global Matrix 3.0 and formulate recommendations for improvement.
Methods
The evaluation of the Global Matrix 3.0 was planned prior to its development and a list of potential process, impact, and outcome indicators were pre-identified. These indicators were informed by online surveys, direct reports, MailChimp’s statistics, Google Analytics of the AHKGA website, and emails sent to the AHKGA Board of Directors.
Results
Five online surveys were completed by 88%–100% of the targeted respondents. High satisfaction ratings were observed for most of the Global Matrix 3.0 methods, key steps, concepts, and the resources (e-blasts and website) provided by the AHKGA. A total of 496 open-ended comments were provided in the five surveys, including 199 comments reporting issue(s), and 38 reporting both positive feedback and issue(s). The participating Report Card teams successfully assigned a grade to each physical activity indicator, produced a Report Card document, and wrote a short Report Card article.
Conclusion
This evaluation process allowed for the identification of needed improvements and the formulation of recommendations for future Global Matrix initiatives. This work highlighted the need for the development of physical activity behavior assessment tools that would be internationally adopted and culturally adaptable to varying contexts to improve the standardization of physical activity surveillance at the global scale.
Keywords: Adolescent, Child, International cooperation, Outcome assessment, Process assessment, Youth sport
Introduction
The Global Matrix on physical activity for children and youth1, 2, 3 is an international initiative where multidisciplinary teams of experts from numerous countries concurrently develop national Report Cards4 with physical activity grades following a harmonized development process. This initiative is led by the Active Healthy Kids Global Alliance (AHKGA), an incorporated not-for-profit organization dedicated to improving the physical activity of children and youth from around the world.5 In 2018, the AHKGA released the Global Matrix 3.0, compiling Report Card grades from 49 countries for 10 physical activity indicators: five behavioral indicators (overall physical activity, organized sport and physical activity, active play, active transportation, sedentary behavior), one individual characteristic indicator (physical fitness), and four sources of influence indicators (family and peers, school, community and environment, government).3 Forty-nine short Report Card articles summarizing the main findings from each participating country,6 three articles discussing results from the Report Cards by predetermined Human Development Index (HDI)7, 8, 9 categories (low and medium HDI, high HDI, and very high HDI),10, 11., 12 an article integrating the findings from all HDI papers, and an article reporting the international impact of the Global Matrix and Report Card initiative13 were published in a special issue of the Journal of Physical Activity and Health (JPAH).
Participation in the Global Matrix represents a significant investment in both human and financial resources for all the involved researchers and stakeholders. Report Card teams have to pay registration fees and follow a specific harmonized development method to develop their Report Card. Therefore, the AHKGA designed a method to evaluate the Global Matrix 3.0 process and outcomes to carefully reflect on opportunities to work more effectively and efficiently in the future. This evaluation process involved the collection of process and outcome indicators through online surveys and reports by email at different stages of the Global Matrix 3.0 development process. The objective of this paper was to present the main findings of these evaluations and to formulate recommendations for the improvement of future Global Matrix initiatives.
Methods
Global Matrix 3.0 and harmonized report card development
The detailed methods for the development of the Global Matrix 3.03 and Report Card,4 have been previously published. In each participating country, one to three Report Card leaders were officially designated to manage the national Report Card development, and to ensure effective communication between the AHKGA Executive Committee and the Report Card team. Report Card teams had to compile the best available published and unpublished evidence to assign letter grades to physical activity indicators following harmonized benchmarks and grading rubric.3
A mentoring system was developed for the Global Matrix 3.0: new Report Card leaders and teams were paired with an intermediate mentor who was an experienced Report Card leader from a country that participated in the Global Matrix 2.0. In addition, six regional mentors (Africa, Asia, Europe, North America, Oceania, and South America) were designated to provide help and guidance to the countries when needed and to report their progress to the AHKGA Executive Committee. The AHKGA Board of Directors coordinated the Global Matrix 3.0 development and provided a variety of resources through e-blasts, a monthly newsletter sharing resources and deadlines, and mentorship to support the harmonized development of Report Cards. The Report Card leaders were also provided with individual access credentials for the “members only” section of the AHKGA website where information on the Report Card development process and resource documents were provided. The Report Card grades from each participating country were submitted with their rationale and were audited by selected AHKGA members. Finally, a short Report Card article was written by each Report Card team following a template provided by the AHKGA and audited by selected AHKGA members before being submitted for copy-editing in JPAH.
Evaluation indicators
An evaluation plan for the Global Matrix 3.0 was designed prior its development by members of the AHKGA Board of Directors. The aim of this plan was to gather data and information to strengthen and improve the Global Matrix development process and approach, and also to assess the international impact of the Global Matrices and Report Cards. The impact indicators monitored the progress of achieving the Global Matrix 3.0 objectives.14 A study entirely dedicated to the international impact of the Report Cards and Global Matrices was recently published in JPAH.13 The reader is invited to consult this open access publication reporting and discussing the Global Matrix 3.0 impact indicators. Process outcomes meeting the SMART framework criteria (Specific, Measurable, Appropriate, Reliable, Time-consistent)15 were identified by reviewing the scientific and grey literature and adapted to the specificity of the Global Matrix. The evaluation plan included the possibility to collect unanticipated indicators by offering the opportunity to report any negative or positive side product of the Global Matrix process in open comments on evaluation surveys and by email to everyone involved at any stage of the development. In addition, specific outcome indicators for the expected Global Matrix 3.0 outputs were also pre-identified to assess if the objectives of the Global Matrix 3.0 were achieved.14 The evaluation of these indicators was informed by online surveys, direct reports, MailChimp’s statistics, Google Analytics of the AHKGA website, and emails that were sent to the AHKGA Board of Directors. MailChimp (The Rocket Science Group, Atlanta) is an email marketing platform that provides analytics on the number of recipients who received the e-blasts, opened them, and clicked on relevant links. Google Analytics (Google, Mountain View) is a Google service that tracks AHKGA website traffic and usage.
Online surveys
Five online surveys were developed using REDCap Software, a secure, web-based application designed exclusively to support data capture for research studies,16 for evaluation purposes and circulated by AHKGA: three sequential Report Card Leader Surveys, and two sequential Mentoring Surveys. A web link to the online surveys was circulated to the Report Card leaders through the monthly AHKGA e-blasts. Regular reminders to complete the surveys were sent to the Report Card leader in subsequent e-blasts and by e-mail. Only one Report Card leader per participating country was allowed to complete the online surveys. The Report Card Leader Surveys were circulated at three time points of the Global Matrix 3.0 development process as seen in Fig. 1, and their content evolved depending on the time point. Open- and close-ended data on satisfaction and potential issues concerning the Global Matrix 3.0 key steps, concepts and process were collected. The Mentoring Surveys were circulated at two time points (Fig. 1). The two Mentoring Surveys were anonymized, identical, and aimed to evaluate the mentorship organization for the Global Matrix 3.0. In these surveys, Report Card leaders rated (from 1 to 10) the responsiveness, engagement, and overall organization of AHKGA, their intermediate mentor, their regional mentor, or their mentee(s) (in the case of mentors completing the surveys). In both types of surveys, satisfaction rates were evaluated using three-point Likert scales.
Fig. 1.
Timeline presenting the key steps of the Global Matrix 3.0 development and the time of the evaluation surveys. AHKGA = Active Healthy Kids Global Alliance; JPAH = Journal of Physical Activity and Health.
Ethics statement
This evaluation protocol was submitted to the Children’s Hospital of Eastern Ontario Research Ethics Board (CHEO REB) (#18/08X). CHEO REB determined that this project did not meet the Tri-Council Policy Statement Ethical Conduct for Research Involving Humans17 definition of research but corresponded instead to a quality assurance project “quality assurance/quality improvement/program evaluation”.
Results
Process indicators
Global Matrix 3.0 country and participant information
Table 1 summarizes the information on countries that participated in the Global Matrix 3.0. While the AHKGA aimed for a maximum of 75 countries to participate in the Global Matrix 3.0, 51 countries registered and had paid their registration fees by the end of January 2018. Over the development of the Global Matrix 3.0, two countries dropped out due to lack of time and lack of new data respectively. Table 2 presents descriptive statistics concerning the number of participants to the Global Matrix 3.0. Global Matrix 3.0 “participants” refers to all the people who have contributed to the development of the Global Matrix 3.0 (i.e. AHKGA Board of Directors, Report Card leaders and team members). Fourteen country Report Cards were led by one or several females, and six were co-led by at least one female. During the development of the Global Matrix 3.0, the AHKGA Board of Directors was composed of four females and six males. The Report Card leaders also reported the number of members who dropped-out their Report Card team, the number of new members who joined, and the reason(s) why in the Report Card leader survey 2 (Table 3).
Table 1.
Global Matrix 3.0 country participation information.
| Indicator | Total | Low and Medium HDI countries | High HDI countries | Very High HDI countries |
|---|---|---|---|---|
| Number of participating countriesa | 49 | 9 | 10 | 30 |
| Country dropout | 2 | 1 | 1 | 0 |
| New participating countries | 15 | 4 | 4 | 7 |
| Global Matrix 2.0 returning countries | 34 | 5 | 6 | 23 |
Notes: HDI = Human Development Index, NA = not applicable. aA total of 51 counties registered and paid their registration fees by January 2018, but two countries dropped out and 49 fully participated in the Global Matrix 3.0 process.
Table 2.
Global Matrix 3.0 participants’ characteristics.
| Indicator | Total | Mean number of members (range) per Report Card team |
|---|---|---|
| Total number of Global Matrix 3.0 participants | 513 | 10 (3–23) |
| Number of males | 258 (50%) | 5 (0–17) |
| Number of females | 255 (50%) | 5 (0–16) |
| Number affiliated with a university (%) | NA | 68% (0%–100%) |
| Number affiliated with a governmental institution (%) | NA | 10% (0%–67%) |
| Number of institutions affiliated with the Global Matrix 3.0 participants | 283 | 6 (1–15) |
Note: NA = not applicable.
Table 3.
Number and reasons of dropped-out Report Card team members and new Report Card team members reported in the Report Card Leader Survey 2.
| Report Card team members who dropped out n (%) | 13 (2%) |
| Report Card team members drop-out reasons | Lack of time (n = 7) Moved to another position/ city/ country or to retirement (n = 4) Removed/quitted due to lack of participation (n = 2) |
| Number of new Report Card team members | 9 (1.7%) |
| New Report Card team members reasons | Expertise needed for the Report Card (n = 4) Replacement of Report Card team member who dropped out (n = 2) Help needed due to large amount of workload (n = 2) Recommended by AHKGA (n = 1) |
Responsiveness, engagement and organization of the Global Matrix 3.0 report card leaders
The three Report Card Leader Surveys were completed by 100% of the targeted respondents (i.e. one Report Card leader per participating country, n = 49) and the Mentoring Surveys 1 and 2 were completed by 98% and 88% of the targeted respondents (i.e. one Report Card leader per participating country and an additional regional mentor, n = 50), respectively. The intermediate and regional mentors rated (from 0 to 10) the responsiveness, engagement and overall organization of the Global Matrix 3.0 participants using Likert scales in the Mentoring Surveys. A summary of these ratings is presented in Table 4. The statistics provided by MailChimp regarding the use of the e-blasts are presented in Table 5. Finally, the number of page views in the AHKGA website member only section over the year 2018 provided by Google Analytics is presented in Fig. 2.
Table 4.
Mean responsiveness, engagement and overall organization ratings of the Report Card leaders evaluated in the Mentoring Survey 1 and 2.
| Indicators | Mentoring Survey 1 | Mentoring Survey 2 |
|---|---|---|
| Number of Report Card leaders rated by intermediate mentors | 13 | 9 |
| Number of Report Card leaders rated by regional mentors | 17 | 34 |
| Mean responsiveness rating (out of 10) | 8.0 | 8.2 |
| Mean engagement rating (out of 10) | 8.0 | 8.1 |
| Mean overall organization rating (out of 10) | 8.1 | 8.4 |
Table 5.
Mailchimp statistics concerning the use of the e-blasts circulated monthly by AHKGA.
| Title of the e-blast | Send Date | Total Recipients | Open Rate | Total Opens | Click Rate | Total Clicks |
|---|---|---|---|---|---|---|
| Global Matrix 3.0 - January 2018 E-Blast | 2018-01-26 16:23 | 68 | 75% | 373 | 51% | 106 |
| Global Matrix 3.0 - February 2018 E-Blast | 2018-02-26 12:59 | 70 | 73% | 940 | 40% | 81 |
| Global Matrix 3.0 - March 2018 E-Blast | 2018-03-29 15:39 | 69 | 75% | 637 | 59% | 157 |
| Global Matrix 3.0 - April 2018 E-Blast | 2018-04-24 10:25 | 67 | 84% | 629 | 57% | 188 |
| Global Matrix 3.0 - May 2018 E-Blast | 2018-05-14 12:03 | 67 | 82% | 1448 | 60% | 190 |
| Global Matrix 3.0 - June 2018 E-Blast | 2018-06-22 15:18 | 77 | 77% | 1343 | 57% | 181 |
| Global Matrix 3.0 - July 2018 E-Blast | 2018-07-17 13:47 | 77 | 79% | 728 | 42% | 186 |
| Global Matrix 3.0 - August 2018 E-Blast | 2018-08-27 13:39 | 76 | 72% | 623 | 42% | 132 |
| Global Matrix 3.0 - September 2018 E-Blast | 2018-09-19 8:18 | 76 | 76% | 734 | 11% | 20 |
| Global Matrix 3.0 - October 2018 E-Blast | 2018-10-14 21:30 | 78 | 73% | 925 | 11% | 16 |
| Global Matrix 3.0 - November 2018 E-Blast #1 | 2018-10-31 17:05 | 78 | 74% | 792 | 53% | 142 |
| Global Matrix 3.0 Materials Needed Reminder | 2018-11-07 11:17 | 77 | 74% | 744 | 52% | 65 |
| Global Matrix 3.0 Materials Needed Reminder #2 | 2018-11-13 13:57 | 76 | 78% | 340 | 60% | 80 |
| Global Matrix 3.0 - November 2018 E-Blast #2 - FINAL ONE BEFORE LAUNCH! | 2018-11-17 15:08 | 123 | 63% | 635 | 38% | 324 |
Fig. 2.
Number of page views in the AHKGA website member only section over the year 2018 provided by Google Analytics.
Responsiveness, engagement and organization of the intermediate and regional mentors
The Report Card leaders rated the responsiveness, engagement and overall organization of their intermediate mentors and/or the regional mentors using Likert scales (from 0 to 10) in the Mentoring Surveys. A summary of these ratings is presented in Table 6. All intermediate and regional mentors reported that they spent ≤1 h per week mentoring countries.
Table 6.
Mean Responsiveness, engagement and overall organization scores of the intermediate mentors, the regional mentors, and AHKGA evaluated in the Mentoring Survey 1 and 2.
| Indicators | Mentoring Survey 1 | Mentoring Survey 2 |
|---|---|---|
| Number of Report Card leaders rating intermediate mentors | 23 (47%) | 17 (35%) |
| Mean intermediate mentors responsiveness score | 7.9 | 7.9 |
| Mean Intermediate mentors engagement score | 7.7 | 7.7 |
| Mean intermediate mentors overall organization score | 7.5 | 7.5 |
| Number of Report Card leaders rating regional mentors | 31 (63%) | 31 (63%) |
| Mean regional mentors responsiveness score | 8.0 | 8.4 |
| Mean regional mentors engagement score | 8.0 | 8.3 |
| Mean regional mentors overall organization score | 8.1 | 8.4 |
| Number of Report Card leaders rating AHKGA | 36 (73%) | 36 (73%) |
| Mean AHKGA responsiveness score | 9.4 | 8.5 |
| Mean AHKGA engagement score | 9.4 | 8.8 |
| Mean AHKGA overall organization score | 9.1 | 8.8 |
Responsiveness, engagement and organization of AHKGA Board of Directors
A total of 12 AHKGA Board of Director meetings took place between January and November 2018 (one per month in addition to a special three-day strategic planning meeting which occurred in January), for a total of 28 h of face-to-face or teleconference meetings. For each of these meetings, an agenda was prepared and circulated by the President of AHKGA, and the attendance, discussions, and list of action items were recorded in the minutes. The President of AHKGA prepared and circulated an e-blast sent to all country leaders within a week following each of these meetings. In addition to these recurring activities, the AHKGA Board of Directors created the resource documents that guided the development of the Global Matrix 3.0; created and maintained the AHKGA website, led the audit of the Report Card grades and of the short Report Card articles; contributed to the writing of the integrated scientific articles; prepared workshops and scientific presentations for the release of the Global Matrix 3.0 at the Movement to Move event; and developed adaptable templates of a press release, key finding document, question and answer sheet, social media material, and infographics dedicated to help the Report Card leaders promote the dissemination of their findings. All Report Card leaders who were not members of the AHKGA Board of Directors were asked to rate the responsiveness, engagement and overall organization of the AHKGA using Likert scales in the Mentoring Surveys; a summary of these ratings is presented in Table 6.
Collateral Impact
A series of success stories related to the Report Card and/or Global Matrix activities was already published elsewhere.13 One of these stories, classified in the “capacity building” category, is a relevant unexpected side effect that resulted from the Global Matrix 3.0 development process: the development of the 2018 Ghana’s Report Card facilitated the training of senior students in physical education and sport in data monitoring and surveillance. The Report Card leader of Uruguay also reported recently that the development of their Report Card was a good training opportunity for the members of his research group and his senior undergrad students.
Satisfaction measures, challenges and issues
The satisfaction of the Report Card leaders concerning key steps and concepts of the Global Matrix 3.0 was assessed though the Report Card Leader Surveys 1, 2, and 3. A selection of satisfaction ratings organized by themes is presented in Fig. 3A–H. The Report Card leaders also reported potential issue(s) through open-ended responses in several sections of the Report Card Leader Surveys and Mentoring Surveys. A total of 496 comments were provided in the five surveys, including 103 positive comments, 156 neutral or informative comments, 199 comments reporting a/several issue(s), and 38 reporting both positive feedback and some issue(s). The number of issues mentioned in the last two categories of comments is summarized in Table 7. Finally, the Report Card leaders were also asked in Report Card Leader Survey 3 to report if they (or someone else in their country) were planning on participating in the next Global Matrix (Global Matrix 4.0). Out of the 47 who replied, 85% replied “yes”, 15% replied “unsure”, and none replied “no”.
Fig. 3.
Satisfaction rates of the Report Card leaders collected in the Report Card Leader Surveys 1, 2, and 3 concerning the registration process (A), the Report Card team (B), the Report Card process (C), the final Report Card documents (D), the short Report Card articles and the integrated articles (E), the release of the Report Cards and of the Global Matrix 3.0 (F), the supportive resources provided by AHKGA (G), and the general satisfaction concerning the Global Matrix 3.0 (H).
Table 7.
Summary of the challenges/issues reported in the Report Card Leader Surveys 1, 2, and 3, and in the Mentoring Surveys 1 and 2.
| Mentoring Survey 1 | Mentoring Survey 2 | Report Card Leader Survey 1 | Report Card Leader Survey 2 | Report Card Leader Survey 3 | Total | |
|---|---|---|---|---|---|---|
| Lack of new data/ good quality representative data | 13 | 48 | 14 | 75 | ||
| Difficulty to find funding | 7 | 15 | 26 | 48 | ||
| Benchmarks/indicators are unclear/not detailed enough | 1 | 3 | 12 | 12 | 28 | |
| Lack of support from the mentor | 7 | 6 | 1 | 2 | 1 | 17 |
| Difficulty to recruit Report Card team members/to have a complete Report Card team | 4 | 6 | 1 | 11 | ||
| All the Report Card team members were not responsive/ contributing at the same level | 5 | 4 | 9 | |||
| Registration fees (for the Global Matrix or Movement to Move event) were too expensive | 4 | 1 | 2 | 7 | ||
| The short Report Card articles were too restrictive | 7 | 7 | ||||
| Difficulty to reach a target audience/ have an impact in the media | 6 | 6 | ||||
| Mentored Report Card leader was not responsive/ cooperative | 1 | 1 | 1 | 2 | 5 | |
| Challenging deadline/ complain about deadline change/ unclear deadline | 1 | 3 | 1 | 5 | ||
| Benchmarks and indicator are not relevant and adapted to all context (low and medium HDI countries) | 2 | 1 | 2 | 5 |
Note: This table presents the count of how many times each issue or challenge was mentioned. Several comments contained more than one mention of issue or challenge. Only the issues that were raised a minimum of five times are presented. As there were several open spaces in each survey, the same issue could potentially be mentioned several times by the same Report Card leader.
Outcome indicators
The expected outputs of the Global Matrix 3.0 included the compilation of physical activity grades,6 the production of Report Card documents, the publication of short Report card articles13 and peer-reviewed articles integrating and comparing the international findings.3,10, 11., 12 The Report Card leaders were also invited to rate their confidence in each physical activity grade of their Report Card by entering a number from 1 to 10 in the Report Card Leader Survey 2. The descriptive statistics of their confidence ratings are presented in Table 8.
Table 8.
Descriptive statistics of the confidence ratings for the physical activity grades reported in the Report Card Leader Survey 2.
| PA | SP | AP | AT | SB | PF | FAM | SCH | COM | GOV | |
|---|---|---|---|---|---|---|---|---|---|---|
| Number of replies | 46 | 46 | 45 | 46 | 46 | 44 | 45 | 45 | 46 | 46 |
| Mean (out of 10) | 8.1 | 7.8 | 6.4 | 8.0 | 7.4 | 6.8 | 6.4 | 7.7 | 7.3 | 7.1 |
| Ratings >6 | 85% | 80% | 64% | 83% | 80% | 68% | 60% | 84% | 74% | 67% |
| Max | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 |
| Min | 4 | 2 | 0 | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
Notes: PA = Physical Activity; SP = Organized Sport and Physical Activity; AP = Active Play; AT = Active Transportation; SB = Sedentary Behaviors; PF = Physical Fitness; SCH = School; COM = Community and Environment; FAM = Family and Peers; GOV = Government. Some Report Card leaders specified in comments that that they decided to rate their confidence level “0” for the incomplete grades of their Report Card while some others specified that they rated them “10”.
Discussion
The collective efforts from the Report Card leaders and team members, intermediate and regional mentors, and AHKGA Board of Directors allowed us to collect a variety of quantitative and qualitative information to inform the evaluation of the process, impact, and outcomes of the Global Matrix 3.0.
Overall, high satisfaction ratings were observed concerning the majority of the Global Matrix 3.0 methods, key steps, and concepts identified in Fig. 3. Table 5 and Fig. 2 show that the resources (e-blasts and website) provided by AHKGA were used all year long by the majority of the Report Card leaders, and the Report Card leaders expressed high satisfaction with these supporting materials (Fig. 3G). The general overview of the findings presented in this article shows that despite the challenges and issues encountered, the Global Matrix 3.0 process facilitated the success of all participating Report Card teams to submit their 10 physical activity grades, publish their short Report Card articles, and potentially have a positive impact nationally or internationally. However, several points of improvement were identified to inform recommendations for future Global Matrix initiatives.
Recommendations
Findings from the Mentoring Surveys showed that the mentorship organization created for the Global Matrix 3.0 was too complex and confusing for the participants. Several participants failed to even identify their mentor or mentee country, and lack of support from the mentor was one of the main issues raised in the comments collected in the surveys (Table 7). In practice, most country leaders contacted AHKGA directly for guidance and mentoring. Consequently, it is recommended that central support should be provided by a smaller number of mentors selected by AHKGA that will guarantee their involvement and responsiveness to help guide and mentor new and requesting Report Card teams. It would also be interesting to take advantage of the expertise of all the future Report Card team members with the creation of a discussion/advice forum on the AHKGA website where anyone could post a question and receive a quick answer from available experts or share advice or resources. It is recommended for AHKGA to pursue providing additional support and resources for the development of the Report Cards through e-blasts, the “members only” section of the AHKGA website and the audit process of the grades and their rationale. In addition, the AHKGA has also provided support for the dissemination of the Report Cards through the AHKGA website and Twitter account, and with the provision of a detailed dissemination package described elsewhere.13 Findings presented in Fig. 3G show that Report Card leaders found these resources useful; therefore, AHKGA is encouraged to continue these efforts.
As reported by several Report Card leaders in the surveys (Table 5), the anticipated deadlines for the submission of the final physical activity grades that were provided at the beginning of the Global Matrix 3.0 development stage were very challenging. Once the registration period was closed, the Report Card teams ended with a strict four-month deadline to complete their grade assignment and a six-month deadline to write their short Report Card article. It is recommended that AHKGA aim to provide more lead time in future Global Matrices, allowing more anticipation and flexibility to the Report Card leaders in the planning of their Report Cards development.
Comments collected in the surveys (Table 5) highlighted the challenge faced by AHKGA in the choice, definition and descriptions of the indicators and their associated benchmarks. More detailed and strict criteria for the benchmarks to evaluate the indicators was requested; however, several Report Card leaders complained as well about the lack of good quality data in their country, in particular of data that fits with the benchmarks that were provided by AHKGA. It was also raised that some of the indicators or benchmarks may be too specific and more adapted to very high HDI countries, and not always relevant for the others. Some Report Card leaders suggested that AHKGA should take the leadership on the development of standardized tools dedicated to the collection of physical activity behavior data, or provide a clear recommendation of tools to use for the Report Card teams willing to collect their own data or contributing to the development of new national physical activity surveillance systems. As several Report Card leaders expressed their interest in contributing to the development of more detailed or clear benchmarks or even specific AHKGA tools to assess and analyze the public policies, a feasible approach in the first instance would be for AHKGA to lead an interactive discussion involving all interested Global Matrix participants to update physical activity indicators, their definitions and associated benchmarks. The development of physical activity behavior assessment tools that would be internationally adapted and adopted could become a parallel project to the Global Matrix initiative if adequate resources permit.
With the increasing number of countries participating in the Global Matrix initiative comes a growing need for human, material and financial resources to successfully coordinate its development. Before the initiation of Global Matrix 3.0, AHKGA decided to adopt a registration fee system that increased with the level of HDI of the countries in order to facilitate the participation of the developing countries.3 The lowest satisfaction ratings reported by the Report Card leaders were related to the registration fees and the availability of funding for the design of the Report Cards (Fig. 3A). And Report Card leaders from countries belonging to each category of HDI reported difficulty to find funding in the open spaces for comments of the surveys (Table 7). However, the difficulty to find funding is not a problem that is uniquely tied to the Global Matrix initiative but widespread in scientific research and knowledge translation globally. The Global Matrix 3.0 registration fees were used to pay for the overall management of the Global Matrix 3.0 initiative, to finance the publication of the Global Matrix 3.0 articles in a special issue in JPAH, develop and maintain the AHKGA website, and assist with expenses related to the release at the Movement to Move event in Adelaide Australia. It is recommended that the benefits and supports provided for the registration fee be clearly presented. It is also recommended that the AHKGA explore strategic partnerships in order to obtain financial support and reduce the registration fees for future Global Matrices and/or offer more support to low- and medium-HDI countries. Meanwhile, Report Card leaders and teams need to be creative, persistent, and support each other in their quest to secure funding to support report card developments, releases and promotions.
Despite the generally good confidence ratings concerning the Report Card grades that were reported by the Report Card leaders (Table 8), the lack of good quality data was the issue most leaders noted (Table 4). It is an important recommendation to highlight to all the future participating countries that a lack of good quality data or lack of new data since a previous Report Card has been published is still a relevant and meaningful finding to report. This is particularly true concerning some of the sources of influence indicators (school, community and environment, government). The main message of a Report Card can be to highlight the efforts or the absence of efforts that have been accomplished to tackle the population health issue of children and youth physical inactivity or to improve the surveillance within a country. Regardless of the availability of good quality data, the Report Card remains a potentially powerful tool for advocacy.13
There are still parts of the world missing in the Global Matrix initiative, and the low- and medium-HDI countries and the high HDI countries are also underrepresented in comparison with the very high HDI countries.3 However, the AHKGA is an active network of interconnected researchers and physical activity experts from various fields, distributed across six continents.13 Therefore, to have more participating countries in the next Global Matrices, in particular from the low- and medium-HDI countries, it is recommended that the international network of the AHKGA keep promoting this project through publications, social media, e-mails, conferences and mouth to mouth interactions to connect with new experts and teams from missing countries, in particular the with the low- and medium-HDI countries and the high HDI countries.
Finally, it is recommended that this evaluation process be continuous. Keeping ongoing records of media impact, publications, presentations, success stories, and issues that arise will need to be proactively pursued in order to facilitate the improvement and optimize the impact of this global initiative.
Strengths and limitations
The main limitation of this evaluation was that it relied essentially on responses to surveys and on reports made by the Report Card leaders, the Report Card team members, the mentors, and the AHKGA Board members. Consequently, the results may be biased and/or incomplete, potentially making the opinion of individual who replied more consistently and reported more information weighting more than others. This evaluation plan was also designed and led by members of the AHKGA Board of Directors, constituting a potential source of personal bias. On the other side, a complete understanding of the Global Matrix and Report Card harmonized development process was essential to the design and understanding of this evaluation process. In addition, this evaluation plan allowed to collect feedback from the Report Card teams at all the stages of development of the Global Matrix 3.0, allowing to identify potential issues that were still addressable at the moment they were raised.
Conclusion
A variety of quantitative and qualitative information was collected to inform the evaluation of the process and outcomes of the Global Matrix 3.0. From these evaluations the Global Matrix 3.0 was a successful international initiative in terms of delivery of the expected outcomes - the physical activity grades for each participating country - despite several identified issues that challenged the Report Card teams. This evaluation process allowed for the identification of several improvements and the formulation of recommendations for future Global Matrix initiatives and highlighted the need for the development of physical activity behavior assessment tools that would be internationally adopted and culturally adaptable for all contexts to improve the standardization of physical activity surveillance at the global scale.
Declaration of competing interest
The authors want to disclose that they were members of the AHKGA Board of Directors during the development of the Global Matrix 3.0.
Acknowledgements
The authors want to thank the current and previous AHKGA Board of Director’s members, the Report Card leaders, and their team members who contributed actively to the successful development of the Global Matrix 3.0 and its evaluation. We are also grateful to Evan Turner for the support he provided during the audit of the short Report Card articles, the writing of the integrated articles, and the evaluation of the international impact of the Report Cards.
Contributor Information
Salomé Aubert, Email: saubert@cheo.on.ca.
Joel D. Barnes, Email: jbarnes@cheo.on.ca.
Mark S. Tremblay, Email: mtremblay@cheo.on.ca.
References
- 1.Tremblay M.S., Gray C.E., Akinroye K. Physical activity of children: a global Matrix of grades comparing 15 countries. J Phys Act Health. 2014;11(s1):S113–S125. doi: 10.1123/jpah.2014-0177. [DOI] [PubMed] [Google Scholar]
- 2.Tremblay M.S., Barnes J.D., González S.A. Global Matrix 2.0: report card grades on the physical activity of children and youth comparing 38 countries and the global Matrix 2.0 research team. J Phys Act Health. 2016;13(S2):S343–S366. doi: 10.1123/jpah.2016-0594. [DOI] [PubMed] [Google Scholar]
- 3.Aubert S., Barnes J.D., Abdeta C. Global Matrix 3.0 physical activity report card grades for children and youth: results and analysis from 49 countries. J Phys Act Health. 2018;15(S2):S251–S273. doi: 10.1123/jpah.2018-0472. [DOI] [PubMed] [Google Scholar]
- 4.Colley R.C., Brownrigg M., Tremblay M.S. A model of knowledge translation in health. Health Promot Pract. 2012;13(3):320–330. doi: 10.1177/1524839911432929. [DOI] [PubMed] [Google Scholar]
- 5.Active Healthy Kids Global Alliance . 2018. About » Active Healthy Kids Global Alliance.https://www.activehealthykids.org/about/ Published. [Google Scholar]
- 6.J Phys Act Health. 2018;15(S2):S251–S435. doi: 10.1123/jpah.2018-0520. https://journals.humankinetics.com/view/journals/jpah/15/s2/jpah.15.issue-s2.xml The Global Matrix 3.0; Sponsored by the Active Healthy Kids Global Alliance. [DOI] [PubMed] [Google Scholar]
- 7.Dumuid D., Maher C., Lewis L.K. Human development index, children’s health-related quality of life and movement behaviors: a compositional data analysis. Qual Life Res. 2018;27(6):1473–1482. doi: 10.1007/s11136-018-1791-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Hastings D.A. Bangkok; 2009. Filling Gaps in the Human Development Index: Findings for Asia and the Pacific.https://www.unescap.org/sites/default/files/wp-09-02.pdf [Google Scholar]
- 9.Land K.C. Global Handbook of Quality of Life. Springer Netherlands; Dordrecht: 2015. The human development index: objective approaches (2) pp. 133–157. [DOI] [Google Scholar]
- 10.Manyanga T., Barnes J.D., Abdeta C. Indicators of physical activity among children and youth in 9 countries with low to medium human development indices: a global Matrix 3.0 paper. J Phys Act Health. 2018;15(S2):S274–S283. doi: 10.1123/jpah.2018-0370. [DOI] [PubMed] [Google Scholar]
- 11.González S.A., Barnes J.D., Abi Nader P., Tremblay M.S. Report card grades on the physical activity of children and youth from 10 countries with high human development index – global Matrix 3.0. J Phys Act Health. 2018;15(S2):S284–S297. doi: 10.1123/jpah.2018-0391. [DOI] [PubMed] [Google Scholar]
- 12.Aubert S., Barnes J.D., Aguilar-Farias N., Tremblay M.S. Report card grades on the physical activity of children and youth comparing 30 very high human development index countries. J Phys Act Health. 2018;15(S2):S298–S314. doi: 10.1123/jpah.2018-0431. [DOI] [PubMed] [Google Scholar]
- 13.Aubert S., Barnes J.D., Forse M. The international impact of the active Healthy Kids global alliance physical activity report cards for children and youth. J Phys Act Health. 2019;16(9):679–697. doi: 10.1123/jpah.2019-0244. [DOI] [PubMed] [Google Scholar]
- 14.Lobo R.G., Brown B.M., McManus A. Perth; 2010. Development of an Evaluation Framework and Evaluation Approaches for Peer-Based Youth Programs – Interim Report. [Google Scholar]
- 15.Mitchell R.J., Williamson A.M., O’Connor R. The development of an evaluation framework for injury surveillance systems. BMC Public Health. 2009;9(1):260. doi: 10.1186/1471-2458-9-260. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Harris P.A., Taylor R., Thielke R., Payne J., Gonzalez N., Conde J.G. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Canadian Institutes of Health Research Natural sciences and engineering research council of Canada, social sciences and humanities research council. tri-council policy statement ethical conduct for research involving humans. Ottawa, Canada. 2018. http://www.pre.ethics.gc.ca/eng/documents/tcps2-2018-en-interactive-final.pdf