Table 2.

Evaluation framework and results.

Principle	Evaluation Question(s)	Indicator(s)	GrizzTracker Application
1. Citizen science programs include a stated purpose and/or scientific outcome, such as generating new knowledge or informing conservation actions, environmental management decisions, or environmental policy.	What is the stated purpose and/or scientific outcome of the program? Are the scientific outcomes sufficiently clear?	Documentation of the program purpose, goals, and/or desired scientific outcomes (e.g., a program plan, a conceptual framework).	Scientific need: lack of a rigorous dataset on grizzly bears to use in recovery management planning. Program Goals: The Northwest Grizzly Bear Team identified the following goals: Improved grizzly bear population modeling; Improved human and wildlife safety; A platform for stakeholder learning; Deeper ecological literacy and a stronger sense of place.
2. Citizen science data are fit to function, collected using standards and protocols appropriate to the purpose and/or scientific outcome, and follow scientific practices in design, implementation, data quality assurance, data management, and evaluation.	How does the program design match the program purpose and/or scientific outcome? How does the program attend to quality assurance and quality control measures needed to produce rigorous, high-quality data? How do participant training and resources match the task (i.e., data collection)?	Documentation of the program design with specific outcome statements. Quality assurance and quality control measures for data (e.g., expert data verification). Participants provided with appropriate training and resources.	Intended purpose of the program: data would supplement grizzly bear monitoring, and public participation would increase scientific and bear awareness, knowledge, and skills. Data collection methods: a smartphone application was developed that supported standardized, automated, and rigorous collection of grizzly bear sightings, including a testing functionality to record observer effort. Quality control measures: Participants were asked to provide confidence in species identification. Unconfident records were removed from the analysis. Participants were provided with training sessions on the program and grizzly bear safety and conservation. A supporting website was developed that included a grizzly bear identification guide and quiz. Data were tested for bias and outliers during analysis. Similar attribute data were standardized between datasets. Post-program assessment: follow-up was limited due to capacity/resourcing limitations; however, an evaluation was conducted (see [35]). Additionally, there remains a lack of clarity on the utility of data for applied management.
3. Citizen science programs operate in an open and transparent manner and, where appropriate, project data, applications, and technologies are shared to encourage a culture of sharing and rapid innovation.	What data collection tools are being used and if new tools were designed could they be shared? Are data ownership and access rights clear and transparent? How is the project making data available? To whom? How is the project sharing results? With whom?	Data and results are shared with participants in suitable formats (e.g., data visualizations).	Open-source technology: a smartphone application was developed using open-source technology and shared/used in other citizen science programs. Data Sharing: Northwest Grizzly Bear Team members were provided with access to anonymized raw data via an administration portal. The project website enabled transparency and accessibility of results by displaying reported observations on a mapping tool after a two-day delay.
4. Citizen science programs are inclusive and encourage active, meaningful, and productive citizen participation.	What participant recruitment strategies were used to engage with a diversity of participants (e.g., gender, age, ethnicity)? To what degree are participants involved in project tasks (e.g., defining research questions, data collection, results interpretation, reporting)?	A diversity of participants are engaged throughout the project. Engagement in all aspects of the program, including defining the research questions and design, identifying objectives, data collection, analysis, and reporting/communicating outcomes.	Diversity and Inclusion: participation was initially limited. After a pilot phase, the program was opened up to broader public participation. Opportunities to engage: public participation focused on data collection, with program design, testing, and implementation performed by a representative stakeholder group.
5. Citizen science programs are designed to provide benefits to all participants, including citizens, practitioners, and researchers. Benefits include publishing research outputs, learning opportunities, personal enjoyment, social interaction, and contributing to scientific evidence. Whenever possible, with permission, participants should be acknowledged in project results and publications.	How did participant perspectives inform the program design? How are participants provided with ongoing opportunities for co-learning and sharing knowledge? How do both the researchers and participants benefit? How are participant contributions acknowledged?	Discussion of potential benefits to participants, including developing their skills or the creation of new knowledge, to help inform environmental decisions. Mechanisms to support co-learning and knowledge sharing exist. Participant satisfaction. Evidence of appropriate acknowledgement of participant effort (e.g., scientific publications, communications, products, public events).	Participants: a multi-stakeholder project team (the Northwest Grizzly Bear Team) with representatives from the Government of Alberta, energy and forestry resource sectors, environmental non-governmental organizations, and an academic research institute was established to identify program goals and benefits. Co-learning and sharing: The program provided a platform for shared learning between land managers and industry through collaborative program design, program implementation, and the sharing of findings. Acknowledgement: The participants were acknowledged during presentations and in published reports. Co-learning and sharing: The response rate to inquiries and the provision of feedback were delayed due to a lack of capacity/resourcing.
6. Citizen science programs take into consideration safety, legal, and ethical standards and guidelines surrounding copyright, intellectual property, confidentiality, data sharing agreements, and the environmental impact of any activities.	How does the project consider participant safety? How does the project consider existing policies and regulations that apply to the governance and management of data and information?	Protocols are established and participants trained on the protocols. Documentation of ethical research practices such as data sharing agreements.	Safety: Volunteer training provided a platform to provide educational information on safety, including human and bear conflict. Policy and Regulation: Data are owned by the Government of Alberta, although data and reports are shared with project partners and the broader public. Raw data for threatened species are not shared publicly. Observations are shared on a mapping tool but with a two-day delay to protect a species at risk.