Table 2.

CA Notify key performance indicators: observations and implications, December 12, 2020–July 1, 2021 ^a

Dimension	Observations	Calculations and estimates	Implications
Adoption (smartphone activations)	• Access restricted to smartphone users who have compatible operating systems (iOS or Android) and are able to overcome potential barriers of language, readability, and usability. • Usability and trust scales across focus group populations suggest that acceptability varied. • Activations peaked in the first week after release.	• Potential CA Notify access is about 85% of population (upper limit estimate of eligible smartphone users based on Pew study). 21 • Actual activations: about 29% of state residents (10.9 million) estimated to have activated CA Notify based on 9 998 611 iOS and 912 360 Android counts. • Acceptability: High trust scores from focus group testing.	• Adoption highest in the launch phase when Apple pushed availability alerts. • Ongoing engagement strategies are needed. • Key groups remain disenfranchised or have access issues.
Retention (active users)	User choices: (1) maintain CA Notify, (2) turn off CA Notify, (3) turn on/off.	Under investigation.	• Engagement strategies required to improve adoption and retention especially among groups at high risk for disease or serious outcomes. • Need for methods to accurately measure active daily users and key risk factors.
Use (appropriate actions followed)	Code use: users must claim and use (share) codes to release EN alerts.	• Number of codes claimed: 446 634 (27% of codes issued) • Number of codes used: 68 107 (15% of codes claimed)	• Codes claimed map to adoption numbers for California population. • Code use can be influenced by improved SMS and more frequent touchpoints.
Identification (identification of new contacts/cases)	Estimated by number of codes used and number of EN page visits.	• Number of EN page visits: 122 970 • Mean code use interval: 4.7 d • Mean (95% CI) [SD] identification ratio: 1.8 (0.2-11.3) [1.76]	• Estimated identification rate is higher than using traditional contact tracing in California. • Likely discovers contacts that do not overlap with traditional methods.
Behavior (change in behavior after receiving a notification)	An EN is received and read, leading to recommended/expected behavior change/action.	Direct assessment of behavior change is limited because of privacy constraints.	• Privacy-preserving architecture can limit measurement of key performance indicators. • Balances must be struck between these dimensions. • Prospective and retrospective surveys can be deployed to gather more information in an opt-in manner.
Impact (measuring public health outcomes and implications)	• Integration of key performance indicators. • Model estimates of cases reached, cases averted, time to recognition change, hospitalizations, deaths, and cost.	Appropriate models that accurately predict/determine impact are lacking for US implementation of EN.	• Preliminary models indicate likely impact; however, none is tailored to the US experience with EN. • Further models are needed.

Abbreviations: EN, exposure notification; SMS, short message service.

^aCA Notify is a digital EN system in California that uses smartphones to anonymously identify and alert individuals who have been in proximity to a person who has received a positive test result for SARS-CoV-2.