Table 2.

Design process

1. Understand the system being measured, including both technical55 and organizational20 considerations.
Determine scope	What is included in the system? What will the metrics be used for?
Understand the causal structure of the system	What is the logic model or theory?56 Is there formal analysis57 or expert opinion58 that can inform this?
Identify stakeholders59	Who will be affected? Who will use the metrics? Whose goals are relevant?
2. Identify the goals	What immediate goals are being served by the metric(s)? How are individual impacts related to performance more broadly?60 What longer-term or broader goals are implicated?
3. Identify relevant desiderata	•Availability •Cost •Immediacy •Simplicity	•Trust •Fairness •Non-corruptibility
4. Brainstorm potential measures for metrics	What outcomes are important to capture? What data sources exist? What methods can be used to capture additional data? What measurements are easy to capture? What is the relationship between the measurements and the outcomes? What is not captured by the measurements?
5. Consider and plan	Understand how measurements will be used to build metrics.61 Consider how the metrics will be used to diagnose issues or incentivize people.30 Consider the use of soft metrics to triangulate.41 Consider avoiding the “reward/punish” dichotomy.62 Identify and mitigate likely failure modes with pre-mortems.34
6. Plan to revisit the metrics	Set a specific date or interval for routine re-evaluation. Identify additional triggers for non-routine re-evaluation.