Table 3.
Not Useful | One Useful | Two Useful | All | |
---|---|---|---|---|
Pre-Delphi | 81 | 28 | 15 | 124 |
Post-Delphi | 19 | 10 | 3 | 32 |
Near-Agreement | 14 | 7 | 4 | 25 |
All | 114 | 45 | 22 | 181 |
We stratified the selected 200 relationships by two dimensions: (1) expert agreement (based on the Delphi rounds), which served as a proxy for task difficulty where less expert agreement implied higher task difficulty, and (2) expert-rated definition utility, which served as a proxy for definition quality for a relationship where the greater number of useful concept definitions for a relationship implied higher task definition quality. Note that some strata contain very few relationships. (For row label explanation see ‘Task Difficulty’ on page 12. For column label explanation see ‘Definition Quality’ on page 13.)