Table 2.
Research engagement actions, subactions, subaction levels, raw utilities, standard errors, and rescaled utility coefficients
| Research engagement action | Subaction | Levels of each subaction | Raw utility coefficient | Standard error | Rescaled utility coefficient† |
|---|---|---|---|---|---|
| 1. Searching for research | a. Policymaker searched academic literature databases and/or physical libraries | i) No | −3.59*** | 0.27 | 0 |
| ii) Yes | 0 | – | 2.83 | ||
| b. Policymaker searched grey literature sources | i) No | −1.81*** | 0.16 | 0 | |
| ii) Yes | 0 | – | 1.42 | ||
| c. Policymaker obtained research by chance, research that was on-hand, or provided by colleagues | i) No | −1.37*** | 0.11 | 0 | |
| ii) Yes | 0 | – | 1.08 | ||
| d. Policymaker requested experts (researchers, working groups, librarians, or other research experts) to identify research | i) No | −1.98*** | 0.16 | 0 | |
| ii) Yes | 0 | – | 1.56 | ||
| e. Policymaker searched generic databases or search engines | i) No | −1.12*** | 0.11 | 0 | |
| ii) Yes | 0 | – | 0.88 | ||
| f. Policymaker examined reference lists, citation indices, or databases of references | i) No | −1.55*** | 0.14 | 0 | |
| ii) Yes | 0 | – | 1.22 | ||
| 2. Research obtained and used | a. Policymaker found systematic reviews and/or meta-analyses | i) No | −1.94*** | 0.24 | 0 |
| ii) Yes | 0 | – | 3.29 | ||
| b. Policymaker found books and/or technical monographs | i) No | −0.34*** | 0.12 | 0 | |
| ii) Yes | 0 | – | 0.58 | ||
| c. Policymaker found primary research and/or theoretical research | i) No | −0.99*** | 0.19 | 0 | |
| ii) Yes | 0 | – | 1.67 | ||
| d. Policymaker found unpublished research and/or conference resources | i) No | −0.49*** | 0.13 | 0 | |
| ii) Yes | 0 | – | 0.82 | ||
| e. Policymaker found internal policies, evaluations, or data | i) No | −0.23* | 0.10 | 0 | |
| ii) Yes | 0 | – | 0.39 | ||
| f. Policymaker found policies, evaluations, or data from external organisations or registries | i) No | −0.60*** | 0.11 | 0 | |
| ii) Yes | 0 | – | 1.01 | ||
| g. Policymaker obtained recent (up-to-date) research from the above categories | i) No (Older research) | −0.73*** | 0.15 | 0 | |
| ii) Yes (Recent) | 0 | – | 1.24 | ||
| 3. Appraising relevance | a. Policymaker assessed whether the research was applicable to the policy context or policy issue | i) No | −1.70*** | 0.17 | 0 |
| ii) Yes | 0 | – | 2.06 | ||
| b. Policymaker assessed whether research recommendations were actionable and/or feasible? | i) No | −1.53*** | 0.18 | 0 | |
| ii) Yes | 0 | – | 1.84 | ||
| c. Policymaker assessed whether the research was consistent with previous research on the issue | i) No | −1.02*** | 0.15 | 0 | |
| ii) Yes | 0 | – | 1.23 | ||
| d. Policymaker assessed if research was compatible with his/her OR the organisation’s values, knowledge, or experience | i) No | −0.97*** | 0.14 | 0 | |
| ii) Yes | 0 | – | 1.17 | ||
| e. Policymaker consulted experts to assess the relevance of research | i) No | −1.07*** | 0.12 | 0 | |
| ii) Yes | 0 | – | 1.29 | ||
| f. Policymaker undertook these actions as part of a pre-specified strategy | i) No (ad-hoc, unplanned) | −1.16*** | 0.18 | 0 | |
| ii) Yes | 0 | – | 1.40 | ||
| 4. Appraising quality | a. Policymaker assessed whether the design or conclusions of the research were valid | i) No | −1.16*** | 0.22 | 0 |
| ii)Yes | 0 | – | 2.00 | ||
| b. Policymaker evaluated whether the design or conclusions of the research were described clearly and comprehensively | i) No | −0.68*** | 0.16 | 0 | |
| ii) Yes | 0 | – | 1.17 | ||
| c. Policymaker assessed whether the source of the research was credible | i) No | −0.64*** | 0.12 | 0 | |
| ii) Yes | 0 | – | 1.10 | ||
| d. Checked if the research cited, or was referenced in other high-quality research or policy documents | i) No | −0.45** | 0.18 | 0 | |
| ii) Yes | 0 | – | 0.77 | ||
| e. Policymaker consulted experts to assess quality | i) No | −0.76*** | 0.19 | 0 | |
| ii) Yes | 0 | – | 1.31 | ||
| f. Policymaker assessed the level of evidence of the research | i) No | −0.88*** | 0.17 | 0 | |
| ii) Yes | 0 | – | 1.51 | ||
| g. Policymaker undertook these actions as part of a pre-specified strategy | i) No (ad-hoc, unplanned) | −0.67*** | 0.22 | 0 | |
| ii) Yes | 0 | – | 1.15 | ||
| 5. Generating new researchers | a. Policymaker expressed explicit intentions to generate or commission new research (to follow-up the current policy) OR stated that he/she had already undertaken this research | i) No (no intentions to generate new research) | −2.08*** | 0.28 | 0 |
| ii) No (uncertain intentions only) | −1.97*** | 0.27 | 0.18 | ||
| iii) Yes | 0 | – | 3.42 | ||
| b. Policymaker mentioned thorough research generation activities | i) No | −1.72*** | 0.19 | 0 | |
| ii) Yes | 0 | – | 2.84 | ||
| c. Policymaker mentioned less intensive research activities | i) No | −0.96*** | 0.14 | 0 | |
| ii) Yes | 0 | – | 1.58 | ||
| d. Policymaker advocated for future research to be undertaken | i) No | −0.60*** | 0.15 | 0 | |
| ii) Yes | 0 | – | 0.99 | ||
| 6. Interacting with researchers | a. Policymaker engaged in thorough collaborative activities with researchers | i) No | −2.56*** | 0.25 | 0 |
| ii) Yes | 0 | – | 3.75 | ||
| a. Policymaker engaged in less intensive interactions with (other) researchers | i) No | −0.91*** | 0.11 | 0 | |
| ii) Yes | 0 | – | 1.33 | ||
| b. Policymaker engaged in sporadic contact with (other) researchers? | i) No | −0.67*** | 0.11 | 0 | |
| ii) Yes | 0 | – | 0.98 | ||
| c. Policymaker actively initiated these interaction activities | i) No | −2.01*** | 0.22 | 0 | |
| ii) Yes | 0 | – | 2.94 |
†Utility coefficients were rescaled so that they became positive, with the lowest level of each subaction having a zero-coefficient, and adding up to 9.
***p < 0.001; **p < 0.01; *p < 0.05.