Table 2.
Descriptions of the reliability scores awarded to studies based on their results from the screening and data quality assessment (see Table 1)
| Reliability score | Screening evaluation | Data quality score | Description | ERA Interpretation |
|---|---|---|---|---|
| R1 | Pass RQ1–RQ5 | ≥80% | This study is well designed and of high quality. No significant issues identified that reduce the reliability. | Potentially suitable for regulatory decision‐making/higher tier ERA. |
| R2 | Pass RQ1–RQ5 | ≥70%–79% | A well‐designed and executed study with minor limitations that somewhat reduce the reliability of the results. | Potentially suitable for regulatory decision‐making/higher‐tier ERA (secondary to R1 studies, if available). |
| R3 | Pass RQ1–RQ5 | ≥60%–69% | The study design and/or execution contained many minor limitations or a major limitation that significantly reduces the reliability of the results. | Preliminary assessment only. Can serve as additional line of evidence, with limitations stated. Useful for prioritizing higher quality studies. |
| Fail RQ1; pass RQ2–RQ5 | ≥60% | |||
| Fail 1 of RQ3–RQ5c | ≥70% | |||
| R4 | Pass RQ1–RQ5 | ≥50%–59% | The study contains many limitations to the point where the results should be interpreted with caution. | Apply expert judgment to determine whether useful for preliminary assessment, but clearly state limitations. b Can serve as additional line of evidence. |
| Fail RQ1; pass RQ2–RQ5 | ≥50%–59% | |||
| Fail 1 of RQ3–RQ5c | ≥60%–69% | |||
| R5 | Pass RQ1–RQ5 | <50% | The study has major design flaws and/or is poorly executed and cannot be considered reliable. | Study not useful for preliminary assessment. Can be supporting evidence if result similar to higher scoring study. |
| Fail RQ1; pass RQ2–RQ5 | <50% | |||
| Fail 1 of RQ1–RQ5c | ≥50%–59% | |||
| R6 | Fail 1 of RQ1–RQ5c | <50% | The study design and/or execution is unsuitable for ERA and the results are highly unreliable. | Disregard, study not reliable or useful for ERA (even as supporting evidence). |
| NA1 | Fail RQ2 | N/A | Study does not pass relevance screening. Data quality score not evaluated. | Disregard, study not useful for problem formulation. |
| NA2 | Fail ≥2 of RQ3–RQ5 | N/A | Study does not pass reliability screening. Data quality score not evaluated. | Disregard, study not reliable or useful for ERA. |
aThe scoring is meant to serve as a guide to help derive a transparent and consistent reliability score, but expert judgment and context should also be considered when awarding the final reliability score.
b
These studies can be used at the preliminary assessment stage, but priority should be given to replacing with higher quality data.
c
In this case RQ1 can either be passed or failed, but RQ2 must be passed. If RQ2 failed, award reliability score of NA1.
ERA = environmental risk assessment.