Table 4.
Description of the 12 indicators that were not recommended
| Rationale and definitions | Numerator Denominator | Methodological challenges and possible improvements | |
|---|---|---|---|
| INDICATORS FOUND ACCEPTABLE/IN NEED OF ADJUSTMENTS | |||
| 1. Rejected referrals | Insufficient referral information makes the specialists less confident in their decisions on whether the referral request should be rejected or not. High quality referral letters can better enable specialists to reject patients not in need of specialised mental health care (instead of seeing them to be “on the safe side”) than can those of low quality. Rejected referral is defined as referral assessed by a specialist as not meeting the criteria for receiving specialised mental health care. | No. rejected referrals No. referrals (total) | Different potential confounding factors. We lack a definition of the optimal number of rejected referrals. Calibration of what the goal should be within different health care systems is needed. A careful exploration to ensure that the rejected referrals are the right ones is essential. |
| 2. Aborted episodes of care | Less informative referral letters can result in incorrect access to specialised health care. This is often detected during the first consultations and the patient is then discharged. Aborted episode of care is defined as terminated by the service after ≤ 3 consultations because of incorrect access to specialised mental health care. | No. episodes of care aborted after ≤ 3 consultations No. episodes of care started | Risks of false positive findings as some episodes of care are completed in 3 or fewer consultations. |
| 3. Severity in high priority patient groupa (severity factors) | High quality referral letters can enable specialists to prioritise patients most in need, as defined by the existence of several ‘severity factors’. ‘Severity factors’ are defined as severe mental illness, risk of suicide, risk to others, care for children, substance abuse and being younger than 23 yearsb. | No. patients with 3 or more severity factors in the high priority group No. all patients in high priority group | Risk of both false positive and false negative findings as the existence of 3 factors does not necessarily indicate a larger severity than 2 factors. |
| 4. Realism in expectation toward specialised mental health care | The realism of expectations toward specialised health care formulated in the referral letter, as assessed by the receiving specialist, can be an indicator for the common understanding of the responsibilities of various services. Degree of realism is assigned a score (0–3). | No. letters with score 2 or 3 No. all referral letters | Uncertainty regarding the causal chain. Some of the present letters do not specify expectations (= missing data). |
| 5. Supportive information gathering | Different initiatives by the specialist to gather additional information are needed when referral letters do not convey the information necessary to decide if and when specialised health care should be conducted. Supportive information gathering is defined as extra activities, such as contacting the referring GP or the patient, conducted by the specialised health care because of insufficient information in the referral letter. | No. activities No. referral letters | Contextual variation in the tradition of collecting additional information is a confounding factor. Very high or very low results should be interpreted with caution. A qualitative exploration of the specialists’ reasons for collecting (or not collecting) additional information is recommended when initiatives are almost always or never taken. |
| INDICATORS FOUND UNACCEPTABLE | |||
| 6. Severity in high priority patient groupa (diagnosis) | High quality referral letters can better enable specialists to select the patients most in need than can referral letters of low quality. | No. patients with diagnosis of severe illness No. all patients | The diagnosis is not a valid indicator for the degree of need for specialised mental health care. |
| 7. Common understanding of the coordinated care plan | High quality referral letters may facilitate a common understanding of the overall plan for the coordinated care among the involved service providers. A survey where involved professionals tick off the interventions/services they think are involved in each patient’s care plan will reveal the degree of common understanding. | No. plans with a high degree of agreement No. all plans | The integrated plan is not usually defined on the basis of the referral information. Low feasibility. |
| 8. Adequate specialist response (referring GP) | High quality referral letters include a well-defined request that can better facilitate an adequate response than can low quality referral letters. Adequate response is defined as 2 or 3 on an ordinal scale from 0 to 3, assessed by the referring GP. | No. letters with score 2 or 3 No. all referral letters | The response depends on factors in addition to the GP’s request, reducing the validity. Many referral letters do not include a concrete, explicit request, negatively affecting the feasibility. |
| 9. Adequate specialist response (patient) | High quality referral letters include a well-defined request that can better facilitate an adequate response than can low quality referral letters. Adequate response is defined as 2 or 3 on an ordinal scale from 0 to 3 on the adequacy of the specialised health care response assessed by the referred patient. | No. letters with score 2 or 3 No. all referral letters | As for indicator 8. Patient involvement in defining the referral letter is often limited (i.e., the patient is seldom fully aware of, or may not fully agree to, the formulated request), reducing the validity. |
| 10. Time to decide priority | Specialists are expected to spend less time assessing high quality referral letters than low quality referral letters. Time is defined as minutes used for assessing the referral letter including time for gathering extra information. | Minutes to decide priority No. referral letters | The decision is often made step by step including individual assessment and interdisciplinary discussion in the team, negatively affecting the feasibility. Long letters can be of high quality but take more time to read. |
| 11. Attendance to first consultation | Informative referral letters can enable facilitation of the first consultation to the patient’s needs, reducing the risk of non-attendance. | No. non-attending patients No. all patients for first consultation | Several confounding factors expected. Limited sensitivity. |
| 12. Attendance to consultations in first 3 months | High quality referral letters can be associated with less drop-out in the first 3 months of treatment by enabling facilitation, compared with low quality referral letters. | No. drop-out in first 3 months No. all patients completing 3 months of treatment | As for indicator 11. Facilitation is usually based on information provided by the patient rather than by the referral letter. |
aAccording to the legal rights for patients in Norway, all patients referred to specialised health care are prioritised by a specialist and given high priority, which entails a legal right to health care with a (medically) defined deadline for when health care should take place; low priority, which means the patient will receive health care, but there is no guarantee as to when it will be conducted; or no priority, which means the patient is not in need of specialised mental health care. The assessment is usually done on the basis of the referral letter, but more information can be gathered
bIn the Norwegian health care system, patients under 23 years old with a substance abuse problem are, by law, given priority when referred to specialised mental health care