Psychometric Properties of the OSPRO–YF Screening Tool in Patients with Shoulder Pathology

Abstract

Purpose: The Optimal Screening for Prediction of Referral and Outcome Yellow Flag (OSPRO–YF) is a screening tool that incorporates many important psychosocial domains into one questionnaire to reduce the burden of completing multiple questionnaires. The objectives of this study were to examine the reliability and validity of the 10-item version of the OSPRO–YF with patients with shoulder conditions. Method: The study group consisted of injured workers with an active compensation claim for a shoulder injury. The control group consisted of patients with a complaint of shoulder pain but without a work-related shoulder injury. We examined reliability (internal consistency, test–retest) and validity (factorial, convergent, known groups). The Hospital Anxiety and Depression Scale; the Quick Disabilities of Arm, Shoulder and Hand; and the short Örebro Musculoskeletal Pain Screening Questionnaire were used for comparison. Results: Eighty patients had an active compensation claim, and 160 were in the control group. The intra-class correlation coefficient values for two observations of the domain scores varied from 0.91 to 0.94. The test–retest reliability of the dichotomous constructs was moderate to perfect for 8 of 11 constructs. The 10-item OSPRO–YF questionnaire had three distinct domains, as conceptualized by the developers: mood, fear avoidance, and positive affect–coping. The Cronbach’s a coefficients for these domains were 0.88, 0.94, and 0.94, respectively. The associations between the psychological constructs and domains and the similar theoretically derived scales were moderate to high and in the expected direction. Of the 11 constructs of the OSPRO–YF, 10 differentiated between patients with and without a work-related injury (p-values ranging from 0.028 to < 0.001). Conclusions: The 10-item OSPRO–YF reduces the burden of using multiple questionnaires and has acceptable test–retest and internal consistency reliability and factorial, convergent, and known-groups validity.

Yellow psychosocial flags refer to fears of pain or injury, negative pain beliefs, and distressed affect and are considered risk factors for prolonged disability.¹ Although psychosocial factors have been examined more extensively in relation to low back pain, their role in the recovery of patients with shoulder pathology has more recently received attention.^2–7 The overall adverse impact of negative attitudes, depression, anxiety disorders, pain-related distress, and societal factors on a patient’s clinical presentation and response to treatment has been well established,^{1, 8–12} but there is little agreement on the best tool for measuring mental health and psychosocial well-being in a musculoskeletal patient population. A number of tools, such as anxiety and depression scales,¹³ the Fear-Avoidance Beliefs Questionnaire (FABQ),¹⁴ and the short Örebro Musculoskeletal Pain Screening Questionnaire (ÖMPSQ–10),¹⁵ have been used to assess various psychological determinants of musculoskeletal disorders; however, each tool is limited in its scope, and clinicians resort to using multiple tools in combination to obtain a good understanding of a patient’s psychosocial status.

The Optimal Screening for Prediction of Referral and Outcome Yellow Flag (OSPRO–YF) is a newly developed screening tool that incorporates key psychosocial constructs into one questionnaire to reduce the burden on patients and clinicians.¹⁶ Its domains include negative mood, fear avoidance, and positive affect–coping. It may be used with patients with a variety of musculoskeletal disorders such as neck, back, knee, and shoulder pain. The original study on the development of the 7-, 10- and 17-item versions of the OSPRO–YF was published in 2016,¹⁶ and several subsequent studies have examined the predictive value of its scores in the development of persistent pain and future use of health care.^17–19 Although each study conducted by the tool’s developers has added to the understanding of this measure’s performance, the same cohort appears to have been used, which may limit the generalizability of the findings.

Reliability and validity are properties of a test in a specific context, for a specific purpose, or in a particular population. They are established incrementally as evidence increases with more research; thus, an ongoing evaluation of the suitability of newly developed tests in different situations is warranted. To date, we are not aware of any independent studies of the OSPRO–YF’s reliability or validity. Given the importance and cost of pain-associated psychological disability after occupational injuries and the potential utility of the OSPRO–YF in identifying patients who could benefit from conjunct psychological assessment and treatment, further evaluation of this screening tool in an independent sample of patients is useful. The objectives of this study were to examine the reliability and validity of the 10-item version of the OSPRO–YF in patients with shoulder conditions.

Methods

Design

This was a case–control study. The cases (study group) had sustained a work-related injury to the shoulder joint and had an active compensation claim. The control group consisted of patients without an active compensation claim. We hypothesized that patients with an active compensable injury would have a higher risk of psychological concerns. This hypothesis was based on the literature, which indicates a differential level of disability and recovery among individuals who have sustained a work-related shoulder injury.^20–26 All subjects were recruited from the same academic tertiary centre, and data were collected from both samples in the same time frame.

Subjects

The cases were assessed on their initial visit by a trained physical therapist and a shoulder specialist orthopaedic surgeon in an upper extremity specialty programme funded by the Ontario Workplace Safety and Insurance Board (WSIB). The WSIB provides parallel pay insurance and finances expedited access to assessment by a specialist and management through surgery.

The controls were patients with a complaint of shoulder pain who had been referred to a publicly funded tertiary shoulder clinic by their family physicians. They were initially assessed by advanced practice physiotherapists (APPs) with advanced training (MSc, PhD) and more than 10 years of experience. The APPs ordered the necessary imaging investigations through delegation mechanisms and provided appropriate management, including a referral for surgery.

All patients were approached at the initial assessment. Inclusion criteria were being aged older than 18 years and having the ability to write and read English. Exclusion criteria were a diagnosis of acute shoulder dislocation, infection, fracture, chronic pain syndrome, fibromyalgia, or diabetic neuropathy or receiving active treatment for a psychological condition. All patients provided informed consent, and subjects’ rights were protected. Approval for the use of human subjects was obtained from the Research Ethics Board of the Sunnybrook Health Sciences Centre.

OSPRO–YF

The OSPRO–YF has 17-, 10-, and 7-item versions.¹⁶ We elected to examine the 10-item version because its accuracy is fairly similar to that of the full 17-item version, and more information is available about its predictive value.^19,27 The 10-item version of the OSPRO–YF uses Questions 1, 3, 4, 7, 8, 10, 11, 14, 15, and 17 from the 17-item version and addresses three domains: negative mood, fear avoidance, and positive affect–coping.

The OSPRO–YF questionnaire provides both continuous data (11 psychological construct scores, three domain scores, and a summary score) and dichotomous data (the presence or absence of yellow flags for 11 constructs).

The negative mood and fear avoidance domains assess pain vulnerability, and the positive affect–coping domain documents resilience. The negative mood domain is derived from items on the following three scales: the Patient Health Questionnaire–9 (PHQ–9), which assesses the degree of depressive symptoms;²⁸ the State–Trait Anxiety Inventory (STAI), which assesses the degree of dispositional anxiety;²⁹ and the State–Trait Anger Expression Inventory (STAXI), which assesses the degree of dispositional anger symptoms.³⁰ The fear avoidance domain is derived from the FABQ Physical Activity (FABQ–PA) and Work (FABQ–W) sub-scales, which examine the degree of fear avoidance in relation to activity and occupation;³¹ the Pain Catastrophizing Scale (PCS), which assesses the degree of exaggerated negative orientation toward pain experiences;³² the Tampa Scale of Kinesiophobia–11 (TSK–11), which assesses the degree of fear of movement and injury;³³ and, finally, the Pain Anxiety Symptoms Scale–20 (PASS–20), which assesses the degree of pain-related anxiety symptoms.³⁴

The positive affect–coping domain is derived from three scales: the Pain Self-Efficacy Questionnaire (PSEQ), which assesses the degree of self-efficacy beliefs in the context of pain; the Self-Efficacy for Rehabilitation Outcome Scale, which assesses the degree of self-efficacy associated with performing various tasks during rehabilitation; and the Chronic Pain Acceptance Questionnaire (CPAQ), which examines the degree of pain acceptance from a functional perspective by focusing on the behavioral aspects of coping with pain.

Continuous data

Scores on the full-length (17-item) and the 10-item questionnaires are estimates of the full-length parent questionnaires (e.g., TSK–11, PSEQ, CPAQ) and represent each of the 11 constructs. The total score for each construct is a continuous metric and is calculated by summing all the item responses, which are then multiplied by their associated linear regression weights with the reverse score for pain resilience items. The scores of the domains are calculated by adding the scores of the relevant constructs.¹⁶ The summary score that represents the total score is a simple sum of the OSPRO–YF items without weighting. These estimates are interpreted clinically in the same way that a full-length parent questionnaire score might be interpreted.

Dichotomous data

To further improve the clinical interpretability of the continuous scores, the developers of the OSPRO–YF have provided formulas that use logistic regression weights for the cutoff values to define the presence of elevated vulnerability or reduced resilience.¹⁶ The yellow flag indicators are binary measures (yes–no) that indicate whether a respondent’s parent questionnaire score estimate (e.g., TSK–11, PSEQ, CPAQ) is high or low enough to signal a yellow flag; a yellow flag is defined as being in the top population quartile for negative constructs or in the bottom population quartile for positive constructs.

Calculating continuous data or cutoff scores is not feasible in a clinic because of the complicated scoring algorithms and complex regression analyses that require the use of a computer; however, an online calculator is provided by the Orthopaedic Physical Therapy Investigator’s Network (OSPRO Yellow Flag Assessment Tool Scoring Portal), supported by the American Physical Therapy Association. To use this calculator, the responses to the 10 questions are entered, and the estimated score on the parent questionnaire and presence of yellow flags for the relevant area are provided.

Outcome measures

To validate the 10-item OSPRO–YF screening tool, we used three questionnaires or scales that measured similar constructs (pain, disability, depression, and anxiety): the ÖMPSQ–10,¹⁵ the Quick Disabilities of the Arm, Shoulder and Hand (QuickDASH),³⁵ and the Hospital Anxiety and Depression Scale (HADS). The ÖMPSQ–10¹⁵ assesses the level of chronic pain, work, and functional disability on a scale ranging from 0 (absence of impairment) to 10 (severe impairment). It has shown validity in patients with musculoskeletal pain.^36–38 The QuickDASH examines disability related to the affected upper extremity. It has 11 questions, uses a Likert scale, and has established validity and reliability in patients with shoulder complaints.^{35, 39} The HADS examines the extent of mental well-being in relation to anxiety and depression.¹³ The total score for both the HADS Anxiety and the HADS Depression sub-scales ranges from 0 to 21. A score of 11 or more indicates clinically significant anxiety or depression symptoms. The HADS has acceptable measurement properties in patients with musculoskeletal conditions.⁴⁰

The questionnaires and an information sheet about the study were mailed to candidates before they visited the clinic. Information on demographics, clinical examination, and diagnosis was collected by the clinicians (HR, VP).

Determining sample size

For the test–retest reliability component using κ values, a plausible number of subjects required to detect a statistically significant κ (p < 0.05) on a dichotomous variable (presence or absence of a trait), with 80% power and proportions of positive rating (range 0.30–0.90) and the null hypothesis of 0.00, was estimated at 32.⁴¹ As with hypothesis testing, we evaluated the lower limit of the CI against a clinically meaningful minimum magnitude, such as 0.40, that indicates a moderate agreement rather than agreement against a zero value. To calculate the internal consistency coefficient of a scale with three to five points, a minimum sample size of 50 is considered sufficient.⁴²

The sample size for the factor analysis was based on the prevalent guideline of a subject:item ratio of 10:1.⁴³ For convergent validity, assuming a Type I error rate of 0.05, a power of 0.80, and an estimated association of r = 0.40, we would require about 50 subjects.⁴⁴ The sample size of the known-groups validity component was based on the case–control design. Because the original studies used an observational study design, and we did not have any information on the odds ratios (ORs) of the OSPRO–YF items, we used information on a similar indicator of psychological distress, the HADS. Previous research with injured workers with shoulder pain demonstrated that patients with more than three psychosocial signs (67%) had a higher anxiety score than patients with no flag signs (12%) and ORs varying from 3 to 8.⁴⁵ If we assumed an OR of 3, a power of 0.80, and an a of 0.05, a minimum sample size of 76 study patients and 154 controls with complete data would be necessary to detect a statistically significant difference between the two groups.⁴⁴ Therefore, we considered a sample size of 80 study patients and 160 controls (N = 240) sufficient for all components of the study.

Statistical analysis

Test–retest reliability

We examined the test–retest reliability of the continuous data of the domain scores using intra-class correlation coefficient (ICC) statistics. κ statistics were used to examine the test–retest reliability of the dichotomous constructs (specific cutoff values that indicate the presence of a psychosocial trait). The upper limit of κ is 1.00, and it occurs when there is perfect agreement between examiners.⁴⁶ Strength of agreement was based on the lower bound of the CI and was interpreted as follows:⁴⁷ less than 00.0 = poor; 0.00–0.20 = slight; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial; 0.81–1.00 = almost perfect. It is important to note that κ can be low despite a relatively high value of percentage of agreement, most likely because of imbalanced cell numbers, which occurs when the number of positives is significantly higher.

Internal consistency

The OSPRO–YF measures three domains of psychological well-being, and some items have a reverse relationship with one another. Each domain documents a distinct psychological area (negative affect, fear avoidance, and positive affect–coping). The internal consistency reliability of the 11 constructs in each domain was examined using Cronbach’s a coefficients. We hypothesized a high a coefficient in each domain. An inter-item correlation of 0.70 was considered necessary for an acceptable a coefficient.⁴⁸

Factor analysis

Factorial validity examined the degree to which covariance among the 11 psychological constructs resembled the covariation of behaviors underlying the domains in a sample that was limited to shoulder conditions. Initially, we used an oblique rotation method that allows constructs to be correlated; this method is commonly applied in psychometric research. We then explored the varimax solution, which assumes the orthogonality (i.e., independence) of the domains on the basis of the instrument’s conceptual model.

Cross-sectional convergent validity of the constructs

We examined the convergent validity of the 11 psychological constructs and domains against the HADS, QuickDASH, and ÖMPSQ–10 scales. Convergent validity was estimated with Pearson’s correlation. Using Cohen’s interpretation of strength of association, or effect sizes (small, d = 0.2; medium, d = 0.5; and large, d = 0.8),⁴⁹ and on the basis of the nature of the external scales, which measure different aspects of psychological well-being, we hypothesized a medium association (d = 0.50) between the HADS Depression sub-scale score and the PHQ–9 construct, which assesses depression, and among the HADS Anxiety sub-scale, STAI, and PASS–20 constructs, which examine different aspects of anxiety.

Fear avoidance beliefs play an important role in the expression of higher pain and dysfunction and difficulty with a successful return to work among individuals with musculoskeletal injuries.^50–53 We hypothesized a medium association (d = 0.50) between the QuickDASH, which measures upper extremity disability, and the FABQ–PA and FABQ–W constructs, which assess different aspects of fear avoidance. The associations among the ÖMPSQ–10, which documents the level of pain, work, and functional disability, and the fear avoidance sub-scales and positive affect constructs were expected to be moderate to high. Considering that each domain of the OSPRO–YF is composed of similar constructs, we hypothesized a stronger association between each domain and the external measures of depression, anxiety, disability, and pain, compared with individual constructs. Finally, we expected low associations between the summary score and the external outcomes because this score represents a compilation of different domains and constructs.

Known-groups validity

To assess known-groups validity, we examined the ability of the dichotomous data of OSPRO–YF to differentiate between cases and controls using χ² analysis (i.e., presence of yellow flags: yes or no). As noted, on the basis of literature that highlights a less optimal recovery after a compensable work injury,^20–26 we hypothesized that patients with an active compensation claim would have a higher number of yellow flags. The ORs were provided as indicators of the strength of the relationship between having a compensable injury and the presence of a certain psychological factor.

Results

A total of 80 patients with an active compensation claim and 160 without a claim participated in the validity component of this study (mean age of 51 [SD 12] y; range, min–max = 20–73); for the test–retest reliability component of the study, 35 patients completed the OSPRO–YF questionnaire on two occasions approximately 2 weeks apart. Table 1 shows the patients’ characteristics according to the presence of the compensation claim factor. Patients in the compensation group had a shorter duration of symptoms because their assessment was expedited by the WSIB. They had a higher prevalence of traumatic injuries and a higher number of full-thickness rotator cuff tears. The compensation group had a greater number of physically demanding jobs and an inability to work at the time of their initial visit.

Table 1 .

Patients’ Characteristics on the Basis of Presence of Work-Related Compensation Claim

Characteristic	No. (%) of participants*		Statistics; p-value
	Compensation claim; n = 80	No compensation claim; n = 160
Age, y, mean (SD)	49 (12)	52 (12)	t1, 239 = 1.71; 0.09
Sex			χ21 = 0.71; 0.39
Female	28 (35)	65 (41)
Male	52 (65)	95 (59)
Symptom duration, mo, mean (SD)	15 (27)	45 (56)	t1, 239 = 5.55; < 0.001
Affected side			χ21 = 0.70; 0.40
Right	50 (63)	91 (57)
Left	30 (38)	69 (43)
Mechanism of injury			FET ≤ 0.0001; < 0.001
Insidious	6 (8)	54 (34)
Repetitive activities	11 (14)	44 (28)
Fall	26 (33)	25 (16)
Traumatic	30 (38)	34 (21)
Traction injury	6 (8)	1 (1)
Other	1 (1)	2 (1)
Type of pathology†
Impingement	31 (39)	67 (42)	χ21 = 0.22; 0.64
PTRCT	10 (13)	33 (21)	χ21 = 2.39; 0.12
FTRCT	19 (24)	49 (31)	χ21 = 3.58; 0.006
Large or massive	13 (68)	21 (43)
Small or moderate	6 (32)	28 (57)
Superior labral pathology	7 (9)	5 (3)	FET = 0.05; 0.11
Instability	5 (6)	13 (8)	FET = 0.19; 0.08
OA GHJ	1 (1)	35 (22)	FET = 0.20; 0.19
Mild	0	15 (43)
Moderate	1 (100)	6 (17)
Severe	0	14 (40)
Employment			FET < 0.0001; < 0.001
Full-time, regular or modified	42 (53)	118 (74)
Part-time, regular or modified	9 (11)	5 (3)
Unable to work	27 (34)	18 (11)
Retired (not included)	2 (3)	19 (12)
Job demand			χ22 = 37.65; < 0.001
Light	5 (6)	64 (41)
Moderate	28 (35)	53 (34)
Heavy	45 (56)	22 (14)
Retired (not included)	2 (3)	19 (12)

Note: Percentages may not total 100 because of rounding.

^*Unless otherwise indicated.

^†Some patients had more than one pathology.

FET = Fisher exact test; PTRCT = partial-thickness rotator cuff tear; FTRCT = full-thickness rotator cuff tear; OA = osteoarthritis; GHJ = glenohumeral joint.

Test–retest reliability

The ICC values for two observations of a random sample of 35 patients were 0.91 (95% CI: 0.84, 0.96), 0.94 (95% CI: 0.87, 0.98), and 0.93 (95% CI: 0.86, 0.97) for the negative mood, fear avoidance, and positive affect–coping domains, respectively. The lower bound of the κ values for the test–retest reliability of the dichotomous OSPRO–YF constructs varied from 0.05 to 0.65 for the negative mood domain, from –0.12 to 1.00 for the fear avoidance domain, and from 0.56 to 0.70 for the positive affect domain (see Table 2). The lowest k was related to PASS–20, or pain-related anxiety symptoms, which included a 0.00 in the lower bound range of the CI. This low value was related to imbalanced cells with agreement on 32 positive responses and 1 negative response and only two disagreements.

Table 2 .

Test–Retest Reliability of Psychological Constructs (Dichotomous Data)

OSPRO–YF constructs	k (95% CI)	Strength of agreement*
PHQ–9	0.30 (0.05, 0.54)	Slight
STAI	0.51 (0.28, 0.75)	Fair
STAXI	0.83 (0.65, 1.00)	Substantial
FABQ–PA	0.76 (0.55, 0.98)	Moderate
FABQ–W	0.71 (0.45, 0.94)	Moderate
TSK–11	0.89 (0.73, 1.00)	Substantial
PCS	1.00 (1.00, 1.00)	Perfect
PASS–20	0.48 (–0.12, 1.00)	Poor
PSEQ	0.77 (0.56, 0.98)	Moderate
SER	0.87 (0.70, 1.00)	Substantial
CPAQ	0.82 (0.64, 1.00)	Substantial

^*Based on lower bound of k values: < 00.0 = poor; 0.00–0.20 = slight; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial; 0.81–1.00 = almost perfect.

OSPRO–YF = Optimal Screening for Prediction of Referral and Outcome Yellow Flag; PHQ–9 = Patient Health Questionnaire–9; STAI = State–Trait Anxiety Inventory; STAXI = State–Trait Anger Expression Inventory; FABQ–PA = Fear-Avoidance Beliefs Questionnaire Physical Activity sub-scale; FABQ–W = Fear-Avoidance Beliefs Questionnaire Work sub-scale; TSK–11 = Tampa Scale of Kinesiophobia–11; PCS = Pain Catastrophizing Scale; PASS–20 = Pain Anxiety Symptoms Scale–20; PSEQ = Pain Self-Efficacy Questionnaire; SER = Self-Efficacy for Rehabilitation; CPAQ = Chronic Pain Acceptance Questionnaire.

Internal consistency

The standardized Cronbach’s a coefficients for the negative mood, fear avoidance, and positive affect–coping domains were 0.88, 0.94 and 0.94, respectively, meeting the criteria for acceptable a coefficients.

Factor analysis

The Promax oblique method, which assumes correlations among items, produced three domains in which two constructs (PHQ–9 and CPAQ) were eliminated as a result of low loading values ( < 0.5). Similarly, the varimax method revealed three distinct domains representing three sets of independent latent variables, with the CPAQ construct showing cross-loading on the two domains of fear avoidance and positive affect–coping (see Table 3).

Table 3 .

Factor Procedure Rotation Method: Varimax

Scale	Rotated factor pattern*
	Factor 1	Factor 2	Factor 3
PHQ–9	–	–	0.67446
STAI	–	–	0.75936
STAXI	–	–	0.89221
FABQ–PA	0.78976	–	–
FABQ–W	0.79713	–	–
TSK–11	0.84327	–	–
PCS	0.70405	–	–
PASS–20	0.78243	–	–
PSEQ	–0.61927	0.71530	–
SER	–	0.85608	–
CPAQ	–0.62614	0.65815	–

Note: Dashed indicate values < 0.5.

^*Variance explained by each factor: Factor 1 = 4.1818554; Factor 2 = 2.6656952; Factor 3 = 2.5595407.

PHQ–9 = Patient Health Questionnaire–9; STAI = State–Trait Anxiety Inventory; STAXI = State–Trait Anger Expression Inventory; FABQ–PA = Fear-Avoidance Beliefs Questionnaire Physical Activity sub-scale; FABQ–W = Fear-Avoidance Beliefs Questionnaire Work sub-scale; TSK–11 = Tampa Scale of Kinesiophobia–11; PCS = Pain Catastrophizing Scale; PASS–20 = Pain Anxiety Symptoms Scale–20; PSEQ = Pain Self-Efficacy Questionnaire; SER = Self-Efficacy for Rehabilitation; CPAQ = Chronic Pain Acceptance Questionnaire.

Construct convergent validity

Table 4 shows the correlations between different constructs of the OSPRO–YF and the related scales. As hypothesized, these associations were mostly moderate to high and in the expected direction. The associations with the external scales improved for the domains as expected because of the higher number of similar constructs in each domain. The summary score for the OSPRO–YF, however, had lower associations with the external scales compared with the individual constructs and domains: 0.42 (95% CI: 0.32, 0.53), 0.44 (95% CI: 0.35, 0.55), 0.37 (95% CI: 0.20, 0.43), and 0.50 (95% CI: 0.36, 0.56) for depression, anxiety, QuickDASH, and ÖMPSQ–10, respectively.

Table 4 .

Cross-Sectional Associations between the OSPRO–YF and Relevant Constructs

OSPRO–YF item*	r (95% CI)
	HADS (D)	HADS (A)	QuickDASH	ÖMPSQ–10
Negative mood domain
PHQ–9	0.60 (0.52, 0.68)	–	–	–
STAI	–	0.68 (0.60, 0.74)	–	–
STAXI	–	–	–	–
Full domain	0.66 (0.59, 0.73)	0.68 (0.60, 0.74)	0.50 (0.40, 0.59)	0.65 (0.57, 0.72)
Fear avoidance domain
FABQ–PA	–	–	0.60 (0.51, 0.67)	0.68 (0.61, 0.75)
FABQ–W	–	–	0.63 (0.55, 0.70)	0.76 (0.70, 0.80)
TSK–11	–	–	–	0.68 (0.61, 0.74)
PCS	–	–	–	0.69 (0.61, 0.75)
PASS–20	–	0.68 (0.61, 0.75)	–	0.75 (0.68, 0.80)
Full domain	0.68 (0.61, 0.74)	0.69 (0.62, 0.75)	0.65 (0.57, 0.71)	0.79 (0.73, 0.83)
Positive affect–coping domain
PSEQ	–	–	–	–0.77 (–0.82, –0.71)
SER	–	–	–	–0.61 (–0.63, –0.45)
CPAQ	–	–	–	–0.75 (–0.73, –0.59)
Full domain	–0.65 (–0.71, –0.57)	–0.64 (–0.71, –0.56)	–0.64 (–0.71, –0.55)	–0.73 (–0.78, –0.67)

Note: Dashes indicate no correlation statistic was calculated for those factors (they were not considered relevant to one another).

^*The negative mood and fear avoidance domains have a positive cutoff and the positive affect domain has a negative cutoff.

OSPRO–YF = Optimal Screening for Prediction of Referral and Outcome Yellow Flag; HADS (D) = Hospital Anxiety and Depression Scale, Depression; HADS (A) = Hospital Anxiety and Depression Scale, Anxiety; QuickDASH = Quick Disabilities of the Arm, Shoulder and Hand; ÖMPSQ–10 = Örebro Musculoskeletal Pain Screening Questionnaire–10; PHQ–9 = Patient Health Questionnaire–9; STAI = State–Trait Anxiety Inventory; STAXI = State–Trait Anger Expression Inventory; FABQ–PA = Fear-Avoidance Beliefs Questionnaire Physical Activity sub-scale; FABQ–W = Fear-Avoidance Beliefs Questionnaire Work sub-scale; TSK–11 = Tampa Scale of Kinesiophobia–11; PCS = Pain Catastrophizing Scale; PASS–20 = Pain Anxiety Symptoms Scale–20; PSEQ = Pain Self-Efficacy Questionnaire; SER = Self-Efficacy for Rehabilitation; CPAQ = Chronic Pain Acceptance Questionnaire.

Cross-sectional known-groups validity

Ten dichotomous constructs of the OSPRO–YF were able to differentiate between patients with and without an active compensation claim (p-values ranging from 0.028 to < 0.0001). The PASS–20 item (see Table 5) was not able to differentiate between the two groups (p = 0.84; 95% CI: 0.29–2.76). The PASS–20 measures fear and anxiety responses specific to pain, and it appears that regardless of the type of injury (e.g., work related), patients with shoulder conditions expressed a similar level of fear and anxiety in relation to moving the shoulder beyond a pain-free range of motion. The ORs in Table 5 show the strength of the association between the compensable injury and presence of a certain yellow flag. For example, the OR of 5.26 for FABQ–W means that the odds of the compensation group having a positive yellow flag in relation to working fear avoidance is 5.26 times greater than the odds for the non-compensation group.

Table 5 .

Known-Groups Analysis Based on Having a Work-Related Compensation Claim

OSPRO–YF item*	No. (%) of participants		χ2; p-value	OR (95% CI)
	Compensation claim; n  = 80	No compensation claim; n  = 160
Negative mood domain
PHQ–9	42 (53)	45 (28)	χ21 = 13.71; < 0.001	2.82 (1.62, 4.93)
STAI	51 (64)	72 (45)	χ21 = 7.50; 0.006	2.15 (1.24, 3.73)
STAXI	51 (64)	78 (49)	χ21 = 4.83; 0.028	1.85 (1.07, 3.21)
Fear avoidance domain
FABQ–PA	54 (68)	59 (37)	χ21 = 20.07; < 0.001	3.56 (2.02, 6.27)
FABQ–W	68 (85)	83 (52)	χ21 = 25.08; < 0.001	5.26 (2.64, 10.46)
TSK–11	62 (78)	82 (51)	χ21 = 15.31; < 0.001	3.28 (1.78, 6.03)
PCS	57 (71)	76 (48)	χ21 = 12.18; < 0.001	2.74 (1.54, 4.87)
PASS–20	75 (94)	151 (94)	χ21 = 0.04; 0.85	0.89 (0.29, 2.76)
Positive affect domain
PSEQ	14 (18)	86 (54)	χ21 = 28.83; < 0.001	5.48 (2.85, 10.55)
SER	18 (23)	67 (42)	χ21 = 8.75; 0.003	2.48 (1.34, 4.57)
CPAQ	16 (20)	83 (52)	χ21 = 22.36; < 0.001	4.31 (2.30, 8.09)

^*The negative mood and fear avoidance domains have a positive cutoff and the positive affect domain has a negative cutoff.

OSPRO–YF = Optimal Screening for Prediction of Referral and Outcome Yellow Flag; PHQ–9 = Patient Health Questionnaire–9; STAI = State–Trait Anxiety Inventory; STAXI = State–Trait Anger Expression Inventory; FABQ–PA = Fear-Avoidance Beliefs Questionnaire Physical Activity sub-scale; FABQ–W = Fear-Avoidance Beliefs Questionnaire Work sub-scale; TSK–11 = Tampa Scale of Kinesiophobia–11; PCS = Pain Catastrophizing Scale; PASS–20 = Pain Anxiety Symptoms Scale–20; PSEQ = Pain Self-Efficacy Questionnaire; SER = Self-Efficacy for Rehabilitation; CPAQ = Chronic Pain Acceptance Questionnaire.

Discussion

Tools that screen for psychological factors referred to as yellow flags assist a clinician in delivering appropriate care. Yellow flags are designed to indicate when psychological distress might be high enough to warrant a more in-depth evaluation of psychological distress, modifications to treatment, or both. The results of these questionnaires are used to set goals and provide patient education and to help determine the options for psychologically oriented treatments for non-musculoskeletal symptoms. Although these measures are generally recognized as important to the assessment and treatment of patients with musculoskeletal disorders, a barrier to any routine screening of psychosocial status is the large number of tools available. Our study supports use of the 10-item OSPRO–YF questionnaire as a screening tool with injured workers with shoulder problems. It identifies patients with higher vulnerability to pain, such as negative affect and fear avoidance, and less resilience to pain, such as positive affect, coping, and self-efficacy, in a single-tool format, facilitating work in busy clinics by reducing the patient or clinician response burden.

The factorial validity findings in this study supported the developers’ conceptual framework of the OSPRO–YF questionnaire in separating the three domains that represented three distinct psychological areas. When the factors are truly uncorrelated, orthogonal and oblique rotations produce nearly identical results.⁴³ In this study, both methods produced three domains. However, the CPAQ, which measures acceptance of chronic pain, had low loading in the oblique rotation and cross-loading in the orthogonal rotation between the fear avoidance and positive affect–coping domains. This finding may indicate that this item measures several concepts in patients with shoulder conditions. Overall, this is not a desirable end product of the analysis because we are looking for distinct factors; thus, assessment of this construct needs further evaluation.

The associations between the continuous OSPRO–YF constructs and the relevant scales were moderate to high in the expected direction. The slightly lower correlation between the STAI and the HADS Anxiety sub-scale may be related to the difference in measuring current and longstanding anxiety by the construct derived from the original STAI against more generalized symptoms as documented by the HADS Anxiety sub-scale.⁵⁴

In this study, the summary score for the OSPRO–YF had lower associations with the external scales used for concurrent validity than did the individual constructs and domains. In general, using a single summated score for a multidimensional tool that measures different domains of psychosocial well-being is problematic because a total score is intended to represent a common theme rather than a compilation of different concepts. It is believed that by merging the scores for negative mood, fear avoidance, and positive affect–coping, the significance of each domain may disappear in the total score because they measure different concepts. For example, treating depression is different than managing low pain self-efficacy, and the summated score will obscure the need to manage each problem.

Most important, when the purpose of a measure is to assess change over time, the items contributing to the total score should display similar change trajectories, which may be a problem when different domains are expected to recover differently or one may improve while the others deteriorate. Adding up the scores in distinct physical and mental domains to obtain one total single score became a common practice with multidimensional shoulder surveys that were developed in late 1990s and early 2000s;^55–58 however, there is no scientific justification or statistical basis for a single number being a meaningful representation of a patient’s status when multiple domains are concerned.^{59, 60}

In terms of known-groups validity, 10 of the 11 constructs (dichotomous data) were able to differentiate between the WSIB and non-WSIB patients; patients with a compensable shoulder injury reported higher levels of negative mood and fear avoidance and poorer coping abilities. Previous research has shown that injured workers have less successful results after shoulder surgery than the general population.^{25, 61–63} In this study, the compensation group had a shorter wait time, an important factor that has been proven to affect response to treatment.⁶⁴ However, the injured workers had more physically demanding jobs, and a larger number of them were unable to work full time. These factors are expected to play an important role in the development of disability and psychosocial factors. One important psychological factor that is reported to affect recovery in injured workers is the fear of re-injury after returning to heavy labor,^{65, 66} a factor that the OSPRO–YF takes into consideration as the fear avoidance domain.

In summary, scores for the constructs and domains proved to be reliable and valid for research purposes in our study, which used patients with shoulder complaints seen at two tertiary clinics. We also showed that the majority of the categorical yellow flags of the OSPRO–YF were reliable (the results were consistent over a stable time period) and valid (the questionnaire differentiated among different levels of psychological distress). If clinicians can identify patients with positive psychological flags and psychological vulnerabilities early, they can more quickly recommend a bio-psychosocial approach to care with appropriate targeted interventions. If clinicians can integrate knowledge of psychological factors into their plan of care, they may be able to prevent unnecessary additional costly diagnostic imaging and misguided treatments.

This study had several limitations. First, we examined the psychometric properties of only the 10-item OSPRO–YF; a further assessment of the 7- and 17-item versions would help clinicians choose the most optimal measure. Second, an inherent weakness of case–control studies is the difficulty of ensuring that cases and controls are a representative sample of the same source population. In this study, to reduce this problem, cases and controls were taken from the same academic centre in the same time frame. Nevertheless, the generalizability of our results may be limited to patients seen in a tertiary care centre who may have a higher prevalence of significant pathology or psychosocial concerns. Finally, construct validity is established with incremental evidence in different populations of patients. Therefore, further research in primary care settings is recommended. The role of OSPRO–YF in predicting referral pattern and outcome will need to be examined in longitudinal studies.

Conclusion

The 10-item OSPRO–YF reduces the burden of using multiple questionnaires and has acceptable test–retest and internal consistency reliability as well as factorial, convergent, and known-groups validity.

Key Messages

What is already known on this topic

Studies of the accuracy of the Optimal Screening for Prediction of Referral and Outcome Yellow Flag (OSPRO–YF) screening tool have been mostly conducted by its developers; we are not aware of any independent studies of its reliability or validity. The available literature shows promising results for this tool; however, construct validity is not simply a property of a measure, but rather of a measure in context, and it can be established only incrementally as evidence increases when the measure is related theoretically to other relevant measures and phenomena. Therefore, further evaluation of this screening tool in an independent sample of patients is warranted.

What this study adds

This study adds to the body of literature in the area of bio-psychosocial research. The 10-item OSPRO–YF is a short questionnaire that eliminates the need to use multiple scales. Our results confirm that it accurately identifies patients with higher levels of negative affect and fear avoidance and lower levels of pain resilience. Using a case–control design allowed us to more efficiently examine the risk factor of an occupational injury. In addition, using an independent sample added to the strength of the previous studies conducted by the OSPRO–YF’s developers.