Satisfaction after total knee replacement for osteoarthritis is usually high, but what are we measuring? A systematic review

Summary

Objective

Patient satisfaction is considered an important outcome measure after total knee replacement, but the construct is complex. There is large variation both in how satisfaction is measured and estimates of the proportion of people who are satisfied after surgery. The aim of this systematic review was to i) evaluate the proportion of people reported to be satisfied after total knee replacement for osteoarthritis; and ii) assess the content validity of the utilised satisfaction measures.

Methods

We searched four literature databases with search phrases ‘Total Knee Arthroplasty’ OR ‘Total Knee Replacement’ AND ‘Patient satisfaction’ for studies that measured satisfaction at least 6 month post-unilateral primary total knee replacement for knee osteoarthritis. Identified studies were assessed for risk of bias, and studies at high risk of bias were excluded (PROSPERO: CRD42017058936). Meta-analysis was not appropriate due to the heterogeneity in satisfaction instruments, thus satisfaction scores were described. The content validity of satisfaction questionnaires was assessed using the COnsensus-based Standards for the selection of health status Measurement Instruments criteria.

Results

The present review found heterogeneity in the satisfaction questions used, as well as the satisfaction estimates from the various studies. Only two satisfaction instruments were relevant for a Total Knee Replacement population and both failed assessment for content validity due to lack of patient involvement during development and testing in accordance with the COnsensus-based Standards for the selection of health status Measurement Instruments criteria.

Conclusion

Future research should focus on qualitative methods to elicit patients' perspectives of satisfaction to build theoretical understanding.

1. Introduction

Total knee replacement (TKR) surgery is considered the gold standard treatment for end stage knee osteoarthritis (OA) due to its cost effectiveness [1] and high rates of symptomatic and functional improvement [2]. However, despite near-flawless surgical procedures, up to 30% of people fail to have clinically meaningful improvements in pain and disability levels post-operatively [3]. These rates of poor response highlight the importance of appropriately determining and measuring success with this procedure to facilitate improvement in outcomes.

The lack of concordance between the surgeon's and patient's appraisals of the intervention [4,5] underscores the importance of understanding the success of a TKR from the patient's perspective. As such, the Osteoarthritis Research Society International (OARSI) has identified cut points of patient-reported changes in pain and function as valid and reliable markers of response to TKR [6]. In addition to this, patient satisfaction is considered an important outcome measure post TKR, as endorsed by a patient and surgeon derived Delphi study conducted by the Outcome Measures in Rheumatology [7].

Despite the importance of measuring patient satisfaction as a reflection of the value of the orthopaedic intervention, the satisfaction instruments and quantification methods used after TKR are highly heterogeneous [8]. A previous systematic review [8] investigated the available literature on satisfaction after TKR and found only 13% of the included studies used a satisfaction instrument which had demonstrated some form of validity. Furthermore, 21.2% did not define how they measured satisfaction, and the remaining 65.8% drew on a variety of questions and quantification methods to measure this construct [8]. These observations may explain why satisfaction estimates have been reported to vary extensively, from as high as 99% [9], to findings as low as 70% [10,11]. The reasons for such heterogeneity have not been rigorously investigated, however a recent study [12] indicates the importance of how the satisfaction questions are framed. The authors found the focus of the satisfaction question (such as general satisfaction as compared to satisfaction with recreational activities) significantly affected the rates of satisfaction by as much as 10% [13].

These findings highlight the importance of understanding the different aspects of satisfaction. According to satisfaction theory, satisfaction is multifactorial and includes numerous variables that are likely to contribute to a patient's appraisal [14]. When considering the complexity of satisfaction theory in combination with the heterogeneity and lack of validation of the commonly used satisfaction instruments, it is not possible for researchers and clinicians to have an understanding of what is actually being captured by the various instruments.

To create certainty around what is being measured by patient reported outcome measures (PROMs), including those assessing satisfaction, confirmation of content validity is essential [15]. Content validity is the degree to which the content of a PROM is an adequate reflection of the construct to be measured, and is considered the most important measurement property of a PROM [15]. Content validity comprises three key aspects: content relevance (all items should be relevant for the construct of interest), content comprehensiveness (no key aspects of the construct should be missing), and content comprehensibility (the items should be understood by patients as intended) [15]. To achieve these three key aspects of content validity, the involvement of the patient in PROM development is essential. This includes patient involvement in theory development, item development, and item testing in terms of understanding of content and response categories.

To facilitate a better and more consistent understanding of patient satisfaction, the aims of this review were therefore to i) evaluate the proportion of people reported to be satisfied after TKR for osteoarthritis; and ii) assess the content validity of the utilised satisfaction measures.

2. Methods

The review protocol was prospectively registered on PROSPERO (CRD42017058936) and reported according to PRISMA guidelines [16]. Assessment of content validity of measures was additional to this protocol as the need for this aspect became apparent during the review process.

2.1. Literature search

We developed an electronic search strategy (See Appendix 1) of all available data from inception until September 2018 to identify eligible studies in the MEDLINE, EMBASE, CINAHL databases and the Cochrane Database of Registered Trials. We searched the databases using the following terms: ‘Total Knee Arthroplasty’ OR ‘Total Knee Replacement’ AND ‘Patient satisfaction’ and imported retrieved titles and abstracts into the Endnote software (Clarivate Analytics, Philadelphia, PA, USA) and removed duplicates.

2.2. Study selection

The inclusion criteria (Table 1) were devised by the research team with clinical and research expertise in TKR (AS, MD, PC); and systematic reviews (AS, PK, RF, SB). Titles and abstracts were uploaded into Covidence (Covidence, Melbourne, Victoria, Australia) to facilitate the screening process. Two reviewers (SB, RF) independently screened titles and abstracts for inclusion. Where information was not explicitly presented in the title and abstract e.g. unilateral versus bilateral TKR, the full text article was retrieved for screening. Full text articles were independently screened by three authors (SB, RF, NK). Disagreements were resolved by consultation with the other authors until consensus was reached. Given the volume of papers requiring screening, if the information to meet inclusion was not reported in the full text article, the articles were excluded without contacting the study authors.

Table 1.

Inclusion criteria.

Criteria	Definition/justification
Unilateral, primary total knee replacement	We included studies in which participants underwent total knee replacement. We excluded studies in which participants underwent unicompartmental knee replacement as satisfaction levels may differ significantly between patients with unicompartmental and total knee replacement [17]. We included studies where <5% of participants underwent simultaneous bilateral TKR*. This is because satisfaction levels may be significantly different among people who receive a unilateral versus simultaneous bilateral TKR [18]. Studies involving participants undergoing their second primary TKR were included. Where it was unclear whether the bilateral TKRs were simultaneous or staged, it was assumed that they were simultaneous. We included studies where <5% of participants underwent revision TKR. This is because satisfaction levels may be significantly different among people who receive a primary versus revision TKR [19]. The 5% cut-off enabled us to include relevant studies where 95% of participants met our criteria. We anticipated that a 5% threshold would not significantly impact satisfaction outcomes reported in this review.
Total knee replacement for osteoarthritis of the knee	We included studies where <5% of participants underwent TKR for pathologies other than osteoarthritis. This is because the concerns and priorities of patients undergoing TKR differ according to their underlying diagnosis and the satisfaction levels may be significantly different between people undergoing TKR for osteoarthritis versus other pathology [20]. Accordingly, we excluded studies that did not explicitly state the reason for performing TKR.
Satisfaction measured ≥6 months post-operatively	We included studies that assessed satisfaction ≥6 months post TKR in order to capture satisfaction with outcome rather than process of care, and in light of evidence that 6 months would be a sufficient minimum time-frame in which to assess satisfaction given the majority of improvement in function after TKR takes place in the first 6 months post-surgery [21].
Satisfaction with total knee replacement outcome	We excluded studies that assessed satisfaction with the process of care, as this is a different construct to satisfaction with treatment outcome. We also excluded studies that did not include a measure of satisfaction with treatment outcome, but instead, inferred patient satisfaction from changes in knee pain or function following TKR [22].
Quantitative studies	We excluded any qualitative studies as our aim was to quantify satisfaction with TKR.
Original, full text articles	We excluded review papers and conference abstracts.
Articles written in English	Given the large scope of this review, for pragmatic reasons we excluded studies that were not written in English

2.3. Assessment of methodological quality

Two reviewers (SB and NK) independently assessed risk of bias using a purposely adapted tool based on an existing tool for assessing risk of bias in prevalence studies [17]. The existing tool was modified to accommodate the range of study designs included in this review such as prospective cohort studies, retrospective studies of registry data and randomised control trials. The adapted tool comprises ten domains; each domain was scored as low or high risk of bias (see Appendix 2). Of the ten assessment items, seven were ‘asterisked’, which indicated immediate exclusion of a study with failure of any of these items (see Appendix 2). Studies meeting all seven asterisked items were included in the review, with studies that failed any of the remaining three non-asterisked items considered to be moderate risk. The risk of bias tool was piloted using studies that did not meet the inclusion criteria to ensure familiarity and consistency of use. The two (SB and NK) reviewers resolved disagreements by consultation until arriving at a consensus decision.

2.4. Data extraction

Two reviewers (SB and NK) independently extracted data from each study using a standardised extraction sheet. Data extracted included characteristics of the study (geographical location, sample size); characteristics of the participants (age, percentage female); characteristics of the outcomes (satisfaction measure, duration of follow-up); and satisfaction outcome scores. Data extraction sheets from the two reviewers were compared for consistency and accuracy.

2.5. Data synthesis and analysis

2.5.1. Description of satisfaction after TKR

Given the multiple ways in which satisfaction was measured the authorship team took various steps to extract percentage satisfied from each study included in the review, which are detailed in Table 2.

Table 2.

Method of extracting percentage satisfied.

Study reporting method	Approach to extract percentage satisfied
Means and SD, or proportion values of categorical satisfaction scales	Percentage satisfied was derived from the sample size
Visual analogue scales reported	A satisfaction threshold was chosen based on the ‘smile face’ scale, where the point at which the face begins to smile was considered to be ‘satisfied’ (see Fig. 1); in a 1–10 scale, a score of 7 or more was chosen; in a 1–5 scale, 4 or more was chosen. The percentage of satisfied people was derived by calculating the number of people in the sample who had scores above the appropriate threshold for the data reported. This was achieved by converting the difference between the sample mean and the threshold into a z-score (the number of SD the threshold was away from the mean). The z-score was then converted to a percentile using the NORMDIST function in Excel v16.11 (Microsoft Corp, Redmond, WA, USA)
Only medians reported	An approximation of mean values was derived from the median range and sample size using the method of Hozo [18] to attain the percentage satisfied
Likert Scales	Outcomes of ‘satisfied’ or ‘very satisfied’ was regarded as a satisfied outcome
Knee Society Knee Score Satisfaction scale [19]. Total score of 40 from 5 items each with a maximum score of 8	A threshold of 28 was selected to indicate satisfaction. This represented a minimum of 4 satisfied answers and 1 neutral answer across the 5 satisfaction items.
The Self-Administered Patient Satisfaction Scale for Primary Hip and Knee Arthroplasty [20]. The items are scored on a 4-point Likert scale, with 4 response options: 25 pts (very dissatisfied), 50 pts (somewhat dissatisfied), 75 pts (somewhat satisfied), or 100 pts (very satisfied), which are averaged to give a total score.	A threshold of 68 was selected. The sum score of 68 represented 3 somewhat satisfied and 1 somewhat dissatisfied responses.
Multiple satisfaction questions under the one questionnaire	Where possible, these were individually reported as well as reporting a composite score
Papers reporting multiple follow-ups	The time point closest to twelve months was selected to be included in the review, based on evidence that this is when maximum improvement in pain and function is attained [21].
Papers only reporting satisfaction outcomes for subgroups of the sample	These subgroup scores were combined into one total group summary score.

Given the heterogeneity of the satisfaction instruments, it was not appropriate to meta-analyse the results [22]. Therefore, the satisfaction results of each paper are displayed in a forest plot with corresponding description (see Fig. 1).

Fig. 1.

0–10 Smile face satisfaction scale.

2.5.2. Assessment of content validity

Studies included in the review were assessed as to whether a citation was provided for the satisfaction instrument used. The citations were evaluated according to their support for content validity of the satisfaction measure, in terms of either a development study or secondary content validity study. In addition, a specific search strategy was developed to retrieve any studies of content validity for specific satisfaction instruments used by studies in this review, which was approved by the university librarian (see Appendix 3).

Content validity of satisfaction measures was evaluated using the COnsensus-based standards for the selection of health Status Measurement INstruments (COSMIN) content validity assessment checklist. The COSMIN methodology details that strong evidence of good content validity is achieved through adequate content relevance, comprehensiveness, and comprehensibility. For a PROM to be assessed on these three aspects, the COSMIN methodology has expanded them to create the ten criteria for good content validity, which includes five items under ‘relevance’, one item under ‘comprehensiveness’, and four items under ‘comprehensibility’ (see column 1 of Table 4a, Table 4ba and 4b). To assess whether these ten criteria have been met, the COSMIN methodology details a systematic three step process, whereby the final stage rates the PROM against the ten criteria. Single questions used by studies that were unsupported by the literature could not be assessed for evidence of content validity, in accordance with the COSMIN assessment. This process was conducted by two authors (AS and NK).

Table 4a.

Content validity assessment of the SAPSS.

	PROM development study 1	PROM development study 2	Content validity study	Rating of reviewers	Overall rating per PROM	Quality of evidence
Self-administered patient satisfaction scale	Development study [67]	Development study [20]	NA	+/−/?	+/−/±	High, moderate, low, very low
Relevance
1. Are the items relevant to the construct of interest?	–	–		+
2. Are the included items relevant for the target population of interest?	–	–		+
3. Are the included items relevant for the context of interest?	+	+		+
4. Are the response options appropriate?	–	–		+
5. Is the recall period appropriate?	–	–		?
Relevance rating	–	–	NA	+	±	Low
Comprehensiveness
6. Are all key concepts included?	–	–		–
Comprehensiveness rating	–	–	NA	–	–	Low
Comprehensibility
7. Are the PROM instructions understood by the population of interest as intended?	–	–
8. Are the PROM items and response options understood by the population of interest as intended?	–	–
9. Are the PROM items appropriately worded?				+
10. Do the response options match the question?				+
Comprehensibility rating		–	NA	+	±	Low
Content validity rating					–	Low

Legend: - Fail + Pass ? Insufficient information ± Inconsistent.

Table 4b.

Content validity assessment of the new KSKSS.

	PROM development study	Content validity study	Rating of reviewers	Overall rating per PROM	Quality of evidence
Satisfaction domain of the New Knee Society Knee Scoring System	Development Study [68]	NA	+/−/?	+/−/±	High, moderate, low, very low
Relevance
1. Are the items relevant to the construct of interest?	–		+
2. Are the included items relevant for the target population of interest?	–		+
3. Are the included items relevant for the context of interest?	+		+
4. Are the response options appropriate?	–		+
5. Is the recall period appropriate?	–		?
Relevance rating	–	NA	+	±	Low
Comprehensiveness
6. Are all key concepts included?	–		–
Comprehensiveness rating	–	NA	–	–	Low
Comprehensibility
7. Are the PROM instructions understood by the population of interest as intended?	–
8. Are the PROM items and response options understood by the population of interest as intended?	–
9. Are the PROM items appropriately worded?			+
10. Do the response options match the question?			+
Comprehensibility rating	–	NA	+	±	Low
Content validity rating				–	Low

Legend: - Fail + Pass ? Insufficient information ± Inconsistent.

The first phase of this process involved assessment of any development study of the satisfaction measure against steps 1a and 1b in the COSMIN assessment (see Appendix 4). Any further content validity studies in addition to development studies were assessed against steps 2a – 2e (see Appendix 5). Step 3 was a final appraisal of the ten criteria for good content validity. Step 3 involved appraising the development study, the content validity study (if available) as well as the reviewers’ opinion against the ten criteria for good content validity (see column 5 of Table 4a, Table 4ba and 4b).

3. Results

3.1. Literature search and risk of bias assessment

Our search strategy identified a total of 5824 records of which 2828 records were non-duplicates. After screening of titles and abstracts, 546 papers remained for full text screening. Following this, we excluded a further 346 articles leaving 152 articles for analysis of risk of bias. Forty-three articles passed the risk of bias assessment to be included in this systematic review (see Fig. 2), with 35 considered moderate risk due to failing one or more of the non-asterisked risk of bias items and eight considered low risk (see Appendix 6).

Fig. 2.

Study selection flow diagram.

3.2. Study characteristics

Individual study characteristics are presented in Table 3.

Table 3.

Study characteristics.

Study	Year	Country	Mean age (SD)	Design	Sample Size	Female (n)	Satisfaction question	Construct
Ali et al. [23]	2016	Sweden	68.5 (4)	RCT	74	21	Degree of satisfaction with the operated knee: ‘very satisfied’, ‘satisfied’, ‘uncertain’, ‘dissatisfied’	Satisfaction with the operated knee
Ali et al. [24]	2017	Sweden	72.9 (9.7)	RCT	186	16	Degree of satisfaction with the operated knee: ‘very satisfied’, ‘satisfied’, ‘uncertain’, ‘dissatisfied’	Satisfaction with the operated knee
Aunan and Rohl [25].	2018	Norway	69.3 (7.4)	Prospective cohort	129	73	Patient satisfaction measures on a VAS	Satisfaction
Baker et al. [26]	2007	England and Wales	70.8 (9.4)	Retrospective cohort	8231	4675	“Are you satisfied with your knee replacement?”: ‘yes’, ‘no’ and ‘not sure’.	Satisfaction with the operated knee
Blyth et al. [27]	2015	Scotland	65.5	Prospective cohort	198	116	Overall satisfaction: ‘very satisfied’, ‘satisfied’, ‘don't know’, ‘unsatisfied’, ‘very unsatisfied’	Satisfaction
Boese et al. [28]	2011	USA	64	Retrospective cohort	128	90	“How happy are you with your implanted knee?”: measured on a scale of 1–5 where 1 = completely dissatisfied to 5 = completely satisfied	Satisfaction with the operated knee
Clement et al. [12]	2018	United Kingdom	68.6 (9.3)	Retrospective cohort	1255	757	‘How satisfied are you with the results of your knee replacement surgery?’ ‘Very satisfied’, ‘somewhat satisfied’, ‘somewhat dissatisfied’, and ‘very dissatisfied’	Satisfaction with the operated knee
Collados-Maestre et al. [29]	2017	Spain	71.2 (6.4)	Prospective cohort	237	164	‘Patient satisfaction was evaluated yearly on a 5-point Liker scale’ ‘very satisfied’, ‘satisfied’, ‘neutral’, ‘dissatisfied’, and ‘very dissatisfied’	Satisfaction with the operated knee
Collins et al. [30]	2017	USA	69.5 (8.5)	Prospective cohort	633	375	‘How satisfied are you with the results of your knee replacement surgery?’ ‘very satisfied’ ‘somewhat satisfied’ ‘somewhat dissatisfied’ and ‘very dissatisfied’	Satisfaction with the operated knee
Culliton et al. [31]	2018	Canada	63.5 (8)	Prospective cohort	345	221	Patient Acceptable Symptom State	Satisfaction with symptoms
Dailiana et al. [32]	2015	Greece	69.2 (6.7)	Prospective cohort	204	162	Patient satisfaction with the results of TKR was assessed in three aspects: overall satisfaction, satisfaction with pain relief, and satisfaction with functional improvement/ability to perform daily activities. Patients were categorized as very/mostly satisfied, somewhat satisfied, and dissatisfied. (modified Self-Administered Patient satisfaction Scale)	Satisfaction Satisfaction with function Satisfaction with pain relief
Escobar et al. [33]	2013	Spain	71.4 (6.9)	Prospective cohort	912	641	Patient Acceptable Symptom State	Satisfaction with symptoms
Gaillard et al. [34]	2017	Germany	72.7	Retrospective cohort	1059	650	Not specified. ‘Very satisfied’, ‘satisfied’, ‘disappointed’	Satisfaction
Gandhi et al. [35]	2007	Canada	69.2 (8.8)	Prospective cohort	87	56	Are you satisfied with your limb alignment? ‘Yes’ or ‘No’	Aesthetics
Genet et al. [36]	2008	France	71.7 (7)	Prospective cohort	45	28	Patient satisfaction measured on a VAS (0–100)	Satisfaction
Gildone et al. [37]	2005	Italy	74.1 (4.8)	Prospective cohort	56	39	Satisfaction questionnaires. No response categories provided	Satisfaction
Giurea et al. [38]	2016	Austria	66 (NA)	Prospective cohort	86	48	Satisfaction with response categories: ‘Yes’ or ‘No’	Satisfaction with the operated knee
Healy et al. [9]	2002	USA	69.9 (8.7)	Prospective cohort	159	–	Patient satisfaction measured with response categories: ‘yes’ or ‘no’	Satisfaction with the operated knee
Hinarejos et al. [39]	2016	Spain	72.2 (7)	Prospective cohort	474	360	Satisfaction measured on a VAS (0 = absolutely dissatisfied, 10 = absolutely satisfied)	Satisfaction
Kawakami et al. [40]	2015	Japan	74.3 (7.8)	Prospective cohort	48	25	Satisfaction domain of the new Knee Society Knee Scoring System questionnaire	Composite
Khuangsiriku et al. [41]	2016	Thailand	76.9 (7.4)	Prospective cohort	144	130	The Self-Administered Patient Satisfaction Scale	Composite
Kim et al. [42]	2009	Korea	68.5 (5.6)	Prospective cohort	186	177	British Orthopaedic Association Patient Satisfaction Score	Satisfaction
Li et al. [43]	2012	China	67.2 (7.2)	Retrospective cohort	130	97	The British Orthopaedic Association Patient Satisfaction Score	Satisfaction
Liebs et al. [44]	2010	Germany	69.8 (7.9)	RCT	136	114	Total Hip Arthroplasty Outcome Evaluation Questionnaire	Satisfaction with the operated knee
Liebs et al. [45]	2012	Germany	69.8 (8.1)	RCT	158	133	Total Hip Arthroplasty Outcome Evaluation Questionnaire	Satisfaction with the operated knee
Lizaur-Utrilla et al. [46]	2016	Spain	69.7 (5.9)	Prospective cohort	192	127	Satisfaction measured with response categories: ‘very satisfied’, ‘satisfied’, ‘neutral’, ‘dissatisfied’, ‘very dissatisfied’	Satisfaction with the operated knee
Mannion et al. [47]	2009	Switzerland	67 (9)	Prospective cohort	112	7	Satisfaction with surgery measured with the response categories: ‘very satisfied’, ‘somewhat satisfied’, ‘somewhat dissatisfied’, ‘very dissatisfied’	Satisfaction with surgery
Matthews et al. [48]	2013	UK	69.2 (7.7)	Prospective cohort	34	20	Patient satisfaction measured on a 10-point VAS.	Satisfaction
Mooney et al. [49]	2016	Australia	68 (11.3)	Cross-sectional	67	43	Knee Society Score containing post-operative satisfaction scores	Satisfaction
Murphy et al. [50]	2014	Australia	70.8 (9.9)	RCT	40	25	Satisfaction with pain relief, physical function and overall outcome measured on a VAS (0 = completely unsatisfied to 10 = completely satisfied)	Satisfaction with function
Nilsdotter et al. [51]	2009	Sweden	72 (8)	Cross-sectional	87	50	Satisfaction with result in general measured on 5-point Likert scale from ‘totally satisfied’ to ‘very dissatisfied’. Questions about satisfaction in relation to pain relief; symptom relief; improvement in activities of daily living; and improvements in sport and recreational function. Dimensions measured on 5-point Likert scale from ‘totally satisfied’ to ‘very dissatisfied’.	Satisfaction Satisfaction with function (activities of daily living and sports and recreation) Satisfaction with symptoms
Petersen et al. [10]	2015	Denmark	65 (6.3)	Cross-sectional	215	139	Satisfaction with surgery measured with response categories: ‘very satisfied’, ‘satisfied’, ‘not completely satisfied’, ‘not satisfied’	Satisfaction with surgery
Pulavarti et al. [52]	2014	UK	69.9 (8.3)	RCT	126	68	Satisfaction measured with response categories: ‘excellent’, ‘good’, ‘fair’, ‘poor’	Satisfaction
Ranawat et al. [53]	2017	USA	71 (7.3)	Prospective cohort	193	138	Satisfaction measured on a VAS (0−10)	Satisfaction
Robertsson et al. [54]	2000	Sweden	–	Cross-sectional	–	–	Satisfaction with the operated knee measured with response categories: ‘very satisfied’, ‘satisfied’, ‘uncertain’, ‘dissatisfied’	Satisfaction with the operated knee
Stickles et al. [55]	2001	USA	69.9 (11.9)	Cross-sectional	1011	637	“How satisfied are you with the results of your joint replacement?”: ‘very satisfied’, ‘somewhat satisfied’, ‘neutral’, ‘somewhat dissatisfied’, ‘very dissatisfied’	Satisfaction with the operated knee
Sun et al. [56]	2012	China	64.7 (4.4)	RCT	132	80	Satisfaction (reported as % satisfied)	Satisfaction
Von Keudell et al. [57]	2014	USA	62.6 (11.2)	Cross-sectional	245	165	Satisfaction in respect to pain, motion, daily living function, return to sport activities and ability to kneel. Each dimension measured on a VAS (0 = not satisfied, 10 = very satisfied)	Composite
Walker et al. [58]	2018	UK	68.9 (9.6)	Retrospective cohort	2578	1396	‘How satisfied are you with the results of your knee replacement surgery’ ‘very satisfied’, ‘somewhat satisfied’, ‘somewhat dissatisfied’, and ‘very dissatisfied’	Satisfaction with the operated knee
Warner et al. [59]	2017	UK	73.1 (8.7)	Prospective cohort	1151	653	‘Individuals were asked to state how satisfied they felt with their total joint replacement using an ordinal scale’ ‘very satisfied’, ‘not very satisfied’, and ‘dissatisfied’	Satisfaction with the operated knee
Williams et al. [60]	2013	UK and Ireland	70.9 (8.6)	Prospective cohort	486	314	‘How do you feel overall about your replaced joint?’ ‘very happy’, ‘happy’, ‘OK (not perfect)’, or ‘never happy’	Satisfaction with the operated knee

3.3. Satisfaction estimates

Due to the heterogeneity in the focus of the satisfaction questions used by the studies, results were grouped into ‘like’ constructs. Composite scales consisting of questions with different foci of satisfaction are reported as composite, and also as single items under specific constructs where possible (see Fig. 3).

Fig. 3.

Proportion of patients satisfied after TKR. ES = effect size.

3.3.1. Single item satisfaction questions

The construct ‘Satisfaction with the operated knee’ included all questions that asked about satisfaction with the total joint replacement (TJR), TKR, operated knee, or surgery on the operated knee. Eighteen studies were included under this construct and the proportion satisfied ranged from as high as 97% (CI 90 to 100) to as low as 69% (CI 60 to 77). Two studies used a question which is part of the multi-domain Total Hip Arthroplasty Outcome Evaluation Questionnaire (THAOEQ) [44,45]. Thirteen studies provided no citation in support of the single item question used [9,23,24,26,28,30,38,54,55,[58], [59], [60], [61]], while 3 studies cited another study that had utilised the same single item question [12,29,46]; however, there was no further citation to support the validity of these questions.

The construct ‘Satisfaction’ included all questions that did not focus on any particular aspect of satisfaction. Of the 17 studies included, the proportion satisfied ranged from as high as 99% (CI 96 to 100) to as low as 73% (CI 58 to 85). One study used a question that was an item from the Self-Administered Patient Satisfaction Scale (SAPSS) [32], and 2 used an item from the surgeon-completed multi-domain British Orthopaedic Association (BOA) grading system [42,43]. Of the remaining studies, 13 provided no citation in support of the single item question used [25,27,34,36,37,39,48,50,52,53,56,62,63], and 1 study provided a citation that had no evidence of satisfaction content [49].

The construct ‘Satisfaction with function’ included all questions that asked about satisfaction with function, ADLs, sport, or recreation. The proportion satisfied from the 3 studies included ranged from 89% (CI 84 to 93) to to 39% (CI 29 to 50). One study used an item from the SAPSS [32], the remaining 2 studies used single items with no supporting citation [50,63].

The construct ‘Satisfaction with pain relief’ included all questions that asked about satisfaction with pain relief. Three were included and ranged from 90% (CI 76 to 97) to 84% (CI 78 to 89). One study cited a questionnaire, the SAPSS [32], while the remaining 2 studies did not have a supporting citation for their single item question [50,63].

The construct ‘Satisfaction with symptoms’ included all questions that asked about satisfaction with symptoms. Three were included and ranged from 85% (CI 76 to 92) to 72% (CI 67 to 77). Two studies used a question previously considered as a Patient Acceptable Symptom State (PASS) estimate [31,33]. The remaining study did not provide a citation in support of the single item question [63].

The construct ‘Satisfaction with aesthetics’ included questions that asked about the visual appearance of the knee. Only 1 study was included, which reported a satisfaction rate of 77% (CI 67 to 85), and did not provide a citation in support of the single item question [35].

The construct ‘Satisfaction with surgery’ included all questions that asked about satisfaction with the surgery but did not have reference to knee TJR, TKR, or operated knee. Two studies were included and ranged from 90% (CI 83 to 95) to 64% (57–71), neither of which provided a citation in support of the single item questions [10,47].

3.3.2. Composite scores

Three studies used composite instruments of items covering different components of satisfaction, with satisfaction estimates ranging from 88% (CI 83 to 92) to 60% (45–74). One study used the five-item satisfaction component of the New Knee Society Knee Scoring System (KSKSS) [40] which covers satisfaction with pain level while sitting, pain level while lying in bed, knee function while performing light household duties, and knee function while performing leisure recreational activities. One study used the four item SAPSS [41] covering overall satisfaction with surgery, satisfaction with pain relief, satisfaction with home and yard work, and satisfaction with recreational activities, and 1 study reported an unreferenced composite score of 5 items including satisfaction in respect to pain, motion, daily living function, return to sport activities, and ability to kneel [57].

3.4. Assessment of content validity

Of the 43 articles included in the review, 15 provided a citation for the satisfaction instrument used. Of these, only 9 studies, using a total of 6 satisfaction instruments, had a citation in support of content validity, in the form of a development study. These instruments included BOA, the new KSKSS, THAOEQ, and SAPSS, and questions previously considered indicators of Patient Acceptable Symptom State (PASS). The latter were excluded from further assessment as they pertain to current symptom state rather than to aspects related to TKR per se [64]. The BOA was excluded as it is completed by the surgeon and therefore not a PROM [65]. The THAOEQ was excluded as it was designed for a total hip replacement population rather than a TKR population [66], and in accordance with COSMIN criteria of ‘relevance’ cannot be considered for assessment of content validity [15]. Furthermore, the extent of development for the THAOEQ was poor and did not include patient appraisal [66]. An additional search was conducted for the SAPSS and new KSKSS to retrieve any further development or content validity studies (See Appendix 3), but none were identified. Both of these instruments were then assessed for content validity as per the COSMIN criteria (see Table 4a, Table 4ba and 4b and Appendix 4, Appendix 5).

Two development studies were retrieved for SAPSS: an abstract from 1998 [67] and a full text article from 2011 [20]. Both of these studies failed to demonstrate all three key aspects of content validity (see Table 4a). Although a Delphi panel of experts was used for development of the SAPSS, this did not include patient input, which is required for content validity. Reviewer rating of the instrument passed relevance and comprehensibility. The overall rating was a low quality PROM (see Table 4a). One development study was retrieved for the new KSKSS [68] and this study failed to demonstrate all three key aspects of content validity. Although the new KSKSS did have patient input in its development, this did not include the satisfaction items [68]. The five satisfaction items of the new KSKSS were based on the four item SAPSS, which as previously described did not include patient appraisal. Reviewer rating of the instrument passed relevance and comprehensibility. The overall rating was a low quality PROM (see Table 4b). Overall, none of the satisfaction instruments included in the review had adequate evidence of content validity.

4. Discussion

The aims of this review were to evaluate rates of patient reported satisfaction after TKR for OA across the literature, and to assess the content validity of the satisfaction measures utilised in evaluated studies. The results demonstrate heterogeneity in not only the focus of the satisfaction questions, but also the estimate of the proportion satisfied across studies.

From the 43 included studies, 8 satisfaction constructs were identified. In addition to heterogeneity in the satisfaction question used, heterogeneity in the estimate of satisfaction was also observed within constructs; most notably 39% (CI 29 to 50) compared to 89% (CI 84 to 93) in satisfaction with function. Due to the heterogeneity in satisfaction questions, it was not possible to pool all estimates, as per the Cochrane guidelines for systematic reviews [22]. Cochrane state that in the absence of longitudinal evidence of correlation of 2 or more PROMs, data pooling should not be conducted, but instead, grouping of like constructs as decided intuitively by the authorship team [22]. These findings are in alignment with the results of Kahlenberg [8] who also reported heterogeneous methods of measuring patient satisfaction after TKR [8].

The present review extends that of Kahlenberg [8] by evaluating the evidence for content validity of the utilised instruments. Two satisfaction instruments (SAPSS and new KSKSS) were cited by Kahlenberg [8] as being validated, but this was only in reference to construct validity, defined as the degree to which the scores of a PROM are consistent with hypotheses, based on the assumptions that the PROM validity measures the construct to be measured [15], or structural validity, which relates to how well the PROM scores reflect the dimensionally of the construct [15], not content validity. These 2 instruments were specifically evaluated for evidence of content validity in this current review, and no evidence for content validity was identified. Although reviewer ratings determined that both instruments had reasonable content relevance and comprehensiveness, the lack of patient involvement in the development of these instruments is a key concern for content validity.

Without patient consultation it is difficult to know whether these instruments include relevant items to accurately capture an individual's satisfaction with their TKR, whether they capture all aspects of satisfaction, or how patients comprehend/interpret the questions. Prior to designing a PROM, theoretical understanding of the construct of interest should be robust so to inform the content of the instrument [14,69]. In the case of satisfaction, PROM development has preceded theoretical understanding, compounding the difficulty in understanding how to measure this construct. This leaves researchers and clinicians to make assumptions regarding what satisfaction instruments are actually measuring. This lack of theoretical grounding in patient satisfaction instruments is a likely contributor to the variability in satisfaction instruments and estimates.

Given the limited understanding of patient satisfaction after TKR, some authors have based the design of satisfaction instruments on other correlates, such as improved pain or other disease-specific questionnaires. This approach is discussed by Robertsson et al. [54], who used a single item question: ‘three questions were asked, including one on satisfaction regarding the operated knee with four possible answers; 1) very satisfied, 2) satisfied, 3) uncertain, or 4) dissatisfied’. This question has not been validated, but has been replicated in three other studies included in the present review [23,29,46]. The authors suggest that a strategy to overcome the lack of content validity in satisfaction instruments is to demonstrate construct validity [54]. However, the presence of an association between a satisfaction instrument and other measures, such as self-reported disability or pain, does not mean the construct of satisfaction has been adequately captured in terms of relevance and comprehensiveness. For example, in the aforementioned study, 11% of patients chose ‘uncertain’ as the response option, and understanding this response is difficult due to a lack of the patients' perspective [54].

The results of the present review also highlighted numerous concerns regarding the appropriateness and consistency of satisfaction instruments. As mentioned earlier, the BOA, which was utilised by two studies [42,43], is designed to be completed by the surgeon rather than the patient, therefore this assessment cannot be considered a PROM [65]. The THOEQ, utilised by two studies [44,45], in addition to not being relevant to the target population, lacked any patient involvement and was only developed from the perspective of an orthopaedic task force that aimed to design a questionnaire from a patient perspective [66]. Questions considered indicators of PASS were utilised by two studies [31,33]. Although development of PASS questions has included patient involvement regarding the relevance and the external anchors during a special interest group meeting [64], they pertain to current symptom state rather than to aspects related to TKR per se [64]. Additionally, the Osteoarthritis Research Society International, which developed the PASS, has identified problems with the consistency of the PASS question and timeline of measurement in this population, suggesting further development studies are required [70]. Lastly, Dailiana [32], who cited the SAPSS, modified the instrument to include only three items of satisfaction as opposed to the four-item questionnaire designed by Mahomed [25], therefore not accurately representing the original intentions of the validated instrument.

Other measures such as ‘would you recommend a joint replacement to a friend?’, ‘would you have a joint replacement again?’, or the Forgotten Joint Score [71] have also been considered to reflect patient satisfaction after TKR in the literature. Although it may seem reasonable to assume these questions would align with satisfaction, this has not been investigated in a TKR population. Patient expectations have also been attributed to patient satisfaction after TKR [11,72]. Despite the literature search retrieving many studies measuring expectations as a means of gauging satisfaction, the authorship team chose not to include expectations as a measure of satisfaction due to it being undertheorised in a healthcare context [14,73]. Presently, expectations are understood from their historical origins in market research, whereby satisfaction is considered an evaluation of a purchase [73]. The role of expectation theory in understanding satisfaction with TKR remains unclear.

This review highlights a need for a better understanding of patient satisfaction after TKR, and suggests more care should be taken in how we interpret studies that use satisfaction as an end point. Future research should focus on conducting qualitative investigations on patient satisfaction after TKR, to build theoretical understanding and provide strong evidence of content validity. To achieve this, researchers may consider conducting focus groups or one on one interviews with patients who have undergone TKR, who have experienced a range of satisfaction, and pain and function outcomes. This has been demonstrated in the development of the Forgotten Joint Score, which sought patient opinion, in addition to multidisciplinary expert opinion, in choosing the items of the instrument [71]. The instrument was then further tested with a second group of patients to test the interpretation of the questions, and refine the question phrasings [74]. These same methods to achieve content validity should be applied to satisfaction instruments after TKR. A better understanding of what patient satisfaction is and how to measure it will optimise the delivery of high quality, patient-centred care in orthopaedics.

Author contributions

All authors have read and approved the present submission to Osteoarthritis and Cartilage Open. Substantial contributions to the research design, or the acquisition, analysis or interpretation of data was completed by NK, SB, PK, MD, PO, RS, RF and AS. Drafting the paper, or critical revisions were completed by NK, SB, PK, MD, PO, RS, PC and AS.

Declaration of Competing Interest

This work was supported by the Centre for Research Excellence in Total Joint Replacement (APP1116325), under a National Health & Medical Research Council, Australia, grant. In addition, personal declarations are as follows:

COI Anne Smith reports grants from National Health & Medical Research Council, during the conduct of the study.

COI Peter O'Sullivan reports grants from National Health & Medical Research Council, during the conduct of the study.

COI Peter Choong reports grants from National Health & Medical Research Council, during the conduct of the study; personal fees from Stryker, personal fees from Johnson & Johnson, grants from Medacta, personal fees from Kluwer, outside the submitted work. Additionally, COI Peter Choong is supported by a National Health & Medical Research Council Practitioner Fellowship (APP1154203).

COI Michelle Dowsey reports grants from National Health & Medical Research Council, during the conduct of the study; grants from Medacta, outside the submitted work.

Additionally, COI Michelle Dowsey is supported by a National Health & Medical Research Council Career Development Fellowship (APP1122526).

Appendix 1. Search strategy in Medline

1. Patient* adj5 satisf*.mp 2. Knee adj3 replac*.mp 3. Knee adj2 arthroplasty.mp 4. 2 or 3 5. 1 and 4

mp denotes keyword.

Appendix 2. Satisfaction with Total Knee Replacement - Risk of Bias Tool

This tool is designed to assess the risk of bias in studies of satisfaction after Total Knee Replacement (TKR). Please read the additional notes for each item when initially using the tool. Note: If there is insufficient information in the article to permit a judgement for a particular item, please answer No (HIGH RISK) for that particular item.

Risk of bias item	Criteria for answers (please circle one option)	Additional notes and examples
External Validity
1. Was the study's target population representative of most TKR populations on relevant demographic and clinical variables, e.g. age, sex, pain severity, osteoarthritis grade?*	•Yes (LOW RISK): The study's target population was a close representation of most TKR populations. •No (HIGH RISK): The study's target population was clearly NOT representative of most TKR populations.	The target population refers to the group of patients to which the results of the study will be generalised. Examples: •The study was a survey of patients in a hospital department and the sample was drawn from a list that included all individuals operated on over a two-year period. The answer is: Yes (LOW RISK). •The study was conducted in one province only, and it is not clear if this was representative of the national population. The answer is: No (HIGH RISK). •The study was undertaken asking responses from people considering revision surgery and it is clear this was not representative of most TKR populations. The answer is: No (HIGH RISK).
2. Was the sampling frame a true or close representation of the TKR population?*	•Yes (LOW RISK): The sampling frame was a true or close representation of the TKR population. •No (HIGH RISK): The sampling frame was NOT a true or close representation of the TKR population.	The sampling frame is a list of the sampling units in the target population and the study sample is drawn from this list. Examples: •The sampling frame was a database of every individual who received a TKR within a hospital. The answer is: Yes (LOW RISK). •The study asked responses from anonymous people in an online chat group. The answer is: Yes (LOW RISK).
3. Was some form of consecutive or random selection used to select the sample?	•Yes (LOW RISK): Some form of consecutive or random selection was used to select the sample (e.g. simple random sampling, stratified random sampling, cluster sampling, systematic sampling). •No (HIGH RISK): Some form of consecutive or random selection was NOT used to select the sample.	In a survey, only part of the sampling frame is sampled. In these instances, consecutive or random selection of the sample helps minimise study bias. Examples: •Every person in a consecutive sample was surveyed. The answer is: Yes (LOW RISK). •The sample was selected using simple random sampling. The answer is: Yes (LOW RISK). •A clinician asked a non-consecutive sample of his/her patients. The answer is: No (HIGH RISK).
4. Was the likelihood of non-response bias minimal?*	•Yes (LOW RISK): The response rate for the study was ≥75%, OR, an analysis was performed that showed no significant difference in relevant demographic and clinical characteristics between responders and nonresponders •No (HIGH RISK): The response rate was <75%, and if any analysis comparing responders and non-responders was done, it showed a significant difference in relevant demographic and clinical characteristics between responders and non-responders.	Examples: •The response rate was 68%; however, the researchers did an analysis and found no significant difference between responders and non-responders in terms of age, sex and clinician status. The answer is: Yes (LOW RISK). •The response rate was 65% and the researchers did NOT carry out an analysis to compare relevant characteristics between responders and non-responders. The answer is: No (HIGH RISK). •The response rate was 69% and the researchers did an analysis and found a significant difference in age, sex and clinical status between responders and non-responders. The answer is: No (HIGH RISK).
Internal Validity
5. Were data collected* directly from the participants (as opposed to a proxy)?	•Yes (LOW RISK): All data were collected directly from the participants. •No (HIGH RISK): In some instances, data were collected from a proxy.	A proxy is a representative of the subject. Examples: •All eligible participants were surveyed directly. The answer is: Yes (LOW RISK). •A clinician, or series of clinicians, estimated how satisfied their patients were. The answer is: No (HIGH RISK).
6. Was an acceptable participant definition TKR used in the study?*	•Yes (LOW RISK): An acceptable participant definition was used. •No (HIGH RISK): An acceptable case participant definition was NOT used.	In a study, the following participant definition was used: “All participants must have had a TKR, which is a surgical procedure to replace the weight-bearing surfaces of the knee joint to relieve pain and disability.” The answer is: Yes (LOW RISK). •In a study, the following participant definition was used: “Participants needed to have received some form of knee surgery.” The answer is: No (HIGH RISK).
7. Was the study instrument that measured satisfaction shown to have reliability and validity (if necessary)?	•Yes (LOW RISK): The study instrument had been shown to have reliability and validity, e.g. test-retest, piloting, validation in a previous study, etc. •No (HIGH RISK): The study instrument had NOT been shown to have reliability or validity	•The authors used a questionnaire, which had previously been validated. They also tested the inter-rater reliability of the questionnaire. The answer is: Yes (LOW RISK). •The authors developed their own questionnaire and did not test this for validity or reliability. The answer is: No (HIGH RISK).
8. Was the same mode of data collection used for all participants?	•Yes (LOW RISK): The same mode of data collection was used for all subjects. •No (HIGH RISK): The same mode of data collection was NOT used for all subjects.	The mode of data collection is the method used for collecting information from the subjects. The most common modes are face-to-face interviews, telephone interviews and self-administered questionnaires. Examples: •All eligible subjects had a face-to-face interview. The answer is: Yes (LOW RISK). •Some subjects were interviewed over the telephone and some filled in postal questionnaires. The answer is: No (HIGH RISK).
9. Was the length of the measurement period of satisfaction with TKR appropriate?*	•Yes (LOW RISK): The shortest measurement period of satisfaction with TKR was appropriate. •No (HIGH RISK): The shortest measurement period was not appropriate	The measurement period is the length of time post-surgery. The shorter the measurement period, the greater the likelihood of the participant's satisfaction being about the operative and rehabilitation process rather than about the medium-term or longer-term residual pain and functional capacity. Examples: •Participants were asked about satisfaction with TKR when they were 12-months post-surgery. The answer is: Yes (LOW RISK). •Participants were asked about satisfaction when they were 2-months post-surgery. The answer is: No (HIGH RISK). Note: A follow-up of >6 months was part of the inclusion criteria for this review. Therefore all studies with <6 months were excluded from further consideration in the full text screening.
10. Were the statistical measures of satisfaction appropriate?*	•Yes (LOW RISK): The paper presented adequate description of how the summary statistics were calculated, the statistics were appropriate and would be possible to be reproduced in a replication study. •No (HIGH RISK): The paper did present adequate description of the statistics or one or more of these were inappropriate.	•The individual items in the satisfaction questionnaire were scored and summarised using the method the questionnaire developers validated, and a group mean score was reported with 95% confidence intervals). The answer is: Yes (LOW RISK). •It is not clear how the measure of satisfaction was scored and/or summarised. The answer is: No (HIGH RISK).
11. Summary item on the overall risk of study bias
•LOW RISK OF BIAS: Further research is very unlikely to change our confidence in the estimate. •MODERATE RISK OF BIAS: Further research is likely to have an important impact on our confidence in the estimate and may change the estimate. •HIGH RISK OF BIAS: Further research is very likely to have an important impact on our confidence in the estimate and is likely to change the estimate.

Items with an asterisk will exclude a paper from further consideration (items 1, 2, 4, 5, 6, 9 and 10).

Appendix 3. Content validity search strategy example

Eg: New Knee Society Knee Scoring System.

MEDLINE, SCOPUS, Embase.

Appendix 4. COSMIN assessment of PROM development studies.

Appendix 5. COSMIN assessment of PROM content validity studies.

Appendix 6. Risk of Bias results of all included studies

	Study target population representative of TKR populations*	Sampling frame a true or close representation of TKR populations*	Consecutive or random sample used	Likelihood of non-response bias minimal*	Data collected directly from participants*	Acceptable participant definition TKR used*	Study instrument that measured satisfaction shown to have reliability and validity	Same mode of data collection used for all participants	Appropriate length of measurement period of satisfaction with TKR*	Appropriate statistical measures of satisfaction*
Ali et al., 2016	+	+	+	+	+	+	–	+	+	+	Moderate risk
Ali et al., 2017	+	+	+	+	+	+	–	+	+	+	Moderate risk
Aunan and Rohl 2018	+	+	+	+	+	+	–	+	+	+	Moderate risk
Baker et al., 2007	+	+	+	+	+	+	–	+	+	+	Moderate risk
Blyth et al., 2015	+	+	+	+	+	+	–	+	+	+	Moderate risk
Boese et al., 2011	+	+	+	+	+	+	–	+	+	+	Moderate risk
Chinnappa et al., 2017	+	+	+	+	+	+	–	+	+	+	Moderate risk
Clement et al., 2013	+	+	+	+	+	+	–	+	+	+	Moderate risk
Collados Maestre et al., 2016	+	+	+	+	+	+	–	+	+	+	Moderate risk
Collins et al., 2017	+	+	+	+	+	+	–	+	+	+	Moderate risk
Culliton et al., 2018	+	+	+	+	+	+	+	+	+	+	Low risk
Dailiana et al., 2015	+	+	+	+	+	+	+	+	+	+	Low risk
Escobar et al., 2013	+	+	+	+	+	+	+	+	+	+	Low risk
Gaillard et al., 2017	+	+	+	+	+	+	–	+	+	+	Moderate risk
Gandhi et al., 2007	+	+	+	+	+	+	–	+	+	+	Moderate risk
Genet et al., 2008	+	+	+	+	+	+	–	+	+	+	Moderate risk
Gildone et al., 2005	+	+	+	+	+	+	–	+	+	+	Moderate risk
Giurea et al., 2016	+	+	+	+	+	+	–	+	+	+	Moderate risk
Healy et al., 2002	+	+	+	+	+	+	–	+	+	+	Moderate risk
Hinarejos et al., 2016	+	+	+	+	+	+	–	+	+	+	Moderate risk
Kawakami et al., 2015	+	+	+	+	+	+	+	+	+	+	Low risk
Khuangsirku et al., 2016	+	+	+	+	+	+	+	–	+	+	Moderate risk
Kim et al., 2009	+	+	+	+	+	+	+	+	+	+	Low risk
Lange et al., 2018	+	+	+	+	+	+	–	+	+	+	Moderate risk
Li et al., 2012	+	+	+	+	+	+	+	+	+	+	Low risk
Liebs et al., 2010	+	+	+	+	+	+	+	+	+	+	Low risk
Liebs et al., 2012	+	+	+	+	+	+	+	+	+	+	Low risk
Lizaur Utrilla et al., 2016	+	+	+	+	+	+	–	+	+	+	Moderate risk
Mannion et al., 2009	+	+	+	+	+	+	–	+	+	+	Moderate risk
Matthews et al., 2013	+	+	–	+	+	+	–	+	+	+	Moderate risk
Mooney et al., 2016	+	+	+	+	+	+	–	+	+	+	Moderate risk
Murphy et al., 2014	+	+	–	+	+	+	–	+	+	+	Moderate risk
Nilsdotter et al., 2009	+	+	+	+	+	+	–	+	+	+	Moderate risk
Petersen et al., 2015	+	+	+	+	+	+	–	+	+	+	Moderate risk
Pulavarti et al., 2014	+	+	+	+	+	+	–	+	+	+	Moderate risk
Ranawat et al., 2017	+	+	+	+	+	+	–	–	+	+	Moderate risk
Robertsson et al., 2000	+	+	+	+	+	+	–	+	+	+	Moderate risk
Stickles et al., 2001	+	+	+	+	+	+	–	+	+	+	Moderate risk
Sun et al., 2012	+	+	+	+	+	+	–	+	+	+	Moderate risk
Von keudell et al., 2014	+	+	–	+	+	+	–	–	+	+	Moderate risk
Walker et al., 2018	+	+	+	+	+	+	–	+	+	+	Moderate risk
Warner e al. 2017	+	+	+	+	+	+	–	+	+	+	Moderate risk
Williams et al., 2013	+	+	+	+	+	+	–	+	+	+	Moderate risk

An asterisk (*) denotes a 'fatal flaw' criteria, where failure of any of these items results in immediate exclusion.