Health Economic Evaluations of Hip and Knee Interventions in Orthopaedic Sports Medicine: A Systematic Review and Quality Assessment

Abstract

Background:

The economic burden of musculoskeletal diseases is substantial and growing. Economic evaluations compare costs and health benefits of interventions simultaneously to help inform value-based care; thus, it is crucial to ensure that studies are using appropriate methodology to provide valid evidence on the cost-effectiveness of interventions. This is particularly the case in orthopaedic sports medicine, where several interventions of varying costs are available to treat common hip and knee conditions.

Purpose:

To summarize and evaluate the quality of economic evaluations in orthopaedic sports medicine for knee and hip interventions and identify areas for quality improvement.

Study Design:

Systematic review; Level of evidence, 3.

Methods:

The Medline, AMED, OVID Health Star, and EMBASE databases were searched from inception to March 1, 2020, to identify economic evaluations that compared ≥2 interventions for hip and/or knee conditions in orthopaedic sports medicine. We assessed the quality of full economic evaluations using the Quality of Health Economic Studies (QHES) tool, which consists of 16 questions for a total score of 100. We classified studies into quartiles based on QHES score (extremely poor quality to high quality) and we evaluated the frequency of studies that addressed each of the 16 QHES questions.

Results:

A total of 93 studies were included in the systematic review. There were 41 (44%) cost analyses, of which 21 (51%) inappropriately concluded interventions were cost-effective. Only 52 (56%) of the included studies were full economic evaluations, although 40 of these (77%) fell in the high-quality quartile. The mean QHES score was 83.2 ± 19. Authors consistently addressed 12 of the QHES questions; questions that were missed or unclear were related to statistical uncertainty, appropriateness of costing methodology, and discussion of potential biases. The most frequently missed question was whether the cost perspective of the analysis was stated and justified.

Conclusion:

The number of studies in orthopaedic sports medicine is small, despite their overall good quality. Yet, there are still many highly cited studies based on low-quality or partial economic evaluations that are being used to influence clinical decision-making. Investigators should follow international health economic guidelines for study design and critical appraisal of studies to further improve quality.

Musculoskeletal diseases account for approximately 30% of the global burden of disease in adults, ⁸ which poses an increasingly heavy economic burden on health care systems, particularly among an aging population. Orthopaedic sports medicine is a growing area of interest in health care, which includes interventions for acute soft tissue and ligamentous injuries; degenerative conditions including osteoarthritis; chronic pain; and other joint impairments. The hip and knee are the most commonly affected joints for these conditions. ^19,31,110 Often, surgical interventions in this field are considered elective with substantial associated costs that continue to increase with further advancements in technology. Comparatively, nonoperative treatments can heavily drive direct health care costs through repeated intervention periods with health benefits that are often transient in nature. Nonoperative treatments can also lead to substantial indirect costs through work absenteeism or the loss of productive output for the economy. It is therefore crucial to identify treatments in orthopaedic sports medicine that provide our best value for care.

Economic evaluations can help policy makers decide on appropriate allocation of health resources. They provide a framework for comparing clinical effect and cost concurrently between competing interventions to assess the best value for scarce health care dollars, especially in public health care systems. ²⁶ Health care delivery cost has become an important focus for health care policy and resource allocation in many countries, ^9,91 with health economics research becoming increasingly vital to help in the decision-making process. This is particularly true in the field of orthopaedic sports medicine. It is therefore essential to evaluate the quality of the methodology for these studies to ensure valid study conclusions.

A systematic review published in 2015 assessed the quality of economic evaluations in orthopaedic sports medicine ⁷⁷ and found that although the quality of included studies was generally good, a small number of studies (n = 12) were evaluated. The systematic review study focused on full economic evaluations that evaluated only surgical interventions published in the United States. Since then, across the globe, several more health economic evaluations have been published in the field, particularly regarding interventions related to the hip and knee joints. We believe it is important to systematically assess economic evaluations internationally and include all hip and knee interventions to get a comprehensive understanding of the current literature.

To be considered a full economic evaluation, a study must include both cost and a health outcome in the analysis and compare 2 or more treatment alternatives. ²⁶ Several methods of full economic evaluation exist (Table 1). Importantly, a study designed as a full economic evaluation is required to make informed decisions on best value for care. To the best of our knowledge, no studies have yet explored the quality of economic evaluations internationally for all hip and knee interventions in the orthopaedic sports medicine literature.

TABLE 1.

Types of Economic Evaluation ^a

Type	Description
Full economic evaluation
Cost-effectiveness analysis	Evaluates 2+ interventions by simultaneously comparing the cost and a health outcome measured in natural units, specific to the disease being studied (eg, survival or adverse events).
Cost-utility analysis	Evaluates 2+ interventions by simultaneously comparing the cost and a health outcome measured through a utility score where years of life are adjusted for health-related quality of life (eg, quality-adjusted life year).
Cost-benefit analysis	Evaluates 2+ interventions by comparing the cost and a health outcome measured in monetary units (eg, assigning a dollar value to health status).
Cost-minimization	Evaluates 2+ interventions by comparing the cost and a health outcome, where the analysis finds no difference in health outcome, and therefore only the difference in cost is reported.
Partial economic evaluation
Cost analysis	Evaluates 2+ interventions by comparing the cost only between 2+ interventions (ie, no health outcome in the analysis).

^a Costs for all design types are measured in monetary units.

Therefore, the aim of this study was to conduct a systematic review to (1) evaluate and summarize reporting of results in the orthopaedic sports medicine literature on cost-effectiveness for knee and hip interventions, (2) assess the quality of published full economic evaluations using the Quality of Health Economic Studies (QHES) tool, and (3) identify areas in which study quality improvement is required. We hypothesized that several studies do not meet the criteria for a full economic evaluation and that specific methodological components need to be improved for overall better study quality and interpretation of results by clinicians.

Methods

Eligibility Criteria

To meet eligibility criteria, studies needed to compare 2 or more interventions, evaluate hip and/or knee interventions that were nonarthroplasty or trauma-related, and report a cost outcome. Included were studies evaluating surgical, nonpharmacological, and pharmacological interventions in orthopaedic sports medicine (eg, anterior cruciate ligament reconstruction, arthroscopy, physical therapy, diet and exercise, pharmacotherapy, etc).

Search Strategy

A literature search was performed using 4 OVID databases including Medline, AMED, OVID Health Star, and EMBASE from inception to March 1, 2020, to identify studies published in English encompassing combined and/or truncated key terms including cost, cost-effectiveness, cost-benefit, cost-minimization, cost-utility, economic, or economic evaluation as well as hip or knee.

Abstract and Full-Text Screening

Studies were divided among 3 reviewer groups (C.A.P. and B.O.Z., L.E.S. and J.D.M.) to independently screen titles and abstracts for studies identified in the literature search. Reviewers determined whether each study should be included for full-text review or excluded based on the eligibility criteria. Duplicate papers, conference abstracts, and study protocols were removed from the list. We also manually searched the bibliographies of systematic reviews for other potentially relevant studies that met the eligibility criteria. The full-text article for all eligible studies was retrieved, and we further examined each full text to determine whether it still met eligibility criteria. We then divided papers among 2 reviewer pairs (C.A.P. and B.O.Z., I.J. and J.D.M.) to determine eligibility and conduct a full-text review for those that met the criteria.

Data Abstraction

All reviewers used a custom form to extract data from the full text of included studies. The data extraction form included the following: (1) year of publication; (2) type of economic evaluation (ie, cost analysis, cost-effectiveness analysis, cost-utility analysis, cost-minimization analysis, or more than 1 category); (3) study design (ie, trial- or model-based); (4) trial design (ie, randomized trial, prospective cohort, retrospective cohort, etc) for trial-based studies; (5) model design (ie, decision tree, Markov model, or other) for model-based designs; (6) study interventions; (7) health condition; (8) country; (9) the summary measure reported (incremental cost-effectiveness ratio [ICER] or incremental cost-utility ratio [ICUR]) or incremental net benefit (INB); (10) whether authors reported uncertainty using 95% CIs, bootstrapping, a cost-effectiveness plane, or a cost-effectiveness acceptability curve; (11) whether sensitivity analyses were performed; and (12) the perspective of the analysis (ie, patient, hospital, payer, societal, or more than 1 category). Reviewer pairs (C.A.P. and B.O.Z., I.J. and J.D.M.) met after completing full-text reviews to discuss the extracted data and resolve any conflicts between reviewers. If consensus was not met between the reviewer pair, we had a third reviewer (from the other reviewer pair) resolve any discrepancies.

Assessment of Quality of Studies

The quality of each full economic evaluation study was evaluated using the QHES tool (see Supplemental Table 1, available online). ^14,79 The QHES is a validated and reliable quantitative measurement tool designed to evaluate the methodology, validity, and transparency of health economic evaluations. ^14,79 The tool is composed of 16 binary questions (ie, answered yes or no) that evaluate several important elements required for a high-quality health economic evaluation (Supplemental Table 1, available online). Each question has an assigned score value ranging from 1 to 9, ^14,79 where questions answered “yes” receive the full point value. Questions answered “no” receive no points. The question scores are summed to obtain a final summary score ranging from 0 to 100 points, where higher scores represent higher quality.

Previous studies have recommended the QHES should be assessed by at least 2 reviewers and that a clear definition for each question needs to be established and made available to all reviewers before scoring studies. ³⁶ Therefore, the same reviewer pair that completed data abstraction independently assessed the quality of each included study using the QHES tool. We used the criteria set by Marshall et al ⁶⁸ to supplement the scoring of each of the 16 questions, with slight modifications and clarifications (Supplemental Table 2, available online). To ensure questions were being interpreted consistently among reviewers, we pilot tested 5 of the studies before completing all QHES evaluations. Once all studies were assessed using the QHES, the reviewer pairs met to discuss any discrepancies. Any conflicts were resolved through the third reviewer.

Interrater Agreement

We assessed the agreement between reviewer pairs for abstract screening by using the Cohen kappa statistic, where 0.81 to 0.99 indicates an almost perfect agreement, 0.61 to 0.80 indicates substantial agreement, 0.41 to 0.60 indicates moderate agreement, 0.21 to 0.40 indicates fair agreement, 0.01 to 0.20 indicates slight agreement, and <0.01 indicates less than chance agreement. ¹⁰⁹ We also reported the agreement between reviewer pairs for each of the 16 questions of the QHES by calculating the percentage of observed agreements for each of the 16 questions (ie, percentage agreed “yes” that a question was addressed or agreed “no” it was not addressed).

Descriptive Analysis

The following descriptive analyses were conducted to summarize findings: (1) the frequency (with percentage) of studies by design characteristics, study population, and year of publication; (2) the mean (with standard deviation) QHES score of all studies; (3) the mean QHES score by publication year, geographical location, and intervention studied; and (4) the frequency (with percentage) of studies that addressed each of the 16 questions of the QHES. Studies were also classified into 1 of 4 quartiles based on the QHES total score: high quality (75-100); fair (50-74); poor (25-49); or extremely poor (0-24). ¹⁰⁰

Results

Studies Overview

A flowchart outlining the stages for study screening are shown in Figure 1. We identified 152 studies that initially met the inclusion criteria and were considered for full-text review. After reviewing full-text articles, we found that 61 studies did not meet eligibility criteria and were therefore excluded. We identified 2 additional articles from other systematic reviews. Therefore, 93 studies ^# that met the inclusion criteria were included in the systematic review (Figure 1). An overall summary of the included studies (N = 93) is provided in Table 2 as well as subdivision for full economic evaluations only (n = 52).

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) study flowchart.

TABLE 2.

Summary of Included Studies Overall (N = 93) and By Those Considered Full Economic Evaluations (n = 52) ^a

	Overall (N = 93)	Full Economic Evaluations (n = 52)
Type of economic evaluation
Partial	41 (44)
Full	52 (56)
Cost-utility analysis		36 (69)
Cost-effectiveness analysis		13 (25)
Cost-benefit analysis		0 (0)
Cost-minimization analysis		0 (0)
Cost-utility and cost-effectiveness analyses		2 (4)
Cost-utility and cost-benefit analyses		1 (2)
Study design
Model-based
Decision tree	10 (11)	10 (19)
Markov model	14 (15)	13 (25)
Unclear	2 (2)	1 (2)
Trial-based
Randomized controlled trial	37 (40)	19 (37)
Prospective	10 (11)	4 (8)
Retrospective	18 (19)	4 (8)
Other b	2 (2)	1 (2)
Uncertainty c
Yes	30 (32)	26 (50)
No	63 (68)	26 (50)
Sensitivity analysis d
Yes	38 (41)	33 (63)
No	55 (59)	19 (37)

^a Data are reported as n (%).

^b Other study designs included a quasi-experimental design or a combination of prospective and retrospective components.

^c Yes = study quantified statistical uncertainty of their estimates using 95% CIs, bootstrapping, a cost-effectiveness plane, and/or a cost-effectiveness acceptability curve.

^d Yes = study conducted sensitivity analyses using a 1-way, multiway, and/or probabilistic approach to account for parameter uncertainty.

Notably, 21 (51%) of the 41 partial economic evaluation studies (ie, cost analyses) claimed that the treatments were cost-effective even though they did not carry out a full economic evaluation. The studies included for this review were published between 1982 and 2019, and the overall frequency of studies published increased over time (Figure 2). When dividing by type of economic evaluation, the frequency of partial economic evaluations increased until the years 2000 to 2004 and then decreased while the frequency of full economic evaluations increased over time (Figure 2). Individual study summaries are also provided in Supplemental Table 3 and Supplemental Table 4 (available online).

Figure 2.

The frequency of economic evaluations relating to sports medicine (N = 93) published in English over time (ie, years). Studies are separated by type of economic evaluation, with partial economic evaluations (n = 41) presented in pink and full economic evaluations (n = 52) presented in purple. The bar labels represent the number of studies for each year interval.

Interventions Studied

Studies evaluated surgical interventions in 41 studies (44%), pharmacological and injection interventions in 22 studies (23%), rehabilitation and/or lifestyle interventions in 21 studies (23%), and surgical interventions compared to nonoperative management in 9 studies (10%).

Of all studies, 83 (89%) evaluated interventions for conditions related to the knee, while 8 (9%) were for the hip and/or knee and 2 (2%) were solely for the hip. The interventions evaluated were for the treatment of knee osteoarthritis (OA) (n = 31; 33%), anterior cruciate ligament injury/rupture (n = 18; 19%), hip and/or knee OA (n = 7; 8%), generalized knee pain (n = 7; 8%), cartilage lesions (n = 6; 6%), or meniscal injury/tear (n = 4; 4%). There were also 15 studies (16%) that evaluated arthroscopic interventions where the underlying health condition was not specified. The remaining 5 studies evaluated interventions related to femoroacetabular impingement, hip OA, generalized acute knee injuries, posterior cruciate ligament injury, and rheumatoid arthritis (localized to the knee).

Interrater Agreement

Interrater agreement was high for title and abstract screening (κ = 0.81, 0.76, and 0.63 for each of the pairings), indicating substantial to almost perfect agreement between each of the 3 reviewer pairs. ¹⁰⁹ The mean interrater agreement across all 16 questions of the QHES for the quality assessment of full economic evaluations was high for both reviewer pairs (92% and 95%, respectively). Generally, raters showed strong agreement for 14 of the QHES questions (Q1-5, Q7, and Q9-16) in both reviewer pairs (ie, agreement ≥ 80%) (Table 3).

TABLE 3.

Interrater Agreement Between 2 Reviewer Pairs in 52 Studies Evaluating Cost-Effectiveness of Interventions in Orthopaedic Sports Medicine Using the Quality of Health Economics Studies (QHES) Tool

		% Agreement
Question	Focus	Reviewer Pair 1	Reviewer Pair 2
1	Clear objective	94	100
2	Stated perspective	82	89
3	Best available data source	97	95
4	Subgroups prespecified	100	100
5	Consideration of uncertainty	94	100
6	Incremental analysis performed (ie, cost-effectiveness ratios)	85	74
7	Methodology for data abstraction	85	100
8	Appropriate time horizon and discounting	79	95
9	Appropriate costing methodology	85	89
10	Primary outcome	97	95
11	Validity/reliability of outcome measure(s)	94	100
12	Description of model	88	89
13	Appropriate model	97	100
14	Potential bias(es)	94	100
15	Justification of conclusion	97	95
16	Funding statement	94	100

Quality of Included Studies

Overall, 52 full economic evaluations were assessed for quality. The mean QHES score of the identified studies was 83.2 ± 19 (range, 35-100), which is considered high quality according to the QHES quartiles. ¹⁰⁰ Most studies (77%) were considered high quality (QHES 75-100), while 15% were considered fair (QHES 50-74) and 8% were considered poor (QHES 25-49) (Figure 3). Study quality did not show a visible trend over time (Figure 4). Individual QHES scores for each study are provided in Supplemental Table 5 (available online).

Figure 3.

The Quality of Health Economics (QHES) total score for all studies (n = 52) by point decile.

Figure 4.

The Quality of Health Economics (QHES) total score for all studies (n = 52) over time (years).

When separated by geographical location, studies conducted in Canada, the United States, and the United Kingdom were generally considered high quality, while studies conducted in European countries (Belgium, France, Germany, Italy, the Netherlands, Norway, Slovenia, and Spain) were considered fair (Figure 5).

Figure 5.

Mean Quality of Health Economics (QHES) total score by geographical location (n = 52).

When separated by interventions studied, studies evaluating surgical interventions (12 studies; 23%), pharmacological and injection (17 studies; 33%), and rehabilitation and/or lifestyle interventions (15 studies; 29%) were considered high quality, while studies evaluating surgical interventions versus nonoperative management (8 studies; 15%) were considered fair (Figure 6).

Figure 6.

Mean Quality of Health Economics (QHES) total score by study intervention group (n = 52).

Frequency of QHES Questions Addressed

For all included studies, 12 questions from the QHES tool (Q1, Q3, Q4, Q6-Q8, Q10-Q13, Q15, Q16) were frequently addressed (ie, >80%) (Figure 7). By comparison, questions that were addressed less frequently were related to whether studies considered uncertainty (Q5), the appropriateness of costing methodology (Q9), and discussion of potential biases (Q14). The question that was addressed the least was Q2 related to whether the perspective of the analysis was stated and justified.

Figure 7.

The percentage of orthopaedic sports medicine studies addressing each of the 16 questions of the Quality of Health Economics (QHES) (n = 52).

Discussion

In this review, we summarized the quality of health economic studies in orthopaedic sports medicine. There were 93 economic evaluations published since 1982 in the field, approximately half of which (41 studies, 44%) did not complete a full economic evaluation. Importantly, over half (21 of 41 studies, 51%) of these studies inappropriately concluded that the treatments were cost-effective even though they did not carry out a full economic evaluation. This is particularly problematic as these studies are often highly cited and used in decision-making for clinical practice, despite potentially misleading study conclusions.

International health economic guidelines recommend clinical and policy decisions be made based on results provided from full economic evaluations that consider both the effect and cost of interventions simultaneously. ^9,75,91 While our results showed an increasing number of full economic evaluations conducted over the past 16 years (Figure 2), the number of published studies in orthopaedic sports medicine remains small. We highlight the importance of conducting more health economic research of high quality in the field to help guide clinical decision-making and policy decisions. Specifically, few studies evaluated interventions for either the hip or knee joints (9%), and even fewer (2%) evaluated interventions targeted at treating diseases of the hip joint alone.

Despite the smaller study volume, the quality of published full economic evaluations (n = 52) was generally high, ¹⁰⁰ with a mean total QHES score of 83.2 ± 19. No observable trend for change in quality was observed over time (Figure 4), but studies conducted in Canada, the United States, or the United Kingdom (69% of studies) were generally of high quality (Figure 5), suggesting methodologies for economic evaluation are likely well-developed in these countries. Most studied intervention types were high quality, on average; however, studies that compared surgical with nonoperative interventions were considered fair (Figure 6).

Nwachukwu et al ⁷⁷ also performed a systematic review of full health economic evaluations in orthopaedic sports medicine; however, the authors included only US-based studies and those that conducted full economic evaluations. Their review identified 12 studies that met the inclusion criteria and reported a mean score of 81.8 on the QHES, in line with our study results. Furthermore, when we looked solely at the US studies from our review (n = 22), the mean QHES score rises to 88 ± 17 with studies published since the review by Nwachukwu et al all scoring above 90. These results are encouraging, as they suggest the overall quality of economic evaluations may be improving in the United States, despite our results not showing a trend for improvement over time internationally (Figure 4). When compared with other fields of health care, the quality of economic evaluations in orthopaedic sports medicine is generally high. ¹⁰⁰ Other reviews that have evaluated interventions in physical therapy, ⁸³ ischemic heart disease, ¹⁰¹ and hip and knee arthroplasty ⁸⁶ have also shown similar study quality using the QHES tool. Comparatively, studies evaluating interventions in radiotherapy for oncology, ²² nursing, ⁶⁸ and digestive diseases ¹⁰⁰ have reported mean QHES scores ranging from fair to poor. ¹⁰⁰ Reporting guidelines such as the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement are also useful in evaluating economic evaluations; however, we used the QHES for our quality assessment as it was developed to specifically measure the quality of economic evaluations and produce an overall quality score, which enables comparison with other studies. ⁷⁷

Through our quality assessment, we identified key areas of health economic studies that require improvement in orthopaedic sports medicine. Most QHES questions were frequently addressed (Figure 7); however, 4 questions were addressed less frequently. These questions were related to whether authors handled estimate uncertainty in their analysis, the methodology for data abstraction and costing, and whether authors justified the model they chose and discussed the potential limitations and biases of their results. On a general note, researchers should be using guidelines ^9,75,91 that provide detailed instructions for conducting a health economic evaluation. For example, these guidelines provide detailed descriptions on the importance of accounting for uncertainty statistically, but also through sensitivity analyses to account for uncertainty of study parameters, such as assumptions made by the investigators. Following guidelines for reporting economic evaluations will help improve transparency of results and, subsequently, the overall quality of studies.

The most poorly addressed question (Q2; 65% of studies) was whether the perspective of the analysis was stated and justified. Similarly, perspective was also poorly addressed in studies that conducted partial economic evaluations. Stating the perspective is important to ensure that all relevant costs are measured appropriately. Use of different perspectives can have meaningful impacts on study results and interpretation. Primeau et al ⁸⁷ found that comparing interventions through a payer costing perspective (ie, direct costs only) versus a societal perspective (ie, also including indirect costs such as time away from work) can provide contradicting results concerning intervention cost-effectiveness. For example, considering solely the direct costs in an economic evaluation could substantially undermine the true benefit of an intervention that leads to large indirect cost savings for the patient or society as a whole. As the distribution of cost varies considerably between operative and nonoperative procedures, it is particularly important that studies comparing such interventions are conducted according to economic evaluation guidelines to appropriately capture all relevant direct and indirect costs and estimate the trade-off between health care costs and clinical outcome, thus ensuring appropriate conclusions regarding cost-effectiveness are drawn. Interestingly, studies that compared these types of interventions scored the lowest on the QHES (Figure 6), and very few studies were identified (n = 8). Reporting the study perspective and providing a justification is therefore crucial for appropriate interpretation of study results in making well-informed health policy decisions. In fact, governing bodies ^9,75,91 often suggest conducting analyses from several perspectives (eg, payer and societal) to cover a range of audiences while clearly identifying the costs and outcomes that comprise each perspective studied. We identified only 10 of our 52 studies (19%) evaluated more than 1 perspective.

Another important aspect to consider is the appropriate interpretation of study results. Over half of the studies (21 of 44, 51%) identified in this review that evaluated only costs reported interventions to be cost-effective. For example, 1 paper (cited 56 times) compared 3 general anesthetic techniques for elective knee surgery and concluded that sevoflurane in oxygen/nitrous oxide was the most cost-effective intervention; however, the authors only compared costs among interventions (no health effect comparison). ⁴² The use of the term cost-effective is considered inappropriate for partial economic evaluations, as no incremental analysis of cost and effect has been conducted. It provides only an estimate of which intervention is more or less costly and not whether the intervention provides a better value for health benefit. Authors should therefore be mindful of the terminology used when interpreting results to avoid confusing and misleading language. While we should discourage clinicians from basing clinical decisions off of data from partial economic evaluations, it also raises the notion that we need to further promote the publication of full economic evaluations.

Current guidelines recommend reporting the ICER, as this measure summarizes the additional cost to achieve an additional unit of effect. An important concept that is often misunderstood is the difference between an average cost-effectiveness ratio (ACER) and an incremental cost-effectiveness ratio (ICER), which answer very different questions. ⁴³ For example, 1 study from this review (cited 19 times) reported the average cost of microfracture was €178 per 1-point improvement on the visual analog scale (VAS) over 5 years (€ = 5150 / VAS = 29) compared with the average cost of €534 per 1-point improvement on the VAS over 5 years (€ = 14,941 / VAS = 28) for autologous chondrocyte implantation in treating focal cartilage defects and thus concluded that microfracture was more cost-effective. ¹ Comparatively, the ICER calculates the ratio between the difference in cost over the difference in effect between interventions. Reporting the ICER allows a more appropriate interpretation for decision makers to evaluate if an additional €9791 per 1-point improvement on the VAS over 5 years for autologous chondrocyte implantation is worth health care dollar spending. Although most papers in this review appropriately reported an ICER, clinicians and researchers need to be aware of the important difference when designing studies and critically appraising articles, as it can have important implications for policy decision-making.

There are limitations for this study. There is a level of subjectivity with scoring the QHES tool. To ensure consistency in quality scoring for all reviewers, we asked team members to supplement their scoring criteria for the QHES using the elaborated descriptors provided by Marshall et al ⁶⁸ as well as a priori modifications to the criteria (Supplemental Table 2, available online). Moreover, we pilot tested the QHES tool on 5 studies before completing subsequent full-text reviews. As a result, we showed consistency between reviewers through excellent interrater reliability scores (Table 3). Also, there are limitations with the scoring of the QHES tool. For example, studies may have lost points for questions on the QHES because of lack of clarity in their descriptions or because information was omitted from the article despite having sound research methodology. Finally, although the QHES tool is valid and reliable, it measures internal validity. The results of the QHES tool do not extend to the external validity (ie, generalizability) of studies. Therefore, a study that is internally valid (ie, high QHES score) may not necessarily be clinically relevant nor applicable, which should be considered by decision makers using this information.

Conclusion

The future of policy decision-making in health care resource allocation relies heavily on the quality of published research. While health economic evaluations are generally of high quality in orthopaedic sports medicine, the number of published studies in the field remains quite low. Also, half of the studies do not perform a full economic evaluation, which is necessary to draw appropriate conclusions regarding the cost-effectiveness of interventions. Promoting the publication of full economic evaluations is therefore necessary. Researchers should also consider the importance of research methodology to further improve study quality, particularly relating to study perspective, statistical uncertainty, costing, time horizon, discounting, model development, and discussion of limitations and biases. Overall study quality may be improved by following instruments such as the QHES and national or international guidelines for economic evaluation. ^9,75,91