Analysis of methodological deficiencies of studies reporting surgical outcome following cemented total-joint arthroplasty of trapezio-metacarpal joint of the thumb

Abstract

Cemented total-joint arthroplasty has been increasingly used in the treatment of end stage arthritis of the thumb trapeziometacarpal joint. Evidence supporting its use in the treatment of trapeziometacarpal disorders in the literature is very limited. Most hand surgeons agree that there are concerns about the methodological quality of the limited literature available. In this study, we analysed the methodological quality of the outcome studies on cemented total-joint arthroplasty of the thumb. We included all the outcome studies published in the English literature on cemented total-joint arthroplasty of the trapeziometacarpal joint of thumb. We analysed these studies for methodological deficiencies and quality of outcome reporting based on the recommendations given by Coleman et al. Our study revealed that there were no uniform standards of outcome reporting. The mean Coleman score for the studies dealing with cemented total-joint arthroplasty of the thumb was 42.9. Major deficiencies were identified in areas like subject selection criteria (0/15), type of study (5.7/15), description of surgical procedure (3.7/5), description of the rehabilitation protocol (2/10), outcome measures (4.4/10) and outcome assessment (3.9/15). The methodological quality of the studies published within the last 10 years (49.9±9.7) was found to be slightly better than the studies published over 10 years ago (39.7±7.8). Our study highlights the need for more evidence in the form of randomised controlled prospective studies conducted with good methodological quality, comparing the cemented total-joint arthroplasty of the thumb to other procedures available for the treatment of disorders of the thumb. Further, to improve the standards of reporting, journal editors should try to standardise the outcome of the reporting by following the surgical procedures on the thumb.

Introduction

Cemented total-joint arthroplasty has been increasingly used by hand surgeons in the treatment of osteoarthritis of the trapeziometacarpal joint of the thumb. De la Caffeniere first reported his experience in 1971 with cemented total-joint arthroplasty for the treatment of thumb trapeziometacarpal arthritis [8]. Since then, various authors have reported their experience with cemented total-joint arthroplasty of the thumb using various prosthesis. De la Smet [9], Braun [3, 4] and De la Caffieniere [8] have reported good to excellent results following cemented total-joint arthroplasty of the thumb. However, other authors like Van Cappelle [17] reported poor outcome following total-joint arthroplasty of thumb. Based on these reports, various manufacturers are still coming up with new implant designs. However, De la Caffiniere prosthesis continues to be the most commonly used cemented total-joint arthroplasty prosthesis. Most of the articles reporting the results of cemented total-joint arthroplasty are case series [1–5, 7–10, 12–15, 18]. In the absence of randomised controlled studies comparing cemented total-joint arthroplasty to other surgical procedures like ligament reconstruction and tendon interposition (LRTI), hand surgeons should rely on the reported case series for their treatment decisions. However, there have been no analyses about the quality of these outcome studies. Methodological deficiencies have been reported in articles reporting surgical outcome for various other conditions [6, 11, 16]. Coleman [6] reported major methodological deficiencies in studies reporting surgical outcome for patellar tendinopathy. Similar low-methodological qualities were identified by Jakobsen et al. [11] and Tallon et al. [16] for surgical outcome studies on cartilage repair and Achilles tendinopathy, respectively. Hence, we conducted this study to analyse the quality of all the studies which have reported outcome following cemented total-joint arthroplasty of the thumb and have suggested few recommendations to minimise this problem.

Materials and methods

We searched for all the studies reporting the outcome of total-cemented arthroplasty of trapeziometacarpal joint of thumb. We restricted our search to reports in English literature. We searched Medline, EMBASE, and CINAHL using OVID interface. We also searched other interfaces like EBSCO Academic premier search and Proquest as well as individual publishers’ websites like MD consult, Science Direct, Lippincott Wilkins and Williams (LWW), Blackwell synergy, Wiley Interscience. Search terms included thumb, trapezium, trapeziometacarpal, carpometacarpal. Within the search results obtained, we used subgroups such as arthroplasty, cemented arthroplasty, total-joint arthroplasty, replacement, implant arthroplasty, Braun, De la Caffiniere, GUEPAR and Steffee. Our search yielded 15 relevant articles which reported the outcome of cemented total-joint arthroplasty. We analysed these articles for the year of publication, quality of methodology, adequacy of outcome assessment, validity and reliability of outcome parameters. Quality of methodology was assessed by two independent observers using the Coleman methodology score. This scoring system was initially used for analysing the quality of studies reporting surgical procedures on Achilles tendon and patellar tendon. Coleman’s criteria takes into account not only the study design and methodology but also assess the quality of outcome reporting and hence we think it a very comprehensive criteria to comment on the overall quality of a study. Reliability was checked by Tallon and Coleman et al. and they reported a reproducibility of 99%. Its content validity is well established and its design validity was demonstrated by Jakobsen et al. in 2005 by revealing high correlation of the methodology score with the level of evidence classification system (r=0.668).

We standardised some of the components of this scoring system to make it suitable for the assessment of the thumb arthroplasty (Table 1). Through an initial run of the pilot project, the components with very high interobserver variability were identified. These components included description of surgical procedure and description of rehabilitation protocol. Discussion was held between the observers and research co-ordinator to reach a consensus. This problem was addressed by clearly defining each subgroup of these components (Table 1). For description of surgical procedure, if the technique used is similar to that described previously, then the authors can mention that their technique is similar to the previously described technique and has to reference it. However, they should mention the areas of variance between their technique and the previously described technique so that the reader can derive any inference regarding outcome based on the technique modifications if used. Each study was also analysed for the adequacy of the reported outcome (Table 2).

Table 1.

Defining components of Coleman methodology scores with high inter-observer variability

Component	Subgroup	Definition	Score
Description of surgical procedure given	Adequate	Implant type and size, incision, surgeon’s experience, cementing technique if cement was used, any mention about variations in technique	5 (only if at least five of these aspects were mentioned)
	Fair		3 (less than five mentioned or technique mentioned and referenced)
	Poor		0 (less than 2 components mentioned or technique not referenced)
Description of postoperative rehabilitation	Well described with >80% complying	Post operative type of immobilization, period of immobilization, need for physiotherapy, type of physiotherapy, duration of recovery	10 (all components mentioned along with compliance)
	Well described with 60– 80% complying		5 (few components mentioned with or without mention about compliance
	Protocol not reported or <60% complying		0 (no mention about any component or compliance)

Table 2.

Assessment chart for adequacy of reported outcome

Areas of concern and their respective scope/characteristics
Clinical	Subjective	Pain
		Satisfaction
		Functional ability
	Objective	Mobility
		Strength
	Comprehensive score	Includes subjective and objective outcome measures amiable for statistical analysis
Radiological	Any one standard method or few standard set of findings
Complications	Immediate, late
Survival analysis	Cumulative survival should be mentioned, end point should be clearly mentioned

Results

Fifteen articles were included in this study [1–5, 7–10, 12–15, 17, 18]. There were 612 patients enrolled in these studies. De la Caffiniere prosthesis was used in 11 studies [1, 2, 5, 8, 9, 12–15, 17, 18], Braun [3, 4] prosthesis was used in two studies, Steffee [10] was used in one study and the Mayo TMC prosthesis was used in one study [7]. The overall methodology score was 42.9±9.4. Studies published more than 10 years ago (score: 39.7±7.8) were found to score less than studies published within the last 10 years (Score: 49.4±9.7; Fig. 1). However, this difference was not found to be statistically significant (p=0.097). Most of the studies were found to be deficient in areas like number of patients studied, type of study, description of surgical procedure and rehabilitation protocol, outcome criteria, outcome assessment and subject selection.

Fig. 1.

Relationship between the year of reporting and Coleman methodological score

Number of patients studied

The mean number of patients studied was 40.8. The mean score obtained for number of patients was 5.7.

Type of study

The type of study is a very important determinant of the quality of the study. It is a well-known fact that retrospective studies represent a relatively poor level of scientific evidence and are just observations with a lot of bias involved. Hence the validity of conclusions derived from those studies is questionable and do not rate in the Coleman’s score (Table 3). There were no randomised controlled studies or prospective comparative studies dealing with total cemented joint arthroplasty of thumb. Most of the articles did not clearly mention the nature of study. The studies were considered retrospective if there was no clear documentation of the preoperative findings. Hence, there were seven prospective case studies [3–5, 9, 12, 13, 18] and eight retrospective studies [1, 2, 7, 8, 10, 14, 15, 17]. The mean Coleman methodology score for this component was 7.3.

Table 3.

Mean Coleman methodology score [16–18]

Components of the Coleman score (maximum score)	Individual components (score)	Mean (for both observers and for all studies analysed)
Study size (10)	>60 (10)	5.7
	41–60 (7)
	<20, not stated (0)
Mean duration of follow-up (5)	>24 (5)	3.2
	12–24 (2)
	<12, not stated or unclear (0)
Number of different surgical procedures included in each reported outcome (10)	One surgical procedure only (10)	10
	More than one surgical procedure, but >90% undergoing one procedure (7)
	Not stated, unclear, or <90% undergoing one procedure (0)
Type of study	Randomized control study (15)	4.7
	Prospective cohort study (10)
	Retrospective study (0)
Diagnostic certainty	In all (5)	5
	In >80%
	In <80%
Description of surgical procedure	Adequate (5)	3.7
	Fair (3)
	Inadequate (0)
Description of postoperative rehabilitation	Well described, >80% complying (10)	2.0
	Well described with 60–80% complying (5)
	Protocol reported or <60–80% complying (0)
Outcome measures	Outcome measures clearly defined (2)	4.4
	Timing of outcome assessment clearly stated (e.g., at best outcome after surgery or at follow-up) (2)
	Use of outcome criteria that has reported good reliability. (3)
	Use of outcome with good sensitivity (3)
Outcome assessment	Subject recruited (5)	3.9
	Investigator independent of surgeon (4)
	Written assessment (3)
	Completion of assessment by subjects themselves with minimal investigators assistance (3)
Selection process	Selection criteria reported and unbiased (5)	0
	Recruitment rate reported: >80% or<80% (5)
	Eligible subjects not included in the study satisfactorily accounted for or 100% recruitment (5)
Total		42.9

Mean duration of follow-up

Of the 15 studies analysed, 11 reported the average duration of follow-up. The remaining four articles reported only the range of follow-up period [3, 4, 7, 10]. The mean Coleman score for this component was 3.2.

Diagnostic certainty

Ten studies included patients with rheumatoid arthritis of the thumb [1–5, 7, 8, 10, 15, 18]. Only patients with inflammatory arthritis were included in one study [14] and only four studies were exclusively on osteoarthritis patients [9, 12, 13, 17]. Radiography was the only investigation used to establish the diagnosis in all studies. The mean Coleman methodology score was 10.

Description of surgical procedure

Minor variations in the surgical technique can significantly affect the outcome. However, there were major deficiencies in the reporting of the exact technique used. Only six articles described the surgical technique and variations in surgical techniques in detail [1, 3–5, 9, 10]. Six articles mentioned the procedure without describing the techniques. Three articles did not mention anything about the surgical techniques [7, 8, 17]. The average score for this component was 3.2.

Description of the rehabilitation protocol

Most studies undermined the importance of reporting this aspect of the procedure. Only studies reporting all the aspects outlined in Table 2 were considered adequate. Only one article was considered to adequately describe the postoperative rehabilitation by both observers [4]. Rehabilitation protocol was fairly well described in six articles [2, 3, 9, 10, 12, 14]. The mean methodology score for this component was 3.6.

Outcome criteria and outcome assessment methods

Outcome measures were not clearly defined in most of the studies and timing of outcome assessment was not mentioned in any of the analysed studies. Mean Coleman methodology score for outcome criteria was 4.4 and for outcome assessment was 3.9 (Table 3). There was great variation in the outcome variables used as well as the methods used to measure these variables. Adequacy of outcome reporting was assessed according to Table 3. Outcome variables like pain were not reported in four studies [3, 4, 7, 13]. Four of the studies used visual analogue scores (VAS) [5, 9, 14, 17] and seven used some form of grading system for the pain. Assessment of functional capabilities was documented in only five studies [9, 10, 12, 14, 17]. Patient satisfaction was not documented in 11 studies, good/excellent/fair/poor grading was used in two studies [13, 14] and VAS was used in the remaining two studies [5, 9]. There was great variability in the reporting of mobility after cemented total-joint arthroplasty of the thumb. All planes of movements were reported in only three [2, 15, 17] studies. Movements such as those recommended by Kapandji [5, 9, 13, 14] were reported in three studies and one article failed to mention movement [4], while eight articles did not mention the preoperative range of movement. Different studies used different instruments and different units to measure grip strength and pinch strength, whereas grip strength was not measured in six studies. Among the studies that reported grip strength, a Jamar dynamometer was used in six of the studies, a pneumatic gauge was used in one and a Martin virgiometer was used in one study, while one study made no mention about the instrument used. Similarly, pinch strength was not measured in two studies. A Preston pinch meter was used in two studies, a Martin virgiometer was used in one study, a manometer was used in one study, a Jamar dynamometer was used in three studies and the remaining eight studies made no mention of the type of instrument used to measure the pinch strength. Pinch strength was reported in kg in four studies, kgf in one study, kPa in one study and in cm Hg in one study. Radiological assessment was done in only 11 studies. Standard radiological criteria were used in only one study [18]. A general health questionnaire like SF36 was not used by any study. Disease specific questionnaires such as the DASH questionnaire was used in one study [9].

Survival analysis was reported by only five studies. Revision alone was used as the end point in two studies and both re-operation and radiographic failure were used as the end points in two studies. Outcome assessment by an independent observer was not mentioned in 11 studies. There was no documentation about the written assessment and degree of assistance from the observers to help the patients fill the questionnaire in any of the analysed studies. The overall score for outcome criteria and outcome assessment were 4.4 out of 15 and 3.9 out of 15, respectively.

Subject selection

Selection criteria were not clearly mentioned or were considered biased in almost all studies by both the observers. Hence, the overall Coleman methodology score for this component was 0.

Discussion

Our study found major methodological deficiencies in all the studies that have reported outcome following cemented total-joint arthroplasty. Deficiencies were found in areas like the sample size, type of study, description of surgical technique, description of rehabilitation protocol, outcome criteria, outcome assessment and subject selection. The overall Coleman methodological score was 42.9 out of 100. Our study shows that there has been a slight improvement in the methodological quality in the last 10 years as compared to studies published over 10 years ago, but that improvement is not statistically or clinically significant.

This study raises major concerns about the quality of studies reporting outcome following other hand procedures. Similar poor methodological qualities have been reported with studies dealing with procedures on other conditions for example, patellar tendinopathy [6], Achilles tendinopathy [16] and articular cartilage [11] defects. With increasing evidence of methodological deficiencies in orthopaedic literature being identified in various specialties like sports medicine, knee surgery and now hand surgery, the authors and journal editors need to make an attempt to reduce this problem in the future studies to be published. In this study we analysed the possible reasons for methodological deficiencies seen in each area and have given some suggestions to avoid these problems with methodology in the future.

Sample size

Some of the studies had as little as 15 patients in their study. This raises some real concerns about any conclusions made based on them. Power analysis and sample size calculation are not appropriate in case series studies. Hence, we think editors should take the initiative to mention in the instructions to the authors about the need to have a minimum number of patients for the manuscript to be accepted for review.

Type of study

Repeatedly publishing different authors’ experience in the form of retrospective case series and prospective studies only adds to the confusion of already available conflicting evidence. Hence, editors should discourage these types of non-comparative studies and should consider only well-conducted comparative studies. Similarly, the authors should refrain from reporting on the findings of their retrospective studies if that has already been done in the past by other authors and should try to initiate a prospective comparative study to confirm the findings of the retrospective studies before reporting on them.

Description of surgical procedure, description of rehabilitation protocol, outcome criteria, outcome assessment and subject

All authors should fill out the modified Coleman methodology sheet when submitting the manuscript. Those manuscripts with a low methodological score should be discouraged. Each journal should clearly define what is meant by an adequate description of surgical procedure and adequate description of rehabilitation protocol as outlined in Table 1. A consensus should be reached among journal editors to standardise the clinical and radiological outcome measures and also to standardise the ways to measure them. Editors should try to design a standardised outcome sheet for each region of the body for example thumb, wrist and shoulder which could be listed electronically on the journal website. In Table 4, we have illustrated a standard reporting protocol for thumb procedure using cemented total-joint arthroplasty as an example.

Table 4.

Standardising outcome measures illustrated with cemented total-joint arthroplasty of the thumb as an example

Standard reporting protocol for thumb procedure
Clinical measures (preoperative postoperative values and values on the normal and abnormal side should be always reported) and an explanation of how to measure the outcomes	Pain	VAS	0–10 score written assessment
	Satisfaction	Four grades	Excellent
			Good
			Fair
			poor
	Function	Ability to perform a set of standard functions for that region	Holding jar
			Turning door knob
			Buttoning shirts
			Holding key to open door
	Mobility	Standard set of movements (ideally movements in all planes or at least Kapandji’s recommended movements should be recorded)	Palmar flexion using goniometer
			Abduction and adduction using goniometer
			Opposition using Kapandji technique
	Strength	Using standard measures measured using standard device in standard units	Grip strength using Jamar dynamometer in kgs
			Key pinch and pulp pinch using a Preston pinch meter in kgs
Radiological outcome	Radiological outcomes	Standard set of parameters should be used	As described by Wachtl et al. [15]
Comprehensive score		Standard validated reliable disease or joint specific score should be reported	Buck Gramkco score or DASH or AUSCAN hand osteoarthritis index
Survival analysis		Same end point should be used	Revision of total-joint arthroplasty

It is clear from this study that there is very little evidence in the form of properly conducted randomised studies to support the use of cemented total-joint arthroplasty in trapeziometacarpal arthritis of the thumb. However, there are a few weaknesses in the conceptual framework of Coleman’s score that need to be addressed. Retrospective studies have traditionally received equal importance as prospective studies in scientific journals. Coleman’s score heavily penalises retrospective studies by providing no score for its retrospective nature and hence the maximum score a retrospective study can obtain is only 65. Furthermore, it penalises retrospective studies twice by including the sample size criteria, which further reduces the maximum score, thereby undermining the value of all retrospective studies. Nevertheless, authors reporting their experience in the future should consider the components of the Coleman methodological score while designing their study and should follow a standardised reporting protocol similar to the one shown in Table 4.