Abstract
The Multicenter Orthopaedic Outcomes Network (MOON) study of anterior cruciate ligament (ACL) reconstruction has achieved >80% follow-up for study subjects who were enrolled from 2002 to 2005; patient-reported outcome measures (PROMs) were reported at 2, 6, and 10 years through a carefully designed protocol that included surgeon involvement to encourage subjects to complete and return questionnaires. The process included emails and telephone calls from the central coordinating center, from research coordinators at each local institution, and lastly, from the subjects’ surgeons for those who were less inclined to complete the follow-up. In order to quantify the effect of site and surgeon involvement, the enrollment year of 2005 was monitored for the 10-year follow-up (n = 516 subjects). In contact efforts made by the coordinating center, 73.8% (381) of study subjects were reached by the central site coordinator, contact information was verified, and questionnaires were subsequently sent, completed, and returned. An additional 54 subjects (10.5% of the overall study population) returned the questionnaire after local study site involvement, indicating the importance of individual surgeon and local site involvement to improve follow-up rates in multicenter studies in orthopaedic surgery. Follow-up rates were higher when a specific individual (the surgeon or the research coordinator) was given the task of final follow-up.
Achieving high rates of long-term follow-up in orthopaedic studies has proven to be challenging, with follow-up of approximately 80% required for Level-I evidence1–5. Achieving that rate of follow-up is even more difficult with young patients, who generally relocate more frequently than older patients6–8. Age is a proven risk factor for greater loss to follow-up in anterior cruciate ligament (ACL) reconstruction6. Our multicenter study of ACL reconstruction has achieved >80% follow-up with patient-reported outcome measures (PROMs) at 2, 6, and 10 years for study subjects who were enrolled from 2002 to 2005. This was accomplished through a carefully designed protocol that included surgeon involvement as a last resort to encourage subjects to complete and return questionnaires9–15. The process included emails and telephone calls from the central coordinating center, from research coordinators at each local institution, and from the subjects’ surgeons for those who remained unresponsive and did not complete the follow-up. The purpose of this study was to determine the impact of each component of the overall strategy on follow-up rates so that other investigators can benefit from these strategies in order to increase their rates of long-term follow-up in future orthopaedic research.
Materials and Methods
Study subjects at each of the 7 study sites (Cleveland Clinic, Cleveland, OH; Hospital for Special Surgery, New York, NY; The Ohio State University, Columbus, OH; University of Colorado, Boulder, CO; University of Iowa, Iowa City, IA; Washington University, St. Louis, MO; and Vanderbilt University, Nashville, TN) who had ACL reconstruction in 2005 were eligible for the 10-year follow-up in this prospective cohort study. Any subjects who had subsequent total knee arthroplasty (TKA), were deceased, were incarcerated, withdrew from the study, or refused follow-up were excluded from further follow-up. Subjects who underwent TKA were not included because TKA is a reconstructive procedure with different outcome measures, and it is a clinical end point for ACL surgery patients, indicating severe osteoarthritis.
On the 10-year anniversary of the index ACL reconstruction surgery, subjects were contacted by a full-time research staff member from the central coordinating site (Vanderbilt University) via the telephone number and/or the email address that had been supplied at the most recent follow-up time point (i.e., baseline or the 2 or 6-year follow-up). One method of contact (i.e., telephone or email) was attempted each week until the subject had been reached or each method had been tried 4 times. If the subject was reached, his or her mailing address was confirmed or updated, subsequent surgery information was obtained, and the questionnaire was mailed to the confirmed address. A telephone call was made or an email was sent 1 week later to confirm the arrival of the questionnaire. The subject was then contacted once every 3 weeks and was reminded to complete the questionnaire. If the subject requested a replacement questionnaire, it was resent to his or her mailing address.
If a subject was not reached during the 4 attempts of each contact method, a questionnaire was mailed to their last confirmed address. Additional attempts to reach the subject were made once every 3 weeks, cycling through the different contact methods. If the subject was not reached after 4 months, the secondary contact that had been provided by the subject at a previous time point was called or emailed for updated subject contact information. Subjects were paid $20 after the completion of the survey.
If the central coordinating center was unable to have the subject complete the follow-up forms, a list of subjects who had not been reached (or who had been contacted but had not returned the questionnaire) was sent to the local sites. The local sites then attempted to contact these patients and encourage completion of the questionnaire. The individual who made the contact attempts varied from site to site and was either the local site research coordinator, the surgeon, or a combination of both. The local research personnel also were asked to check their medical record systems for updated contact information. The actions that were taken by these sites were documented. Each local site developed its own logistical flow for how to handle contacting the subjects. At some sites, the coordinator made a first pass prior to the surgeon, and at other sites, the coordinator was the sole contact person (calling on behalf of Dr. “XXX”). Sometimes, the surgeons took sole responsibility for contacting the subjects on their call list without the aid of the coordinator. Because of this mixed model, we reported it as contact by “surgeon/coordinator.” We are not aware of any cases in which both the local coordinator and the surgeon called in parallel.
The local sites received bimonthly lists starting in January of the year after the start of the follow-up year. This means that the site was asked to get involved at different points of the follow-up cycle depending on what month the subject had been enrolled and how long it took to initially send the questionnaire. Subjects who had been enrolled in January and were reached quickly may have been contacted every 3 weeks for several months. Subjects who were enrolled in December may have just had the questionnaire mailed to them before the site was asked to get involved.
The central coordinating site continued to attempt to contact these subjects in parallel with the local sites. An updated list of subjects was sent to the local sites every 2 months for a year after the tenth-year anniversary of the index ACL reconstruction surgery, or until 1 year after the last day within the follow-up year. Hence, subjects who underwent surgery early in the follow-up year could have been contacted for follow-up for nearly 2 years. If subjects were reached but were unwilling or unable to complete the PROMs, they were asked if they had undergone any subsequent surgery on either knee, and this information was recorded (Fig. 1).
Fig. 1.
Methods diagram.
Results
There were 553 patients enrolled in our study who had ACL reconstruction in 2005. Of these, 516 were eligible for the 10-year follow-up study. The 37 subjects who were excluded had undergone subsequent TKA (n = 11), were deceased (n = 4), were incarcerated (n = 3), had withdrawn from the study (n = 2), or had refused follow-up at some point over the 10-year period (n = 17; Fig. 2).
Fig. 2.
Study flow diagram.
The mean age of the eligible subjects at the 10-year follow-up was 38.2 years, and 54.5% of the subjects were men. At the conclusion of the follow-up period, 84.3% of the subjects (n = 435) had been reached by telephone for subsequent surgery information and had returned the questionnaire. Another 32 subjects had not returned the questionnaire but were successfully reached for subsequent surgery information. Therefore, in addition to the 84.3% who completed the PROMs and subsequent surgery information, another 6.2% (n = 32) completed only the subsequent surgery questions, for a total of 90.5% (467) of all eligible subjects who were successfully reached for subsequent surgery information. The number of subjects and the overall questionnaire return rate by site are shown in Table I. The mean age of the subjects who returned the questionnaire was 38.7 years compared with a mean age of 35.4 years for subjects who did not return the questionnaire (p = 0.005). Although this difference was found to be significant, the clinical relevance is questionable. Of subjects who returned the questionnaire, 47.6% were women. Conversely, 34.6% of the subjects who did not return the questionnaire were women (p = 0.03).
TABLE I.
Follow-up by Study Site
| Site | Patients (no.) | Questionnaires Returned (%) |
|---|---|---|
| 1 | 102 | 84.3 |
| 2 | 104 | 89.4 |
| 3 | 121 | 85.1 |
| 4 | 54 | 77.8 |
| 5 | 67 | 83.6 |
| 6 | 44 | 77.3 |
| 7 | 24 | 87.5 |
In the follow-up efforts that were made by the central coordinating site, 73.8% (381 subjects) were reached by the central site study coordinator, contact information was verified, and questionnaires were returned. The remaining 26.2% (135 subjects) who could not be reached or who were reached but did not return the questionnaire were placed on call lists. Some of the subjects who were reached but did not complete the PROMs were able to provide information on subsequent knee surgery. The call lists were sent to the local sites for assistance in contacting the subjects and encouraging questionnaire completion. While the sites attempted to contact these subjects, the central site study coordinator continued follow-up efforts in parallel. The local sites successfully contacted 89 of the 135 subjects on the call lists by telephone, email, or both. A total of 54 (10.5% of the overall study population) of the subjects who were included on the call lists returned the questionnaire. Another 32 subjects (6.2% of the overall study population) did not return the questionnaire but were successfully reached by either the central site study coordinator or the local sites for subsequent surgery information.
At the local site level, of the 89 subjects with whom telephone contact was successfully made and/or an email was sent, the person who made telephone contact and/or sent the emails was the surgeon for 56 subjects, the coordinator for 8 subjects, and both the surgeon and the coordinator for 25 subjects. Questionnaires were returned by 29 (52%) of the subjects who were successfully contacted by telephone and/or emailed by the surgeon, 6 (75%) of the subjects who were contacted by the coordinator, and 4 (16%) of the subjects who were contacted by both the surgeon and the coordinator. When a single individual (the surgeon or the coordinator) made telephone contact with and/or emailed the subjects, the follow-up rate was 55% (35 of 64) compared with 16% (4 of 25) when a specific individual was not identified (p < 0.001). Of the remaining 46 subjects on the call lists who were not successfully contacted by telephone or sent emails, 15 (33%) returned their questionnaires, even without being contacted.
Discussion
Loss to follow-up is common with cohort studies and often leads to bias16–21. To achieve high rates of follow-up in our multicenter study, we integrated local study site efforts only after the central site follow-up coordinator had exhausted his or her attempts to encourage the completion of the follow-up questionnaires. Through a very well-defined and specific protocol, we were able to receive follow-up in the form of completed PROMs from >80% of participants in the first 4 years of study enrollment (2002 to 2005) at 10 years. Overall, we achieved a questionnaire return rate of 84.3%, and the return rates from the individual local sites ranged from 77.3% to 89.4%. We also were able to collect subsequent surgery information on an additional 32 subjects by telephone, for a total of 467 subjects (90.5%) who were reached for subsequent surgery information.
The rate of follow-up at our site ranged from 77% to 89%, which demonstrates that there was not a vast difference among sites. Moving forward, we would emphasize that the more the surgeon touts the study at the time of enrollment (i.e., the importance of the study; wanting to know how the patient is doing, for good or for bad, after surgery; and how completing the follow-up questionnaire will help us to improve patient outcomes), the more successful the follow-up will be. It is essential to get “buy-in” from the patient early on. If the surgeon deems it important, then the patient will likely elevate its importance as well.
This longitudinal study opted for a priority of central follow-up over local site follow-up in order to maintain better study consistency. The majority of our local sites had research assistants and coordinators who changed annually over the course of the 2, 6, and 10-year follow-up periods. Conversely, the central site used only 2 individuals over the course of >15 years for subject follow-up. The subjects were alerted at the time of their enrollment that a central site (Vanderbilt University) would be contacting them at their 2, 6, and 10-year follow-up time points. Besides being familiar with the enrolling surgeon, these subjects were much more familiar with the follow-up coordinator at Vanderbilt than they would have been with a different local coordinator for each of their follow-up time points. With regard to study protocol on telephone call limits, there were no restrictions placed on the number of calls. At the start of the prospective cohort study in 2002, the institutional review board at each study site did not ask us to place a 3-call limit on follow-up.
The subjects who returned the questionnaire were slightly older than their counterparts who did not return the questionnaire, and there was a larger proportion of women in the group of subjects who returned the questionnaire. This latter finding is consistent with the work of Ramkumar et al., who reported that subjects who were men and non-White were at increased risk for loss to follow-up with PROMs at 2 years6.
It is also important to note that at the 2, 6, and 10-year follow-up time points of our cohort of ≥2,000 subjects who underwent ACL reconstruction, to the best of our knowledge, there were not any complaints by the study subjects to any of the participating institutions regarding being contacted for follow-up. Therefore, institutional review boards can consider that this strategy for contacting subjects for follow-up after surgery for research purposes is not overly bothersome or intrusive to subjects.
A possible limitation to the study was that most telephone calls were made during business hours, and subjects may have been less likely to answer if they were occupied with work. Subjects may have been more likely to answer during free time on the weekends, when calls were made less frequently. The study also did not include documentation of why subjects did not complete their PROMs, which might have provided additional insights. Whether there was a notable difference in subject response rate depending on whether it was the research coordinator or the surgeon who reached out to the subject was not recorded for each site. It is also possible that subjects who were unhappy with their surgical outcomes may have been less likely to complete the PROMs, which may have influenced the study results on some level.
Additional changes to this study may also be made for the year 2021, given new opportunities to utilize alternative technological platforms. Electronic PROMs in REDCap (Research Electronic Data Capture) software would likely be easier for subjects to fill out rather than the paper forms that must be mailed back. Using more convenient platforms may reduce the risk of the finding that young men have the highest risk rate of noncompliance. A secure application (app) for iPhone (Apple) or Android devices can be created that would automatically send notifications in the app to remind participants that it is time to follow up and complete the PROMs; additionally, the app could send reminders until the PROMs were completed. Younger people may be more likely to complete the PROMs in this format, and smartphone notifications would serve as a consistent reminder. Finally, text message reminders also can be utilized, and the addition of a link to the PROMs in a text message might encourage participants to complete the PROMs.
As detailed above, the efforts of the central site follow-up coordinator included multiple telephone calls and emails on a fixed schedule, verification of contact information, and regular reminders to complete the questionnaires. The questionnaires were returned by an additional 10.5% of subjects after the involvement of the local study sites, which included telephone calls and/or emails from surgeons and/or the research coordinators at these sites. This additional follow-up by the local sites, which was critical to achieving a total follow-up of >80%, was more successful when a single individual at the site was charged with the task of follow-up, which, in the majority of cases, was the surgeon who had cared for the subject.
In conclusion, in addition to our protocol as outlined above, we recommend assigning the final attempt at the time of follow-up to either the surgeon or a closely affiliated designee. We believe that contact by and engagement with the surgeon had a positive influence on the subjects and an important impact on our follow-up rate at 10 years following ACL reconstruction.
Acknowledgments
Source of Funding
Research reported in this publication was partially supported by the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under Award Number R01AR053684 (K.P.S.). The content is solely the responsibility of the authors and does not necessarily represent official views of the National Institutes of Health.
Footnotes
We wish to thank Dr. Jack Andrish, the research coordinators, the analysts, and the support staff from the Multicenter Orthopaedic Outcomes Network (MOON) sites, whose efforts that were related to regulatory requirements, data collection, subject follow-up, data quality control, analyses, and manuscript preparation have made this consortium successful. We also thank all of the subjects who generously enrolled and participated in this study.
The MOON Group includes Ned Amendola, MD; Robert H. Brophy, MD; Charles L. Cox, MD, MPH; Warren R. Dunn, MD, MPH; David C. Flanigan, MD; Christopher C. Kaeding, MD; Robert A. Magnussen, MD, MPH; Matthew J. Matava, MD; Eric C. McCarty, MD; Richard D. Parker, MD; Brian R. Wolf, MD; and Rick W. Wright, MD.
Disclosure: The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJS/G683).
References
- 1.Araki D, Kuroda R. Level 1 Evidence: Long-Term Clinical Results. In: Musahl V, Karlsson J, Hirschmann MT, Ayeni OR, Marx R, Koh JL, Nakamura N, editors. Basic Methods Handbook for Clinical Orthopaedic Research. A Practical Guide and Case Based Research Approach Springer; 2019. p 285–7. [Google Scholar]
- 2.Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am 2003. Jan;85(1):1–3. [PubMed] [Google Scholar]
- 3.Dettori J. Class or level of evidence: epidemiologic basis. Evid Based Spine Care J 2012. Aug;3(3):9–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Norvell DC. Study types and bias-Don’t judge a study by the abstract’s conclusion alone. Evid Based Spine Care J 2010. Aug;1(2):7–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O’Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer T, Varonen H, Vist GE, Williams JW Jr, Zaza S; GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ 2004. Jun 19;328(7454):1490. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ramkumar PN, Tariq MB, Amendola A, Andrish JT, Brophy RH, Dunn WR, Flanigan DC, Huston LJ, Jones MH, Kaeding CC, Kattan MW, Marx RG, Matava MJ, McCarty EC, Parker RD, Vidal AF, Wolcott ML, Wolf BR, Wright RW, Spindler KP; MOON Knee Group. Risk Factors for Loss to Follow-up in 3202 Patients at 2 Years After Anterior Cruciate Ligament Reconstruction: Implications for Identifying Health Disparities in the MOON Prospective Cohort Study. Am J Sports Med 2019. Nov;47(13):3173–80. Epub 2019 Oct 7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Madden K, Scott T, McKay P, Petrisor BA, Jeray KJ, Tanner SL, Bhandari M, Sprague S. Predicting and Preventing Loss to Follow-up of Adult Trauma Patients in Randomized Controlled Trials: An Example from the FLOW Trial. J Bone Joint Surg Am 2017. Jul 5;99(13):1086–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Booker CL, Harding S, Benzeval M. A systematic review of the effect of retention methods in population-based cohort studies. BMC Public Health 2011. Apr 19;11:249. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wright RW, Dunn WR, Amendola A, Andrish JT, Flanigan DC, Jones M, Kaeding CC, Marx RG, Matava MJ, McCarty EC, Parker RD, Vidal A, Wolcott M, Wolf BR, Spindler KP; MOON Cohort. Anterior cruciate ligament revision reconstruction: two-year results from the MOON cohort. J Knee Surg 2007. Oct;20(4):308–11. [DOI] [PubMed] [Google Scholar]
- 10.Hettrich CM, Dunn WR, Reinke EK, Spindler KP; MOON Group. The rate of subsequent surgery and predictors after anterior cruciate ligament reconstruction: two- and 6-year follow-up results from a multicenter cohort. Am J Sports Med 2013. Jul;41(7):1534–40. Epub 2013 May 30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Magnussen RA, Reinke EK, Huston LJ, Hewett TE, Spindler KP, Amendola A, Andrish JT, Brophy RH, Dunn WR, Flanigan DC, Jones MH, Kaeding CC, Marx RG, Matava MJ, Parker RD, Vidal AF, Wolcott ML, Wolf BR, Wright RW; MOON Knee Group. Effect of High-Grade Preoperative Knee Laxity on 6-Year Anterior Cruciate Ligament Reconstruction Outcomes. Am J Sports Med 2018. Oct;46(12):2865–72. Epub 2018 Sep 7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Westermann RW, Spindler KP, Huston LJ, Wolf BR; MOON Knee Group. Outcomes of Grade III Medial Collateral Ligament Injuries Treated Concurrently With Anterior Cruciate Ligament Reconstruction: A Multicenter Study. Arthroscopy 2019. May;35(5):1466–72. Epub 2019 Mar 14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Spindler KP, Huston LJ, Chagin KM, Kattan MW, Reinke EK, Amendola A, Andrish JT, Brophy RH, Cox CL, Dunn WR, Flanigan DC, Jones MH, Kaeding CC, Magnussen RA, Marx RG, Matava MJ, McCarty EC, Parker RD, Pedroza AD, Vidal AF, Wolcott ML, Wolf BR, Wright RW; MOON Knee Group. Ten-Year Outcomes and Risk Factors After Anterior Cruciate Ligament Reconstruction: A MOON Longitudinal Prospective Cohort Study. Am J Sports Med 2018. Mar;46(4):815–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kaeding CC, Pedroza AD, Reinke EK, Huston LJ, Hewett TE, Flanigan DC, Spindler KP; MOON Knee Group. Change in Anterior Cruciate Ligament Graft Choice and Outcomes Over Time. Arthroscopy 2017. Nov;33(11):2007–14. Epub 2017 Aug 26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Westermann RW, Marx RG, Spindler KP, Huston LJ, Amendola A, Andrish JT, Brophy RH, Dunn WR, Flanigan DC, Jones MH, Kaeding CC, Matava MJ, McCarty EC, Parker RD, Reinke EK, Vidal AF, Wolcott ML, Wolf BR; MOON Knee Group. No Difference Between Posterolateral Corner Repair and Reconstruction With Concurrent ACL Surgery: Results From a Prospective Multicenter Cohort. Orthop J Sports Med 2019. Jul 30;7(7):2325967119861062. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kristman V, Manno M, Côté P. Loss to follow-up in cohort studies: how much is too much? Eur J Epidemiol 2004;19(8):751–60. [DOI] [PubMed] [Google Scholar]
- 17.Somerson JS, Bartush KC, Shroff JB, Bhandari M, Zelle BA. Loss to follow-up in orthopaedic clinical trials: a systematic review. Int Orthop 2016. Nov;40(11):2213–9. Epub 2016 May 3. [DOI] [PubMed] [Google Scholar]
- 18.Howe CJ, Cole SR, Lau B, Napravnik S, Eron JJ Jr. Selection Bias Due to Loss to Follow Up in Cohort Studies. Epidemiology 2016. Jan;27(1):91–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Howe LD, Tilling K, Galobardes B, Lawlor DA. Loss to follow-up in cohort studies: bias in estimates of socioeconomic inequalities. Epidemiology 2013. Jan;24(1):1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Tariq MB, Vega JF, Westermann R, Jones M, Spindler KP. Arthroplasty studies with greater than 1000 participants: analysis of follow-up methods. Arthroplast Today 2019. May 7;5(2):243–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Murray DW, Britton AR, Bulstrode CJ. Loss to follow-up matters. J Bone Joint Surg Br 1997. Mar;79(2):254–7. [DOI] [PubMed] [Google Scholar]