To MOON and Back: Lessons Learned and Experience Gained Along the Way

“That’s one small step for man, one giant leap for mankind.”

These words were immortalized by Neil Armstrong as he stepped out of the lunar module Eagle in the summer of 1969, marking the successful culmination of the Apollo space program and the fulfillment of the late President John F. Kennedy’s 1961 promise to land a man on the moon before the end of the decade. Given that only a few months had elapsed between Russian cosmonaut Yuri Gagarin’s maiden voyage into space and President Kennedy’s promise, some might have considered the president’s goal impossible. Today, some might refer to such a promise as a BHAG – a big, hairy audacious goal – a term coined by James Collins and Jerry Porras in their 1994 book Built to Last: Successful Habits of Visionary Companies. It took only a mere 9 years from the first human trip into space until Armstrong and Aldrin’s famous moon walk. Needless to say, there were countless hours poured into planning the Apollo 11 mission, and equally as many hurdles to overcome in order to make President Kennedy’s dream a reality. Much like Apollo 11, the story of MOON is one of vision, teamwork, perseverance, and (thankfully) success. And, also like Apollo 11, MOON’s story began long before its official “launch,” with more than a decade of thought and planning preceding the enrollment of our first study participant in 2002.

The journey to the Multicenter Orthopedic Outcomes Network (MOON) really began during my fellowship year at Cleveland Clinic in 1990, a full 12 years before we enrolled our first patient into what has now become the largest prospective anterior cruciate ligament reconstruction (ACLR) cohort with at least 80% follow-up in the world. The story starts with a much smaller prospective cohort study involving a mere 54 patients that had undergone acute (within three months of injury) ACLR at Cleveland Clinic during the year of my fellwoship.²³

At the time, it was well established that knees undergoing primary ACL repair had a high failure rate by their 4^th postoperative year, and we were beginning to realize that ACL reconstruction utilizing an autograft led to a more anatomically stable knee while also reducing the incidence of subsequent meniscus tears. In addition, chronic ACL deficiency seemed to be associated with both worse outcomes and the development of post-traumatic osteoarthritis.^1,6,13,18 Additionally during this time period, a growing body of literature suggested that ACL reconstruction improved knee stability and function, at least in the short-term (2 years after surgery).⁵ However, what remained largely unknown and at the forefront of many orthopedic sports medicine surgeons’ minds was, given the myriad of associated injuries which concomitantly occurred at the time of an ACL tear (e.g., meniscus tears, articular cartilage injuries, bone bruises, etc.), which types of injuries or treatments were predictive of clinically relevant outcomes. Furthermore, whether ACLR decreased the incidence of future post-traumatic osteoarthritis was unknown. Thus, beginning in the fall of 1990, we enrolled 54 patients with the goal of determining the association between bone bruises seen on MRI and meniscus and articular cartilage injuries. In addition, we hoped to follow this cohort for 10 years to shed some light on longer term outcomes. We were naïve to believe that we could determine which pre-operative and intra-operative variables (specifically the presence of bone bruising and/or meniscus or articular cartilage injuries) could be used to predict long-term outcomes in such a small dataset.^9,23

It did not take long for us to realize that there were likely a multitude of variables beyond intra-articular injuries that impacted both the development of post-traumatic osteoarthritis and long-term patient reported outcomes measures (PROMs). Furthermore, with the increased utilization of ACL reconstruction rather than the traditional repair, new and important questions arose which also needed answering such as what graft to use (autograft or allograft, hamstring or bone-patellar tendon-bone [BTB]) and how to decide. Consequently, it became quite obvious that our prospective cohort of 54 patients would not suffice to answer such complex questions.

It was on a hot summer day in 1991 that Dr. Jack Andrish (Cleveland Clinic, Cleveland, OH) and I found ourselves on an early morning bike ride in Sun Valley, Idaho, deep in the midst of a discussion revolving around the metaphorical elephant in the room – the fact that it would likely take hundreds, if not thousands, of ACLR patients in a meticulously-designed prospective cohort study to answer the myriad of multifaceted questions surrounding predictors or risk factors of ACLR outcomes as well as ACLR’s long-term impact. Naturally, once we conceded that there would be no way around this hurdle (short of performing dozens of parallel cohort studies or randomized trials), the discussion shifted to how we could possibly enroll, in a reasonable period of time, and follow thousands of patients that undergo a procedure that, at the time, was being performed less than 90,000 times nationwide annually.³ The answer, of course, was to collaborate between institutions and form a multicenter network.

In 1991, having finished my fellowship at Cleveland Clinic, I headed south to start my career as an assistant professor at Vanderbilt University (Nashville, TN), and, thus, the Vanderbilt Sports Medicine-Cleveland Clinic Foundation (VSM-CCF) ACLR Registry was born. Between the latter half of 1991 and 1998, we captured baseline demographics, PROMs (e.g., Lysholm scale), the type of ACLR, and the treatment of meniscus and articular cartilage injuries. We enrolled a total of 1201 ACLR patients between three surgeons (Drs. Jack Andrish, Richard Parker, and myself) with the aim of following these patients for 5 years. Around this time, we also developed a new relationship, this time with The Ohio State University (Dr. Christopher Kaeding), which was also assembling a similar database of its own ACLR cohort. This new partnership nearly doubled the size of our existing cohort, bringing the final number to 2286 ACLRs between three institutions over roughly a decade. However, we had no follow-up mechanism for these ACLR patients from which we captured baseline information from. Given the size of this new cohort, if we could achieve reasonable follow-up, we would be able to not only observe the natural history of an ACL reconstructed knee, but we would also be able to perform more complex multivariable regression analyses to identify preoperative and intra-operative variables that could predict outcomes.

Unlike our initial 52-patient cohort, this initial multicenter ACLR registry would require a significant investment of resources in order to achieve meaningful follow-up. As a result, we applied for a Prospective Clinical Research Grant from the Orthopaedic Research and Education Foundation (OREF), and, as is often the case when applying for funding, our first submission was rejected. However, we were successfully funded after revising our initial submission, thus providing us with the necessary financial resources to create a research infrastructure capable of achieving 70–80% follow-up of a subset (over 300 ACLR) that at least 5 years removed from surgery.

Our goal of achieving follow-up on a large ACLR cohort spanning multiple institutions came to fruition in 2005, when we were able to demonstrate the feasibility of our concept and provide a small sample of the wide range of information that could be gleaned from a large, well-designed prospective cohort.²⁴ In total, the three-institution ACLR registry produced 8 publications on a wide range of topics including predictors of intra-articular injuries, a comparison of intra-articular injury patterns between high school and recreational athletes, a description of common intra-articular findings in the multi-ligamentously injured knee, and outcomes of two medial meniscal repair techniques.^{2,7,8,12,19,20,22,24}

What made our initial registry unique was not only its size (at the time, it was one of the largest prospectively assembled ACLR cohorts in the United States), but also its overall design. In addition to investing an incredible amount of effort into collecting outcomes, we also recorded a variety of preoperative and intra-operative variables. Doing so allowed us to perform multivariable regression modeling to identify predictors of outcomes, a novel concept to sports medicine and orthopedics at the time. Thus, with our single cohort, we were able to answer multiple questions simultaneously. Possibly the most unique (perhaps controversial would be more accurate) facet of our initial multicenter cohort was the use of two PROMs that had very recently been designed and validated for use in a knee surgery population – the International Knee Documentation Committee Subjective Knee Form (IKDC-SKF) and the Knee injury and Osteoarthritis Outcome Score (KOOS).^11,15

Our use of the IKDC-SKF and the KOOS as primary endpoints represented a major paradigm shift in the way in which orthopedic research was conducted and critically evaluated. Consequently, our design – to use PROMs as a primary outcome without “objective” data to accompany it (such as radiograph measurements or arthrometer recordings) was met with considerable trepidation, which, in the fall of 2005 and spring of 2006, played out in a very public way.^10,28 Nevertheless, our methodology was ultimately accepted as a scientifically valid approach to ACLR follow-up, thereby paving the way for MOON, which, at the time, was already underway but in need of additional external financial support, to be seriously considered as a concept worth federal funding.

Despite the knowledge that we had gained from our three-center cohort, it seemed that ten new questions arose for each one that we were able to address. Again, we were humbled by the need for more information, and, again, we had to admit that there were many additional questions that our three-center ACLR cohort, which was designed to understand the long-term impact of intra-articular injuries and treatment sustained during ACL rupture, simply could not answer. Also complicating matters was the development of new methods for measuring outcomes, like the IKDC and the KOOS, which debuted after our enrollment was completed, thereby creating a gap in our baseline dataset and rendering us unable to compare baseline values to those collected postoperatively. Thus, we knew we needed to establish a new prospective cohort that was larger and derived from more patients than our three institutions could provide. Answering more questions would require more data, which meant more patients, more surgeons, and more institutions.

Ultimately, we decided to design our multicenter cohort to identify predictors of short, intermediate, and long-term ACLR outcomes which would include PROMs, ACL graft failure, and the development of post-traumatic osteoarthritis. We aimed to follow this cohort for 10 years, with short and intermediate follow up data collection at 2 and 6 years postop, as we knew that certain outcomes of interest, such as the onset of post-traumatic osteoarthritis, would take years rather than months to develop, particularly in a young, athletic population. Like our previous cohort, we opted to use PROMs as our primary outcome, as doing so would allow us to follow this large cohort of patients over time without necessitating in-person follow-up (and we had already invested the time and publications into justifying to the scientific community that PROMs were a valid outcome measure in orthopedic research). However, because we wanted to better understand the relationship between ACLR, preoperative and intra-operative risk factors, and the structural development of post-traumatic osteoarthritis, we created a smaller, nested cohort within the larger group that would be followed with longitudinal specialized radiographs, limited use of advanced imaging (MRI), and physical exams, in addition to the PROMs that could be completed remotely. Lastly, because we opted to include graft failure as an outcome of interest, we knew that our cohort needed to be large, as ACL revision had occurred only approximately 10% of the time in our previous cohort.

Three obvious questions then arose: 1) how many patients would we need to enroll and how many sites/surgeons would that require?; 2) how could we collect these preoperative and intra-operative variables reliably without overburdening participating surgeons, and in a way that was scientifically valid? and; 3) how could we afford this?

To estimate the size of our cohort, we looked to our least-likely outcome, which would almost certainly be ACL graft failure, and, given that we wanted to perform multivariable regression modeling with ACL graft failure as the dependent variable, our cohort needed to be large enough to accommodate roughly 15 graft failures per suspected predictor. Thus, to include 15 predictors in our model, we would need 225 ACL graft failures, and, given that we observed failure in approximately 10% of our previous cohort, our entire cohort would need to be on the order of 2,250 patients.

To enroll such a large number of patients in a timely fashion, we aimed to assemble a team of surgeons that, combined, would perform approximately 600 ACLRs per year, allowing us to complete enrollment in roughly 4 years. With the VSM-CCF ACLR Registry, we demonstrated successful collaboration between two institutions, and, with The Ohio State University coming into the fold shortly thereafter, we added a third high-volume ACLR center. Through various personal connections and professional relationships, we came to include Hospital for Special Surgery (Dr. Robert Marx), the University of Colorado (Drs. Eric McCarty, Michelle Wolcott, Armando Vidal), the University of Iowa (Drs. Ned Amendola, Brian Wolf), and Washington University in St. Louis (Drs. Rick Wright, Matthew Matava, Robert Brophy), as well as additional surgeons at Vanderbilt University (Dr. Warren Dunn, John Kuhn), the Cleveland Clinic (Dr. Morgan Jones), and The Ohio State University (Dr. David Flanigan), bringing our final tally to 17 surgeons at 7 institutions across the country.

With a team of experienced surgeons assembled, our sights turned to creating a system that would allow for accurate but rapid collection of relevant data that could be quickly and easily implemented at the 7 involved institutions. Given that the Apple iPhone would not debut for another 6 years, we initially utilized a paper-based system that involved data collection forms which could be scanned into a database using optical character recognition software (Teleform, OpenText, Waterloo, ON). Creating an electronic data capture system was not realistically possible at this point in time, especially given our limited resources. With the help and guidance of our late friend, Dr. Sandy Kirkley, we designed a series of standardized PROM questionnaires and surgeon data capture forms. The PROMs were completed by patients prior to their ACLR, and the participating surgeons captured all of the desired intra-operative variables at the time of the ACLR. In 2004–05, we developed an electronic surgeon capture system utilizing the Compaq iPAQ, a pocket PC that debuted in April of 2000 (Figure 1). Unfortunately, this proved to be an unreliable mechanism after using it for a little over a year, and was later abandoned (returning back to paper forms).

Figure 1.

Compaq iPAQ. (Courtesy of Hewlett-Packard, Inc, Palo Alto, CA; with permission.)

One challenge that we encountered while developing our data collection system was getting all of our surgeons to agree on what and how certain intra-operative variables would be classified. For example, among our intra-operative variables were meniscal tear location, depth, type (degenerative vs. acute), and management strategy (repair vs. meniscectomy). To demonstrate that our classification scheme, data capture form, and treatment decisions were reliable and reproducible, we performed an inter-rater agreement study with video recordings of 18 meniscal tears and asked participating surgeons to classify them accordingly.⁴ We then duplicated the same strategy to demonstrate agreement in classification of articular cartilage lesions, another important intra-operative variable collected by MOON surgeons.¹⁷

Yet another hurdle encountered was not how surgeons classified intra-operative variables, but rather how surgeons performed the ACLR itself. Of greatest concern was whether the participating surgeons placed their tibial and femoral tunnels in the same location, as large differences could certainly impact the result of the surgery. Again, we performed an inter-surgeon and intra-surgeon tunnel variability study utilizing both cadavers and surgical patients to demonstrate that the MOON surgeons placed their tunnels in the same locations, and that the individual surgeons varied little in their tunnel placement from patient to patient.^25,26

Then we needed to overcome possibly one of the most significant hurdles facing a group of 17 senior orthopedic surgeons - their personal preference toward BTB or hamstring autograft. Rather than try to convince those preferring BTB to perform all hamstring autografts or vice versa (which likely would have cost us multiple surgeons and numerous arguments), we performed a systematic review which showed no clinically relevant differences between the two choices.²¹

So, we had assembled a final team of 17 surgeons at 7 institutions and developed a system that would allow us to collect large amounts of relevant, reliable, and reproducible data in a (relatively) pain-free fashion. The last, and most important, question became how to pay for MOON, especially early on before it received any grant funding.

Consider that MOON carried with it an estimated annual cost of nearly $200,000 at Vanderbilt University alone, and it becomes easy to see why many thought MOON to be impossible. In total, MOON cost nearly $1.4 million to operate between 2001 and 2006, of which none came from National Institute of Arthritis and Musculoskeletal and Skin Disease (NAIMS) grants. The bulk of the early funding was given in the form of unrestricted gifts from Smith & Nephew ($450,000) and Aircast ($200,000), followed by a grant from the National Football League (NFL) Charities ($125,000) and the remainder of the OREF Prospective Clinical Research Grant ($149,000) that we had received to cover costs related to following up the VSM-CCF ACLR Registry. The final $450,000 was provided by Vanderbilt University Sports Medicine, generated by an internal tax on my partners (Drs. Eric McCarty and John Kuhn) and myself.

Finally, in January of 2002, we enrolled the first MOON patient, but the challenges did not stop there. We relied heavily on regular communication to discuss any new and/or ongoing concerns. These were not limited to concerns from the participating surgeons, but also included any issues encountered by the large research support staff that carried out the bulk of the patient enrollment and follow-up, as well as the numerous biostatisticians that helped us to make sense of so much data. This was typically done with a group conference call on the 2^nd Monday of every month – a call that has taken place nearly every month for the last 15+ years. We also made it a point to meet in person at least once per year.

By the end of 2005, 3 years into MOON, we had enrolled 2,340 patients and had followed up with 93% of the cohort by phone and had received 85% of our PROMs completed at 2 years. Between 2007–08, the group enrolled another 1200 ACLRs which were needed to investigate the impact of meniscus tears and treatment, as well as articular cartilage injuries and treatment on our multivariable models.

While assembling such a large cohort and achieving follow up rates well above 80% represented a huge success and validated our approach, doing so came with significant expense. These costs included a full-time staff of research assistants and coordinators, a project manager, and biostatistician and database support besides co-investigators with expertise in clinical studies. There were also additional costs associated with the nested cohort, which included training (and re-training) radiology personnel at three separate institutions, advanced imaging (MRI), radiographs, blinded evaluations by another surgeon and physical therapist, and more.

Additionally, with such a large (and growing) cohort, from which hundreds of variables were collected from each individual patient, we quickly came to a greater appreciation for database design and management. Currently, if one had a Space Shuttle (roughly 122 feet in length) for each data point in the MOON database, the shuttles would span the distance between the surfaces of the earth and the moon (10 million shuttles, covering 1.2 billion feet). What is more astounding is that even with so much data, there has been 98% surgeon compliance (completing intra-operative variable collection), 99% patient participation (agreeing to take part in the study), and 98% completion rates of baseline questionnaires.

By February of 2004, we had assembled sufficient preliminary data to apply for a National Institutes of Health (NIH) R01 grant - and we were rejected. We subsequently revised our proposal and resubmitted again in November of 2004, only to be rejected a second time. We then reorganized our specific aims and objectives and submitted a new NIH grant for the third time in early 2005, and, for the third time, we were rejected. However, we were close, and we followed the reviewers’ critiques and planned a resubmission. Despite having amassed considerable data which demonstrated the worth, validity, and feasibility of MOON, it took a fourth submission before we “struck oil,” and were successfully funded in 2006, a full 15 years after we came to the conclusion that MOON was an answer to our problem. It is worth mentioning again that, at the time (late 2004 to early 2006), the use of PROMs as a primary outcome was a novel concept and a paradigm shift that had not been received with unanimous support by any means.²⁸ To make our use of PROMs as a primary outcome even more controversial was the gamble that we took by adopting two PROM questionnaires that, at the time of MOON’s inception in 2002, were both less than 4 years old. Despite our early failed attempts to receive federal funding, MOON has successfully renewed its funding on 3 separate occasions to allow for collection of 2, 6, and 10-year follow up data.

The results of MOON and the impact that it has had and will continue to have on ACLR practices nationwide have been discussed elsewhere.¹⁶ Possibly of greater importance than the conclusions drawn from the MOON cohort data is the template that MOON has created for conducting high quality orthopedic research in the modern age. As a result, MOON has led to several spinoffs including the Multicenter ACL Revision Study (MARS) and MOON Shoulder. Additionally, almost all of the Meniscal Tear and Osteoarthritis Research (MeTeOR) sites were original MOON and/or MARS sites.^14,27

In closing, the journey to MOON has been a long one, full of ups and downs, successes and failures, and countless lessons learned, but it would have been impossible were it not for the incredible team of surgeons and supporting staff that came together in 2001 (and well before that) to become a part of something bigger than themselves for the betterment of their patients. Now, where will we go next?