Learning from an Unsuccessful Study Idea: Reflection and Application of Innovative Techniques to Prevent Future Failures

Developing an impactful study design can be a daunting task for researchers of all skill levels. Oftentimes, a failed study can be attributed to an ineffective study design rather than an inferior theme or an uninspiring area of focus. In major medical journals such as The Lancet, BMJ, and Annals of Internal Medicine, only about 6% of submitted papers are accepted, whereas 70% are rejected outright, and 24% are rejected following a peer review.¹ Furthermore, of the studies presented at large conferences hosted by organizations such as the American Academy of Orthopaedic Surgeons, or the American Society for Surgery of the Hand, only about 50% are published in peer-reviewed journals.^2,3 The “publish or perish” school of thought has permeated through academia since its conception in the early twentieth century.^4,5 Consequently, many researchers emphasize quantity over quality of research proposals. As a result, these hastily created study designs often do not withstand scientific scrutiny, or have little to no practical implications.⁶

In 2016, our research team, the Michigan Center for Hand Outcomes and Innovation Research, published 41 scientific papers in 11 peer-reviewed journals, with the goal to contribute knowledge to improve the quality of medical care for all patients, but particularly those with hand disorders or injuries. However, we often find it challenging to create a well-organized and impactful study. Consider one of our failed proposals in which we aimed to reveal the economic burden of the treatment of Dupuytren contracture. Our initial idea was to use insurance claims from a nationally representative database to determine the trends in the national total cost of treatment for Dupuytren contractures over time. The idea was put forward because a new treatment option, collagenase injections, has gained popularity in recent years and we wanted to consider the impact of the changing treatment landscape on the national cost of treatment for the disease.^7-9 However, upon taking preliminary steps to begin the study, we faced several critical problems resulting in an inability to complete the original proposed study.

This type of scenario is not uncommon when undertaking a new study. Nonetheless, these “failures” should not simply be disregarded, but should be critically examined to impart new insight on why a study was not successful. To understand what makes an effective research proposal, one must also have a deep understanding of the sequence of mistakes that causes an ineffective research proposal. We will present an example of a previous failed study from our laboratory to share our analyses using two innovative techniques, conceptual models and root cause analysis (RCA), to inform researchers how to critically review and learn from a failed research idea. Although the methods described are not new to the fields of research or medicine, the novel application of these strategies on a broad scale is influential in avoiding the time and expenses wasted with future failures.

Conceptual Models

A conceptual model is a researcher’s visual representation of the research questions he or she is tackling and can be used to give direction to a study. They can also be adapted to make experimental flow diagrams that show possible courses through the study that can be used to critically assess the approach to the research problem. ^10,11 Conceptual models are used in a variety of situations in addition to scientific studies, such as news articles and business meetings. A traditional conceptual model shows connections between broad concepts and the variables affecting them, illustrating relationships between all aspects of a situation.

In addition to illuminating interactions and possible problem areas along with providing a visual overview of an issue, conceptual models can also provide context for understanding the findings of a study.¹² Conceptual models are particularly effective when trying to communicate findings with other colleagues or team members who can then combine various ideas and modify the framework to improve the overall study design.¹² On the other hand, conceptual models may sometimes require too many interconnected variables or become too detailed becoming overcomplicated and ineffective. Conversely, if the researcher errs on the side of simplicity, important variables may be left out, creating a model that lacks accuracy and can lead observers to draw incorrect conclusions from a lack of information.

Our initial conceptual model for our failed proposal on the national costs of treatment for Dupuytren contractures is depicted in Figure 1. For the purposes of visualizing the effectiveness of our study, we chose to create our conceptual model as an experimental flow diagram. There were two treatment paths that we wanted to investigate: open fasciectomy, which is the surgical option, and the newer collagenase injections.^13-15 Therefore, we began our conceptual model by showing these two treatment options. The next step was to list out possible results from our observations. From our initial review of the literature, we already knew that collagenase injections are cheaper than the open fasciectomy procedure.^8,9,16,17

Figure 1.

Conceptual model created for the original study idea.

Taking into consideration these variables and the previous knowledge, we began the prospective part of our experimental flow design. Because we were examining national trends in total cost of treatment, there were only three possible outcomes that we could find after analyzing the data; the national cost of treatment would have increased, decreased, or stayed the same. We then needed to consider the conclusions we would draw from each of these cases. In the case that the national cost of treatment for Dupuytren contractures increased over the time period analyzed in the study, we would encourage physicians to treat as many patients as possible with collagenase injections in hope of bringing the total cost down in future years to prevent a continuing upward trend. In the case that the cost had stayed the same, we would still encourage physicians to use injections in hopes that we could actually lower total costs. Finally, in the case that there was a decreasing trend, we would, again, encourage the use of injections in hopes of lowering the total cost further. This exercise highlighted the fact that every outcome led to the same conclusion, leading to a study that would have no impact because, regardless of our findings, there would be no practical application. Furthermore, the creation of the conceptual model clearly unveiled this flaw, as each possible path through the study converged upon the same conclusion.

Root Cause Analysis

RCA is a tool that can be used during an investigation of past events to identify solutions rather than ascertain blame.^18-21 Although historically used in the fields of psychology and systems engineering to analyze problems such as national disasters, and failing business models, RCA was introduced into medicine in the mid-1990s to provide a framework for retrospective analysis of errors and problems.^22,23 For example, safety errors, such as accidental administration of fast-acting insulin instead of basal insulin, are common problems addressed in medical practice using RCA.²³ There are 3 distinct steps to conduct an RCA: 1) define the problem, 2) break the problem down using a visual map to analyze the root cause, and 3) formulate an action plan to help solve the problem.²⁰ One helpful tool for creating the visual method is 5-whys method. As the name suggests, the 5-whys method requires investigators to assess cause and effect relationships by asking the question “why” at least five times.

RCA is an effective strategy to identify the underlying cause of a problem. It employs a simple, systematic approach for investigators to visualize the issue and to link concepts from different stages during the progression of the study. Often, the root of a problem is not a direct result of a single action, but rather a compilation of many. Visual maps present the interrelationships of each cause and solution, subsequently facilitating the formulation of an efficient, well-organized action plan. Troubles with RCA may arise if one encounters difficulties formulating corrective actions after identifying a problem.²³ To make a sustainable action plan, authors must first review the existing process and create a plan that not only prevents future problems, but also considers the efficiency of proposed changes. In addition, the actual event or problem on which the RCA was performed cannot be changed with this analysis because RCA draws on past problems as lessons to improve or solve future problems.

We applied RCA to the aforementioned failed proposal on Dupuytren contracture (Figure 2). We began with the initial problem that the “study failed to draw an impactful conclusion.” We subsequently asked “Why?” five times to determine the root cause. The first iteration of “Why?” showed us that our study was not impactful because the possible conclusions will not differ depending on the results. In the second iteration, we determined that this was because conclusions from previous studies already provide more insight into the issue. Previous publications already determined that collagenase injections are a less costly form of treatment for Dupuytren contractures than the surgical procedure. The third “Why?” led us to uncover that we chose an inappropriate aim for our study design. We determined that there were two causes for the weakness in study design; first, the study question lacked depth because it only examined overall trends and disregarded all subsets of treatment and patient characteristics, and, second, there was insufficient detail in the data to analyze certain characteristics. Originally, we intended to use International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes compiled in an administrative claims dataset. However, there are a number of limitations to this coding scheme including an inability to identify clinical details such as the severity of the Dupuytren contracture, the number of the fingers affected, and even the affected hand. As a result, we concluded that the root cause of our original problem of having a study proposal with no impact was lack of knowledge about weaknesses in the dataset we intended to use.

Figure 2.

Visual map for root cause analysis done on failed Dupuytren study idea.

Following the creation of our visual map and pinpointing of the root cause of our problem, we then created an action plan to prevent similar failed studies in the future The revised study proposal needed to compensate for weaknesses in the dataset, and, in light of the insight we gained from our conceptual model, these compensations would also need to alter the study proposal in such a way that it would lead to multiple conclusions, lending our study impact. To do so we would better educate the team on limitations to administrative data.

Revision of the Study Proposal

Using a conceptual model and RCA to analyze our failed study, we highlighted three major flaws that needed to be addressed to re-work our study design. First, we needed to compensate for the lack of clinical detail in the dataset. Second, we needed to analyze a question with more depth. And third, our new research question had to lead to multiple possible conclusions depending on the outcome.

The lack of clinical data was the most debilitating weakness of our study. This limited both what questions we could investigate and the variety of variables we could analyze to add depth. In some cases, the necessary data are simply not available in any dataset. To prevent this issue from arising in most cases, study ideas should be constructed with known datasets in mind. However, in this case, an alternative dataset does exist. We decided not to use the ICD-9-CM, but rather the tenth revision (ICD-10-CM), which has specific codes that provide information on severity and anatomical information. However, ICD-10-CM only has information from October of 2015 so the study will have to be performed in the near future once there is more data. This solved our first critical flaw of a lack of clinical detail in the dataset.

To add depth to our study, we examined some common variables that are analyzed in observational studies, instead of simply exploring broad overall trends in cost. These included patient age, the number and nature of comorbidities, the recurrence rate, and severity of the contracture. Among these variables, severity is the most subjective in an administrative dataset. Additionally, comorbidities can be difficult to track because they only appear in the data if the patient seeks medical intervention for the comorbidity. Missing or incorrect diagnoses could damage the quality of our study. Accurate data on patient age are readily obtainable; however, for the purposes of our study on Dupuytren contractures, it would not lead to particularly impactful conclusions. Older patients are more likely to require multiple treatments, have more comorbidities, or require more drastic treatment, all of which can be concluded intuitively. The recurrence rate of individual treatments, in other words, how many patients who originally got an injection needed more treatments in the future compared to patients who got open fasciectomies, was an unknown factor that had not been researched before and the necessary data were not obtainable through the dataset.

Analyzing the recurrence rate of treatment also led to an impactful study by reaching multiple possible conclusions as can be seen in Figure 3. Rather than researching solely the overall national trends in costs for Dupuytren contracture treatments, we decided to examine the total cost for a patient initially treated with a collagenase injection, taking into account the average number of times these patients required subsequent injections of open fasciectomies. We would then compare that cost to the total treatment cost for a patient who was initially treated with an open fasciectomy including the average number of subsequent procedures and injections. If the total average cost per patient including subsequent procedures for recurrence was higher for patients who initially received injections, we would encourage physicians to consider open fasciectomy more often as an initial procedure. Conversely, if the same average cost per patient was higher for those who received open fasciectomy initially was higher, we would encourage physicians to consider injections more often. Finally, if there was no significant difference between the costs, we would encourage physicians not to consider the cost of treatment when making their decision on which procedure to use.

Figure 3.

Conceptual model for revised study idea.

There are still a number of weaknesses in this study design. One of the most relevant is that there are many factors that go into deciding which treatment path to follow. There may be confounding variables in which patients who initially receive one treatment or the other are more likely to have initially presented with a more severe contracture, or a number of other variables, leading to a higher likelihood of recurrence. These questions could be answered with further research following our initial study. Finally, the ICD-10 coding system has only been in effect since October, 2015, limiting the amount of relevant data available for analysis.

Conclusion

When used as described and in conjunction with each other, RCA and conceptual models can offset some of their individual limitations. Research-informed design is an emerging school of thought in which prototyping and existing resources are applied to create a final design.²⁴ Our use of conceptual models and RCA falls into this category. Although conceptual frameworks are traditionally considered useful for investigators to review overarching themes and relationships between ideas, it has value to develop and assess solutions to issues identified from RCA. Conceptual models are also applicable for reviewing studies from a broader, “big picture” perspective. Conversely, RCA is beneficial for examining specific, individual problems, assisting investigators in finding practical solutions to prevent similar, future problems. Additionally, using both of these tools together opens up opportunities for analysis at multiple stages within the study because conceptual models are usually used at the beginning of a study whereas RCA is performed after an event (Table 1). Combining the principles behind these contrasting methods, researchers can identify and address problems from a failed study attempt to create an altered study proposal that is more likely to succeed.

Table 1.

Summary of Steps to Review and Learn from a Failed Research Idea

Step	Tips
1. Create a conceptual model	Use the model to depict all possible conclusions before starting your analysis Consider the implications of your findings on clinical practice Share your model with colleagues who are unfamiliar with the idea to ensure clarity and impact of the study proposal
2. Conduct a Root Cause Analysis	Perform a 5-why’s questioning technique to delineate the reasoning behind your failed proposal Consider factors on multiple levels (e.g. limiting factors of your research team as well as limitations of the study design) Create and implement an action plan that will prevent the recurrence of a similar failure
3. Revise	Revise existing ideas to be impactful based on the findings of creating a conceptual model and/or performing a root cause analysis Revising an existing idea saves time and effort because many of the resources used previously remain relevant Create a revised conceptual model to showcase all new possible conclusions and relevance If necessary, investigate other routes that effectively overcome the fatal biases of the previous proposal
4. Reflect	Take note of the reasons behind your failed proposal Educate your team members to prevent additional waste of time and resources

e.g.; for example