Abstract
Background
Placebo-controlled trials are an important tool when assessing the efficacy of spinal surgical procedures. The most common spinal surgical procedure in older adults is decompression for lumbar spinal stenosis. Before conducting a placebo-surgical trial on decompression surgery, an investigation of patients’ willingness to participate in a placebo-controlled trial of decompression surgery and barriers to participation were explored.
Materials
An online survey.
Methods
Descriptive analyses of demographic and clinical data, and participants' willingness to participate in a placebo-surgical trial. Logistic regression was used to examine potential predictors of willingness to participate. Two independent researchers performed a coded framework analysis of patients’ barriers to participation.
Results
68 patients were invited and 63 participants completed the survey (91.3% response, mean (SD) age 69.5 (10.9) years, 52% females), 71% suffered from moderate to very severe pain. Ten participants (15.9%) were willing to participate in a placebo-controlled trial. Being married was associated with decreased odds of participating (OR: 0.2; 95% CI, 0.05 to 0.8; P = 0.03), while the main barriers were a lack of information about the procedure, reassurance of a positive outcome with participation, and concerns about the risks and benefits of placebo surgery.
Conclusions
A minority of patients with lumbar spinal stenosis were willing to participate in a placebo-controlled trial of surgery. The identified barriers indicate that educating eligible patients about: the need for placebo-surgical trials, the personal risks and benefits of participation, and the importance and potential benefits of placebo trials to others, may be crucial to ensure adequate recruitment into the placebo-controlled surgical trial. Conclusions should be read cautiously however, given the small sample size present in this study.
Keyword: Surgery
1. Background
Placebo-controlled trials are an invaluable tool for evaluating the efficacy of a surgical procedure [1, 2, 3]. The inclusion of placebo allows for more reliable blinding of participants and outcome assessors than other comparators (i.e. no treatment or conservative care), essential for minimising selection and performance bias [4, 5]. However, despite these benefits, placebo controls remain infrequently used to investigate the efficacy of surgical interventions. A recent systematic review identified only 53 randomised placebo-controlled trials of surgery published up until 2013 [6]. In approximately half (n = 27; 51%), surgery was not superior to placebo, questioning the continual use of the investigated procedure. The review also found that placebo surgical interventions were associated with less frequent and less severe adverse events than the investigated procedure. This contradicts popular belief among patients that an invasive placebo is associated with greater risk than the investigated intervention, a belief often presented as a barrier to trial participation [7, 8].
A recent review of 62 studies reporting data on participation in randomised trials of surgical interventions found that non-recruitment rates are as high as 92% for some trials [9]. The main reasons for non-participation included preference for one form of treatment over the other, dislike of randomisation and refusal to be in a non-treatment arm [9]. Patients were also less willing to participate in trials that included a placebo arm [9]. In another study that included 53 participants with Parkinson's disease, 35 (66%) indicated they would be willing to participate in a placebo controlled trial of surgery to treat their condition [7]. Those who were willing to participate were more educated, while those who were not, appeared to be focused on the perceived invasive and risky nature of the placebo procedure. In another study by Mittal et al, investigating reasons for or against participation into surgical randomised controlled trials, they found higher education levels were associated with decreased willingness to participate [10]. Mittal et al also found that altruism was a motivator towards patient participation in randomised controlled trials [10].
In order to determine the feasibility of a planned randomised placebo-controlled surgical trial of decompression for lumbar spinal stenosis, we performed a survey of the target population. Lumbar spinal stenosis is the most common indication for spinal surgery in older adults [11, 12], and currently has limited evidence suggesting it is more effective than conservative care [13, 14]. The aim of our study was to determine the proportion of potentially eligible patients who would agree to participate in a placebo-controlled trial, and any barriers to participation. We also investigated what (if any) demographic and/or clinical characteristics predict willingness to participate in order to inform recruitment strategies for the planned trial.
2. Methods
2.1. Recruitment procedure
Consecutive eligible participants were recruited from two private surgical practices in New South Wales and Victoria, Australia between November 2015 and August 2016. Following a clinical and radiologic assessment to determine that a patient with central lumbar spinal canal stenosis was suitable for surgical decompression, two study surgeons discussed the risk and benefits of the procedure. The surgeons then outlined the design of a randomised placebo-controlled surgical trial, and invited patients to participate in the survey. Those who consented were contacted by study researchers who discussed the study details and provided them with a link via email to the online survey. Consent to participate in the survey was obtained online. Ethics approval was granted by The University of Sydney Human Research Ethics Committee (#2015/557).
2.2. Participants
To be included, participants needed to be older than 18 years of age; diagnosed with central lumbar spinal canal stenosis (confirmed by imaging and clinical assessment); be considered by the treating surgeon to be suitable for lumbar decompression; have access to the internet via a computer, tablet or smartphone; and consent to participate. Participants were excluded if they had any known or suspected serious spinal pathology (e.g. fracture, metastatic, inflammatory or infective diseases, or widespread neurological disorder).
2.3. Survey
Consenting participants completed an online survey. Demographic data requested included age, gender, education level (completed year 10 or less, completed year 11 or 12, Technical and Further Education (TAFE)/University not completed, TAFE completed, completed higher University or higher), marital status (married, divorced/separated, single/never married, widowed), employment status (yes, no) and occupation (manager, technician or trade worker, clerical or administrative worker, machinery operator or driver, professional, community/personal service worker, sales worker, labourer).
Clinical data collected included imaging used for diagnosis (x-ray, computed tomography and/or magnetic resonance imaging (MRI)), use of medication, pain and disability. Pain severity was measured by asking “how much leg pain have you had during the last week?”, with participants given the options: none, very mild, mild, moderate, severe, or very severe. Disability was measured by asking participants to rate how pain interfered with normal life in the past week (not at all, a little bit, moderately, quite a bit, or extremely). Both the pain and disability questions were items from the Short-form (36) Health Survey questionnaire.
Willingness to participate in a placebo-controlled trial of decompression for lumbar spinal stenosis was investigated through a series of closed-ended questions with yes or no responses. Participants were asked if they understood when trials are needed, use of placebo surgery in controlled trials and in trials specifically for lumbar spinal stenosis. They were then asked if they would be willing to participate in a study investigating the effects of surgery for lumbar spinal stenosis, and if they would still be willing to participate if the trial randomised half the participants to receive placebo surgery (included a "maybe" response option). Finally they were asked if they were allocated to and received placebo surgery, whether or not they would be willing to wait for six months after their placebo surgery before receiving traditional surgery (also included a "maybe" response option). Depending upon their responses, they were asked a series of open-ended questions to explore their barriers to participation.
The survey questions were based upon previous research into patient attitudes to placebo trials7 15 and piloted on both researchers and surgeons. Data were captured using an online server (REDCap) and exported to STATA, V. 14 for statistical analysis [16].
2.4. Analysis
Descriptive analyses were conducted for all demographic and clinical data. Participants’ responses to the closed-ended questions were reported as frequencies and percentages. The association between acceptability of placebo-controlled trials and willingness to participate in a clinical trial or placebo-controlled trial and age, gender, marital status, employment status, educational level, pain and disability severity were each examined using logistic regression analyses. Confidence Intervals were set at 95% (95% CI) with outcomes reported as an Odds Ratio (OR). A p-value of less than 0.05 was used for indicating statistical significance.
Outcomes with multiple responses were re-categorised for the purposes of the analyses. Leg pain severity responses were re-categorised into three groups: no to mild pain, moderate pain, and severe to very severe pain; disability responses were dichotomised into not at all to moderately disabled, and quite a bit to extremely disabled; and education level was dichotomised into higher education completed or not completed. Age was also converted into a binary variable: ≤65 years and >65 years of age.
Two independent investigators (EM, GM) coded the responses to the open-ended questions using a qualitative coding system [17]. This allowed for a highly systematic abstraction of the data into categories and subcategories defined by decision rules and positive examples, that focused on relevance to the research question [17]. The two investigators' versions were discussed and combined into one final coding frame. A third independent researcher (MF), arbitrated. Cohen's Kappa was calculated for inter-rater agreement of all coded answers, with scores >0.81 considered very-high agreement [18].
3. Results
3.1. Participants characteristics
Of the 68 participants invited to participate in the study, 63 completed the survey (92.6%). Their demographic and clinical characteristics are shown in Table 1.
Table 1.
Participant characteristics, N = 63.
| Mean (SD) age, years | 69.5 (10.9) N (%) |
|---|---|
| Male | 30 (47.6) |
| Higher education completed | 30 (48.4) |
| Marital Status | |
| Single/Never Married | 5 (7.9) |
| Married | 45 (71.4) |
| Divorced/Separated | 5 (7.9) |
| Widowed | 8 (12.7) |
| Occupation∗ | |
| Not employed/Retired | 44 (71.0) |
| Manager | 4 (21) |
| Technician or trade worker | 2 (11) |
| Machinery operator or driver | 2 (11) |
| Professional | 7 (37) |
| Community/Personal service worker | 1 (5) |
| Sales worker | 2 (11) |
| Self-reported medication use for lumbar spinal stenosis | 29 (46.8) |
| Leg Pain levels∗ | |
| None | 2 (3.2) |
| Very mild | 7 (11.3) |
| Mild | 9 (14.5) |
| Moderate | 24 (38.7) |
| Severe | 14 (22.6) |
| Very severe | 6 (9.7) |
| Disability status∗ | |
| Not at all disabled | 6 (9.7) |
| A little bit disabled | 10 (16.1) |
| Moderately disabled | 15 (24.2) |
| Quite a bit disabled | 24 (38.7) |
| Extremely disabled | 7 (11.3) |
missing 1 participant data.
3.2. Patients’ perspectives
Table 2 presents the results of the participant's responses to the participation in placebo-controlled clinical surgical trials.
Table 2.
Numbers (percentages) of yes or maybe responses where relevant to the knowledge and willingness to participate questions.
| Question | N (%) |
|---|---|
| Do you understand that when the benefits of a particular treatment are uncertain a clinical study (trial) is needed? | |
| Yes | 59 (95.2) |
| Did you know that some clinical trials use placebo surgery as a control (comparison) to the traditional surgery? | |
| Yes | 28 (45.2) |
| Do you understand what placebo surgery for lumbar spinal stenosis is? | |
| Yes | 19 (30.6) |
| If you were invited to participate in a study investigating the effects of surgery for lumbar spinal stenosis, would you be willing to participate? | |
| Yes | 42 (67.7) |
| If this study would involve patients to be randomly divided equally into two groups (by chance), one group receiving surgery and the other group receiving a placebo surgery, would you still be willing to participate? | |
| Yes | 10/42 (23.8) |
| Maybe | 13/42 (30.9) |
| If you were allocated to a placebo surgery group, and later decided you would like to receive traditional surgery, would you be willing to wait for six months to do so? | |
| Yes | 12 (20.0) |
| Maybe | 13 (21.7) |
3.3. Associations with willingness to participate
Compared with those reporting lower levels of disability, people describing ‘quite a bit’ to ‘extreme’ levels of disability demonstrated more than three times the odds of being willing to participate in a clinical trial for lumbar spinal stenosis (OR: 3.6; 95% CI, 1.2 to 11.4; P = 0.03) (Table 3). No other independent predictors of willingness to participate in a clinical trial for lumbar spinal stenosis were identified. Compared with those who were not married, people who were married had lower odds of being willing to be randomised (one group receiving surgery and the other group receiving a placebo surgery) in a placebo-controlled trial for lumbar spinal stenosis (OR: 0.2; 95% CI, 0.05 to 0.8; P = 0.03) (Table 3). There were no other independent predictors of willingness to participate in a placebo-controlled trial of surgery for lumbar spinal stenosis.
Table 3.
Association with participants’ willingness to participate in a clinical trial and a randomised placebo-controlled trial for lumbar spinal stenosis.
| Willingness to participate in clinical trial for lumbar spinal stenosis∗ |
Willingness to be randomised to placebo surgery▲ |
|||||
|---|---|---|---|---|---|---|
| N | OR (95% CI) | p value | N | OR (95% CI) | p value | |
| Age | ||||||
| ≤65 | 13 | Reference | 13 | Reference | ||
| >65 | 49 | 0.6 (0.1–2.3) | 0.43 | 48 | 0.6 (0.05–2.6) | 0.47 |
| Employment status | ||||||
| Not employed | 44 | Reference | 43 | Reference | ||
| Employed | 17 | 1.9 (0.5–7.8) | 0.38 | 17 | 1.9 (0.5–7.8) | 0.38 |
| Marital status | ||||||
| Not married | 18 | Reference | 18 | Reference | ||
| Married | 44 | 0.7 (0.2–2.5) | 0.63 | 43 | 0.2 (0.05–0.8) | 0.03 |
| Gender | ||||||
| Male | 30 | Reference | 30 | Reference | ||
| Female | 32 | 0.8 (0.3–2.4) | 0.71 | 31 | 1.0 (0.2–3.7) | 0.96 |
| Education level | ||||||
| Higher education not completed | 31 | Reference | 30 | Reference | ||
| Higher education completed | 30 | 1.0 (0.3–2.8) | 0.92 | 30 | 0.4 (0.1–2.0) | 0.29 |
| Pain | 0.11 | 0.47 | ||||
| No to mild pain | 17 | Reference | 16 | Reference | ||
| Moderate pain | 24 | 2.7 (0.7–10) | 0.13 | 24 | 3.9 (0.4–37.5) | 0.23 |
| Severe to very severe pain | 20 | 4.5 (1.0–19.2) | 0.04 | 20 | 3.7 (0.4–37.4) | 0.26 |
| Disability | ||||||
| No at all to moderately disabled | 30 | Reference | 29 | Reference | ||
| Quite a bit to extremely disabled | 31 | 3.6 (1.2–11.4) | 0.03 | 31 | 2.5 (0.6–11.0) | 0.21 |
OR = Odds ratio; 95% CI = 95% Confidence Interval.
1 participant did not respond to this question.
2 participants did not respond to this question.
3.4. Participants’ reasons and beliefs behind participation
There were 215 responses to the open-ended questions. The inter-rater agreement of coding of these answers was Kappa value of 0.78 (p < 0.001) with an asymptotic standardized error of 0.03. The final summarised responses are shown in Table 4.
Table 4.
Summarised themes for responses to open-ended questions.
| N (%) | |
|---|---|
| Participants' general opinions towards placebo surgery in clinical trials | |
| Do not see any value/benefit in placebo surgery | 15 (27.8) |
| Have concerns about placebo surgery (e.g. ethical, costs) | 11 (20.4) |
| Have not formed an opinion yet (e.g. due to lack of information) | 12 (22.2) |
| Understand the importance and benefits of placebo surgery | 12 (22.2) |
| Other | 4 (7.4) |
| Reasons participants gave for responding no or maybe to participation in a randomised placebo-controlled surgical trial | |
| Have concerns about the risks and benefits of placebo surgery | 12 (27.9) |
| Have a lack of information or understanding about placebo surgery | 2 (4.7) |
| Have unmodifiable personal reasons against placebo surgery | 7 (16.3) |
| Have a clear preference for treatment | 6 (13.6) |
| Have financial and/or ethical concerns on the cost of the surgery | 5 (11.6) |
| Understand the importance of placebo surgery in clinical trials and/or want to contribute to research | 3 (7.0) |
| Other (disagree with any kind of surgery or answers not interpretable) | 8 (18.6) |
| Factors that would increase willingness to participate in a placebo-controlled trial | |
| More information about procedure and/or study | 6 (14.6) |
| Certainty and reassurance of good outcome | 8 (19.5) |
| Free choice of which treatment received | 5 (12.2) |
| Pain levels or functional status | 2 (4.9) |
| Advantages to participation | 4 (9.8) |
| Support of clinical evidence | 1 (2.4) |
| Nothing/unlikely to improve willingness | 11 (26.8) |
| Other | 4 (9.8) |
| Reasons participants gave against waiting 6 months for surgery (if allocated to placebo group) | |
| Do not see the value of randomised trials or only want surgery | 9 (22.5%) |
| Need reassurance about positive outcomes after placebo surgery | 5 (12.5%) |
| Level of pain and functional status | 9 (22.5%) |
| Require immediate treatment due to personal reasons (e.g. age, work) | 11 (27.5%) |
| Require more information | 1 (2.5%) |
| Other | 5 (12.5%) |
4. Discussion
The results from this survey provide insight into the barriers to, and willingness of participation in randomised placebo-controlled spinal surgical trials among patients with lumbar spinal stenosis. Almost all participants (95.2%) acknowledged the need to conduct clinical trials to evaluate treatments with unknown efficacy and 15.9% (n = 10) were also willing to be randomised to a placebo surgical group. The level of willingness decreased when asked if they would be happy to participate in a clinical trial of decompressive surgery.
When participants were asked if they understood what placebo surgery for lumbar spinal stenosis was, less than one third agreed. This may explain why one third of the participants who supported being involved in a clinical trial were uncertain if they would participate in a randomised placebo-controlled trial of surgery. Results from the open-ended questions support this, with approximately one third of the responders stating that with more information or reassurance of outcomes (with placebo) they would have an increased willingness to participate. Additionally, given only a minority of the responders stated that there were no reasons that would increase their willingness to participate, it is likely that the majority of participants were open to participation.
Responses to the open-ended questions suggest that uncertainty or unwillingness to participate in a randomised placebo controlled surgical trial stems from a concern of the risks associated with surgery (e.g. infection) and of having an unknown outcome (if they receive placebo). Given placebo surgical trials are safe [6], and placebo surgery itself is associated with fewer risks and complications in placebo trials of surgery [6], there may be potential for a greater willingness to participate amongst these participants if the correct information is provided. Considering the low proportion of participants expressing a willingness to be randomised in a placebo-controlled surgical trial, further education of potential participants would both reassure potential participants of their concerns for a randomised placebo-controlled surgical trial for lumbar spinal stenosis and improve recruitment rates.
The data analysis demonstrated that increased disability was the only factor associated with willingness to participate in a clinical trial of surgery, but it was not associated with willingness to be part of a randomised placebo-controlled trial. Increased education levels had no association with willingness to participate in a clinical trial or a randomised placebo-controlled trial. This was in contrast to previous research [9] which assessed patients' willingness to participate in a randomised placebo-controlled surgical trial, and reported higher-education levels were associated with an increased willingness to participate. In our survey, marital status was the only category with a statistically significantly association with participants’ willingness to participate in a randomised placebo-controlled surgical trial. There was no insight from the open ended-ended questions to explain this, and it is possible that this was a chance finding or type I error. It is also possible that being married reflects unmeasured behaviours and beliefs regarding risk. The lack of willingness present in this surveyed population may be explained by the lack of understanding among the participants of what placebo surgery is and the structure and importance of placebo trials and misunderstanding of the associated risks. Educating the community and potential participants of placebo-controlled surgical trials will be an important part of recruitment in future trials.
4.1. Study limitations
Limitations in this study included the small sample size. The small number of participants willing to participate (N = 10) was also a limitation, as it decreased the power of the logistic regression for that response. Interpretation of the responses to the open-ended question was also a limitation, as they can be subjective. We did limit this however, by having multiple reviewers, and by having them trained in how to interpret open-ended responses. We also had a third reviewer to resolve any disagreements.
5. Conclusion
This survey has provided modest insight into both patients' attitudes to a placebo-controlled surgical trial for lumbar spinal stenosis and their rationale for these beliefs. Although there was a relatively low level of willingness to participate amongst the participants surveyed, this was likely based upon the participants’ concerns about the safety and unknown treatment outcomes of placebo trials. It is therefore suggested placebo-controlled surgical trials consider educating potential participants to both the risks, and the likely benefits of these trials. This may assist in reassuring participants that placebo-surgical trials are relatively safe, associated with fewer complications, and are necessary to determine effectiveness of procedures with subjective outcomes. This approach may also improve recruitment into these trials, which remains a difficulty for researchers of clinical trials. Given the low sample size present in this study, all results should be interpreted with caution.
Declarations
Author contribution statement
David B Anderson: Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Ralph Mobbs: Performed the experiments; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Jillian Eyles: Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Eileen Meyer, Gustavo C Machado: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Gavin Davis, Ian A Harris, Rachelle Buchbinder: Conceived and designed the experiments, Contributed reagents, materials, analysis tools or data; Wrote the paper.
Manuela L Ferreira: Conceived and designed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Funding statement
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Competing interest statement
The authors declare no conflict of interest.
Additional information
No additional information is available for this paper.
References
- 1.Beard D.J., Rees J.L., Cook J.A. Arthroscopic subacromial decompression for subacromial shoulder pain (CSAW): a multicentre, pragmatic, parallel group, placebo-controlled, three-group, randomised surgical trial. Lancet. 2018;391:329–338. doi: 10.1016/S0140-6736(17)32457-1. 2017/11/25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Moseley J.B., O'Malley K., Petersen N.J. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N. Engl. J. Med. 2002;347:81–88. doi: 10.1056/NEJMoa013259. 2002/07/12. [DOI] [PubMed] [Google Scholar]
- 3.Macklin R. The ethical problems with sham surgery in clinical research. N. Engl. J. Med. 1999;341:992–996. doi: 10.1056/NEJM199909233411312. 1999/09/25. [DOI] [PubMed] [Google Scholar]
- 4.Brim R.L., Miller F.G. The potential benefit of the placebo effect in sham-controlled trials: implications for risk-benefit assessments and informed consent. J. Med. Ethics. 2013;39:703–707. doi: 10.1136/medethics-2012-101045. 2012/12/15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Machado G.C., Ferreira P.H., Yoo R.I. Surgical options for lumbar spinal stenosis. Cochrane Database Syst. Rev. 2016;11 doi: 10.1002/14651858.CD012421. 2016/11/02. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wartolowska K., Judge A., Hopewell S. Use of placebo controls in the evaluation of surgery: systematic review. BMJ. 2014;348:g3253. doi: 10.1136/bmj.g3253. 2014/05/23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Frank S.A., Wilson R., Holloway R.G. Ethics of sham surgery: perspective of patients. Mov. Disord. 2008;23:63–68. doi: 10.1002/mds.21775. 2007/10/27. [DOI] [PubMed] [Google Scholar]
- 8.Welton A.J., Vickers M.R., Cooper J.A. Is recruitment more difficult with a placebo arm in randomised controlled trials? A quasirandomised, interview based study. BMJ. 1999;318:1114–1117. doi: 10.1136/bmj.318.7191.1114. Clinical Trial Controlled Clinical Trial Research Support, Non-U.S. Gov't. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Abraham N.S., Young J.M., Solomon M.J. A systematic review of reasons for nonentry of eligible patients into surgical randomized controlled trials. Surgery. 2006;139:469–483. doi: 10.1016/j.surg.2005.08.014. [DOI] [PubMed] [Google Scholar]
- 10.Mittal R., Harris I.A., Adie S. Factors affecting patient participation in orthopaedic trials comparing surgery to non-surgical interventions. Contemp. Clin. Trials Commun. 2016;3:153–157. doi: 10.1016/j.conctc.2016.05.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Deyo R.A., Ciol M.A., Cherkin D.C. Lumbar spinal fusion. A cohort study of complications, reoperations, and resource use in the Medicare population. Spine. 1993;18:1463–1470. [PubMed] [Google Scholar]
- 12.Deyo R.A., Gray D.T., Kreuter W. United States trends in lumbar fusion surgery for degenerative conditions. Spine. 2005;30:1441–1445. doi: 10.1097/01.brs.0000166503.37969.8a. discussion 1446-1447. [DOI] [PubMed] [Google Scholar]
- 13.Ammendolia C., Stuber K.J., Rok E. Nonoperative treatment for lumbar spinal stenosis with neurogenic claudication. Cochrane Database Syst. Rev. 2013 doi: 10.1002/14651858.CD010712. 2013/09/03. [DOI] [PubMed] [Google Scholar]
- 14.Zaina F., Tomkins-Lane C., Carragee E. Surgical versus non-surgical treatment for lumbar spinal stenosis. Cochrane Database Syst. Rev. 2016 doi: 10.1002/14651858.CD010264.pub2. 2016/01/30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.S. Stata Statistical Software: Release 14. StataCorp LP; College Station, TX: 2015. [Google Scholar]
- 17.Schreier M. Sage Publications; 2012. Qualitative Content Analysis in Practice. [Google Scholar]
- 18.McHugh M.L. Interrater reliability: the kappa statistic. Biochem. Med. 2012;22:276–282. [PMC free article] [PubMed] [Google Scholar]