An increasing number of qualitative research papers in oncology and palliative care: does it mean a thorough development of the methodology of research?

Abstract

Background

In the second half of the nineties, a scientific debate about the usefulness of qualitative research in medicine began in the main medical journals as well as the amount of "qualitative" papers published on peer reviewed journals has noticeably increased during these last years. Nevertheless the label of qualitative methodology has been assigned to an heterogeneous collection of studies. Some of them show a complete awareness of the specificity of this kind of research, while others are still largely influenced by the quantitative paradigm prevailing in the medical field. The concern with the rigour and credibility of qualitative methods has lead to the development of a number of checklist for assessing qualitative research. The purposes of this review were to describe the quality of the development of qualitative research in the medical field, focusing on oncology and palliative care, and to discuss the applicability of a descriptive checklist.

Methods

A review was conducted on Medline and PsycINFO databases. On the basis of their abstract, papers found have been classified considering: publication year, kind of journal, paper type, data gathering method, sample size and declared methodological approach. A sub sample of the previous papers was than selected and their methodological characteristics were evaluated based on a descriptive checklist.

Results

351 abstracts and 26 full papers were analysed. An increase over time in the number of qualitative studies is evident. While most of the papers before 1999 were published on nursing journals (43%), afterwards also medical journals were largely represented. Psychological journals increased from 7% to 12%. The 22% of studies used a sample size lower than 15 and the 15% did not specify the sample size in the abstract. The methodological approach was also often not specified and the percentage increased in the second time period (from 73% to 80%). Grounded theory was the most employed methodological approach while phenomenology shows a decrease. Interview remains the most used data gathering method in both periods, even if it shows a 10% reductions, while focus group and multiple methods application both increase to 12%. The use of the descriptive checklist on the full text of the 26 papers shows that all the items present a larger percentage of satisfaction after 1 January 1999 than it was for the paper published before 1999. There seems to be two different types of quality criteria: specific and unspecific. The first ones mainly refer to qualitative paradigm (such as the relationship with the subject of research or evidence about how subjects perceived the research) and they are often not satisfied. In contrast unspecific criteria (such as the connection to an existing body of knowledge or systematic data gathering) which are mainly shared with the quantitative paradigm are more frequently satisfied.

Conclusions

In oncology and palliative care the publication of qualitative studies increased during the nineties, reaching its peak in around 2000. The use of descriptive checklists even if it was not easy to apply, allows researchers to get a deeper insight into methodological facets that a global judgement may leave out.

Background

In the second half of the nineties a scientific debate about the effectiveness of qualitative research in medicine began in the main medical journals. The opening paper of a series dedicated to this issue on the BMJ in 1995 highlighted that qualitative research is important "to reach the parts that other methods cannot reach" [1]. It also offers insight into emotional and experiential phenomena in health care to determine what, how and why [2]. In the same period the main handbooks to conduct qualitative research have been published allowing a larger number of researchers to apply this kind of methodology to their specific field [3]. In Palliative care, for example, the amount of papers on peer-reviewed journals which employed qualitative methodology has noticeably increased during these last years [4].

Nevertheless the label of qualitative has been assigned to an heterogeneous collection of studies. Some of them show a complete awareness of the specificity of this kind of research, while others are still largely influenced by the quantitative paradigm, prevailing in the medical field. Therefore it is important to emphasise the difference between paradigms which have different objectives (to measure, control and predict in quantitative research, to describe and deepen the understanding of the complexity of human realities in qualitative research) [5] to which different methodologies are adequate. This is a fundamental step to reach a good integration between the two research methodologies.

Specifically in the medical field, while the quantitative paradigm is widely known and accepted, the qualitative one still needs to prove its effectiveness and validity.

The concern with the rigour and credibility of qualitative methods has lead to the development of a number of checklists for assessing qualitative research [6-8]. Checklists can play an important role in granting respectability to qualitative research and in convincing potential sceptics of its thoroughness. They can prepare those unfamiliar with this approach to evaluate or review qualitative work (by providing guidance on crucial questions that need to be asked) and in reminding qualitative researchers of the need for a systematic approach (by providing a memorandum of the various stages involved in research design and data analysis). Checklists can be useful for improving qualitative research methods, but overzealous and uncritical use can be counterproductive. Reducing qualitative research to a list of technical procedures is overly prescriptive [9].

The general purpose of this review was to describe the quality of the development of qualitative research in the medical field, focusing on oncology and palliative care.

The specific aims of the present review were:

• to describe the trend over time in the number and in the methodological characteristics, as reported in the abstract, of published papers which declare to employ a qualitative methodology;

• to analyse and discuss the quality of the full text of a sub sample of those papers, applying a descriptive checklist;

• to discuss the applicability and the usefulness of a descriptive checklist.

Methods

The review was conducted on 31 august 2002 on Medline and PsycINFO databases, being both main databases in the field of interest.

The search strategy was:

1 Tumor* OR oncolo* OR cancer Field: Title/Abstract,

2 "palliative care" OR "terminally ill" OR "terminal patient" Field: Title/Abstract,

3 "qualitative research" OR "qualitative study" OR "qualitative methods" Field: Title/Abstract,

4 (#1 OR #2) AND #3 Field: Title/Abstract,

Papers were excluded if:

• not in English

• without abstract available on Medline/ PsycINFO

• with a research issue related to biology, or laboratory test or diagnostic test (which usually employs qualitative evaluation method for the response of the tests)

• containing the word "qualitative" yet not referring to the methodology applied.

Based on their abstract, papers were classified using the following descriptive variables:

• publication year

• kind of journal (classified as medical, nursing, psychological, bioethical or social science, public health, multidisciplinary)

• paper type (original article, methodological paper, review)

• data gathering method (observation, interview, focus group, narrative, multiple method, non specified)

• sample size

• declared methodological approach (phenomenological, grounded theory, anthropological, ethnographic, non specified, other,)

• typology of subjects involved

• main theme considered (in this variable the main theme was identified. When it was not possible to identify one theme, papers were classified as "miscellanea")

Data gathering method, sample size and declared methodological approach were evaluated for original papers only.

For the second aim, a subsample of the previous papers was selected using the following criteria:

5 ((#1 OR #2) AND #3) AND ((quality AND life) OR (patient AND reported AND outcome) OR (pro) OR (health AND outcome)) Field: Title/Abstract,

The methodological characteristics of the papers in this sub sample were evaluated applying a descriptive checklist suggested by Seale [6] and partially modified as presented in table 1.

Table 1.

Criteria for the evaluation of the qualitative research papers

1)	Are the methods of the research appropriate to the nature of the question being asked?
2)	Is the connection to an existing body of knowledge or theory clear?
3)	Is the selection of cases or participants theoretically justified?
4)	Is the choice of the employed instrument justified?
5)	Is the instrument or the procedure to construct it carefully described?
6)	Are there clear accounts of the data collection?
7)	Was the data collection and record keeping systematic?
8)	Has the relationship between fieldworkers and subjects been considered?
9)	Is there evidence that the research was presented and explained to its subjects?
10)	Is there evidence about how the subjects perceived the research?
11)	Is reference made to accepted procedures for analysis?
12)	How systematic is the analysis?
13)	Is the evidence available for independent examination?
14)	Is there adequate discussion of how themes, concepts and categories were derived from the data?
15)	It is sometimes inevitable that externally given or predetermined descriptive categories are used, but have they been examined for their real meaning or any possible ambiguities?
16)	Is a clear distinction made between the data and their interpretation?
17)	Is there adequate discussion of the evidence both for and against the researcher's arguments?
18)	Have measures been taken to test the validity of the findings?
19)	Is the research clearly contextualised ?
20)	Are the data presented systematically?
21)	Though the presentation of the discursive data is always going to require more space than numerical data, is the paper as concise as possible?
22)	Are the results credible and appropriate?
23)	Have ethical issues been adequately considered?

Other checklists have been considered (2, 7, 8). We chose Seale's checklist because we found it was clearer and more detailed in its formulation. It presents 20 main categories: each category is divided into one or more sub-categories that specify its content or focus on related themes. The checklist's author doesn't provide any specific answer procedure. A dichotomised answer procedure was then chosen in order to evaluate the "satisfaction" or "not satisfaction" of each criterion and not to judge the quality of its fulfilment.

All the papers were examined by 3 independent evaluators (BoCl, MG, LM): in the case of disagreement the evaluation reported by two of them was assigned. Data on agreement was also reported.

We did not obtain a global score to judge the quality of each paper because judgements about the quality of qualitative studies should depend not on the number of criteria met, but on the importance and balance of multiple criteria. Our aim was to summarise the level of satisfaction of each item in the sample of papers examined and not to show the "quality" of each of the articles considered.

All the results are presented stratified by publication period: before and after 1 January 1999.

Results

Out of the 411 papers identified from Medline by the first search, 130 were eliminated as they did not meet the criteria. Out of the 85 papers identified from PsycINFO, 10 were already present in Medline and 5 were eliminated as they did not meet the criteria. Finally 351 abstracts were analysed.

Figure 1 shows the number of papers by publication period; data of 2002 refers only to the first eight months; an estimate of the total number based on twelve months is also reported (the hyphenated line). The increase over time and the stabilisation of the temporal trend over the last four years is evident.

Figure 1.

Number of papers by publication year

Table 2 shows that while most of the papers before 1999 were published on nursing journals (43%) afterwards medical journals were also largely represented. Psychological journals increased from 7% to 12%. Review and methodological papers constitute a small percentage only: 10% over the whole period. The 22% of studies used a sample size lower than 15 and for the 15% of the papers the sample size is not specified in the abstract. Also the methodological approach was often not specified and the percentage increases in the second time period (from 73% to 80%). Grounded theory is the most applied methodological approach while phenomenology shows a decrease. Interview remains the most used data gathering method in both periods, even if it shows a 10% reduction, while focus group and multiple methods application both increase to 12%. In both periods almost half of the studies involve cancer patients. The only percentage decrease occurring after 01.01.1999 (from 13% to 3%) regards the studies involving nurses. The main theme explored is "illness experience" in both periods considered. Also "cancer prevention" is largely represented even if its percentage decreases from 12% to 10%. Issues concerning palliative care, illness experience, communication, and needs assessment-satisfaction show the larger increase. On the contrary the percentage of studies concerning "nursing practice" and "psychological aspects" is lower after 01.01.1999

Table 2.

Characteristics of the papers by publication period

	Publication year
	before 01.01.1999	after 01.01.1999	total
	N = 160	N = 191	N = 351
	%	%	%
Kind of journal
medical	23	30	27
nursing	43	29	35
psychological	7	12	10
bioethical or social science	4	4	4
public health	11	13	12
multidisciplinary	12	12	12
Paper type
original article	91	88	89
methodological paper	4	7	6
review	4	5	4
not evaluable	1	0	1
Sample size*
<15	19	24	22
15–50	26	22	24
>50	38	40	39
Not specified	17	14	15
Declared methodological approach*
phenomenological	8	2	5
grounded theory	10	10	10
anthropological	1	1	1
ethnographic	4	3	4
Other	4	4	4
Not specified	73	80	76
Data gathering method*
observation	1	2	2
interview	65	55	59
focus group	8	12	11
narrative	3	1	2
multiple method	4	12	8
Not specified	12	15	14
Typology of subjects involved*
Cancer patients	46	49	48
Family	15	18	17
Nurse	13	3	7
General population	11	10	11
Other health care workers	9	11	10
Miscellanea	5	6	6
Not evaluable	1	3	5
Main theme considered*
Illness experience	17	22	20
Information/communication	13	17	15
Cancer prevention	12	10	11
End of life	9	11	10
Psychological aspects	9	5	7
Nursing practice	7	1	4
Palliative care	6	14	10
Cultural aspects and attitudes	5	2	3
Pain	5	3	3
Fatigue	3	2	2
Coping style	3	5	4
Quality of life	3	2	3
Needs assessment/satisfaction	1	4	2
Miscellanea	5	1	3
Not evaluable	2	3	2

* Calculations performed only on original articles (total n = 314)

Out of 33 papers selected according to the second aim of the study, 7 were not analysed because it was not possible to find them in the available libraries. Table 3 shows for each item of the descriptive checklist the results of its application on the full text of the 26 papers [10-35] by publication year. The percentage of agreement (i.e. percentage of papers for which the judgement of all three independent evaluators were the same) is also shown.

Table 3.

Percent of papers satisfying each of the items, by publication period, and percent of paper with a complete agreement among the three independent evaluators

Item Number	% of papers with item satisfied		% of papers with complete agreement
	before 01.01.1999	after 01.01.1999
	N = 13	N = 13
1	92	100	81
2	92	100	96
5	85	92	73
6	92	92	77
7	92	100	54
8	15	23	81
9	38	54	50
10	0	15	73
11	77	100	77
12	85	92	73
13	38	54	77
14	69	77	54
15	8	38	50
16	85	100	81
18	0	31	81
19	69	77	58
20	92	100	88
21	77	92	62
22	100	100	85
23	31	54	69

Low level of agreement (< 50%) was found for three items: 3, 4 and 17. For this reason the data regarding these items was considered insufficiently reliable and thus not shown in the table.

Out of the 20 presented items, 13 had a percentage of agreement higher than 70% (1, 2, 5, 6, 8, 10, 11, 12, 13, 16, 18, 20, 22) showing globally a good level of reliability.

All the items present a larger percentage of satisfaction after 1 January 1999. In some cases, the improvement is substantial: item 18 (test of the validity of the findings) was satisfied for almost 1/3 of the papers after 1 January 1999 while it was completely unsatisfied before. Item 11 (reference to an accepted procedures of analysis) was fully satisfied in the second period while before it was satisfied in 77% and item 15 (consideration for the real meaning or any possible ambiguities of the categories adopted) increase from 8% to 38%.

Other items completely satisfied after 1 January 1999 were item 1 (appropriateness of methodology to research question), item 2 (connection to an existing body of knowledge), item 7 (systematic data gathering), item 16 (distinction between data and interpretation), item 20 (systematic presentation of the results) and item 22 (credibility of the results).

Items 8 (consideration of the relationship between fieldworkers and subjects) and 10 (evidence about how the subjects perceived the research) even if improving after 1 January 1999 they still maintain a very low percentage (from 15% to 23% the first) and (from 0% to 15% the second/).

Discussion

According to our analysis, the growing interest for qualitative research in the medical field is proved both by the numerical improvement of these kinds of studies and by the tendency to use and integrate different qualitative instruments.

The qualitative papers' distribution on journals belonging to several disciplines confirms the wider acceptance of this methodology among researchers from different subjects. The large improvement in the proportion of papers published in medical journals confirms the growing acceptance of this methodology in the medical field where studies are traditionally based on the quantitative paradigm. This is consistent with the decrease in the percentage of studies involving nurses or concerning nursing practice.

The higher percentage of studies exploring "Illness experience" or "Information/communication issues" proves that personal and relational experiences are more easily approached by qualitative methodologies.

Cancer prevention is one of the most represented issues probably because this kind of studies focus on specific psychological, social and cultural aspects that influence the subjective cancer risk perception.

The percentage increase of studies concerning "Palliative care" and "End of life" indicates the major efforts by the researchers both to get inside the meaning of end of life experience and inside the discussion on palliative care philosophy.

The increase of the qualitative research concerning the significance of need and the components of satisfaction show a need to reach a better understanding of the real meaning of these issues in order to complete the quantitative findings in this area.

The large percentages of abstracts that do not specify the methodological approach, the data gathering method and the sample size, still suggest a low attention to the communication of methodological aspects of qualitative studies. To report methodological aspects in the abstract is very important not only for the quantitative methods but also for the qualitative ones because these elements allow a first evaluation of the accuracy adopted by researchers to approach the study object.

As regards the applicability and effectiveness of the descriptive checklist on the full content of the published studies, this tool doesn't have an easy application as shown by the low agreement for about 1/3 of the items.

The dichotomous answering procedure we chose proved to be problematic particularly for those items which have a margin of ambiguity in their formulation. For instance, item 22 of the checklist: "is the paper as concise as possible?" concision is an evidently subjective datum, as well as contextualisation in item 19: "Is the research clearly contextualised?"

An answering procedure based on a five points range would probably have benefited from a better agreement among the evaluators in these cases.

However, it's important to underline that the reliability of rating on individual items must not be considered the most important function for a checklist applied to a qualitative study. Although in our application the agreement between evaluators was not always good, the checklist proved useful in guiding readers to consider a large number of important facets before reaching a global judgement on it.

In our study the application of the checklist shows that methodological accuracy has grown over the time. This is an important result even if it does not imply necessarily an improvement of the global quality of each study evaluated in this review. In fact, none of the "technical fixes", considered according to the descriptive checklist, can confer quality to a study by itself.

Items 1 (appropriateness of methodology to research question), 2 (connection to an existing body of knowledge), 7 (systematic data gathering), 11 (reference to accepted procedure of analysis) 16 (distinction between data and interpretation), 20 (systematic presentation of the results) and 22 (credibility of the results) highlight the methodological aspects for which a very good level of knowledge and application have been reached (100% of satisfaction). It is interesting to observe that these items represent the main steps of a qualitative study. This may suggest the existence of a methodological core widely shared among researchers. The low percentages of items 8 (consideration of the relationship between fieldworkers and subjects) and 10 (evidence about how the subjects perceived the research) highlight the persistence of practical difficulties in considering relational implications to the subjects involved in a qualitative study. This data confirms the difficulty in dealing with these complex aspects considering the absence of neutrality that characterises the qualitative researcher's role. Item 9 (research presentation and explanation to the subjects) presents a higher percentage (54%) probably because it is similar to informed consent procedure usually employed in the research field.

To judge the methodological rigor of qualitative research report, readers need to critically appraise the study's design and analysis [2]. Several procedures have been proposed to test the validity of results [6]. In our sample still less then 30% of the papers examined were judged to have made any test on the validity of the findings. These data strengthen the impression of a not complete methodological awareness of qualitative researchers.

There seems to be two different types of quality criteria: specific and unspecific. The first type mainly refers to qualitative paradigm (such as the relationship with the subject of research or evidence about how subjects perceived the research) and it is often not satisfied. On the contrary unspecific criteria (such as connection to an existing body of knowledge or systematic data gathering) which are mainly shared with the quantitative paradigm are more frequently satisfied.

Conclusion

In oncology and palliative care the publication of qualitative studies increased during the nineties, reaching its peak in around 2000. The use of descriptive checklists such as that used in this review allows to focus on the whole picture of methodological components of qualitative research. This way it contributes to the raising of awareness regarding the lack of attention to some methodological aspects, in particular to those specific of qualitative paradigm. The achievement of high methodological standards and the attention to an appropriate reporting of fundamental methodological aspects (such as the methodological approach or the means of collecting data and the sample size)also in the abstract, are indispensable conditions for a complete affirmation of qualitative methodology in the medical field.