Skip to main content
PMC home page
Wiley Open Access Collection logoLink to Wiley Open Access Collection
. 2018 Nov 8;75(1):224–233. doi: 10.1111/jan.13867

Enhancing validity through cognitive interviewing. A methodological example using the Epilepsy Monitoring Unit Comfort Questionnaire

通过认知访谈提高有效性。使用癫痫监测器舒适度问卷的方法学示例

Andrea Egger‐Rainer 1,2,✉
PMCID: PMC7379296  PMID: 30289559

Abstract

Aims

To identify problematic items, assess completeness and user‐friendliness, and undertake modifications to enhance face validity of the newly developed Epilepsy Monitoring Unit Comfort Questionnaire.

Design

Qualitative‐descriptive cross‐sectional study.

Methods

Five iterative rounds of cognitive interviewing were conducted with members of the target population between July ‐ November 2017. Think‐aloud technique, verbal probing, and observation were used, to assess how respondents understood and answered questions. Data analysis was based on the framework method; an eight‐column framework matrix was created for this purpose.

Results

In 25 interviews, problems regarding completeness of the item pool, comprehension of items, retrieval of information, judgment while finding answers and reporting the appropriate responses could be detected. According to the results, 27 items remained unchanged, 11 items were reworded, and six items were added. Instructions section of the questionnaire was reworked, too.

Conclusion

Although time‐consuming, cognitive interviewing turned out to be a valuable approach for revealing problems in an instrument, which would, otherwise, remain undetected and threaten validity.

Keywords: cognitive interviewing, comfort questionnaire, epilepsy monitoring unit, face validity, instrument development, nursing, qualitative method, question‐and‐answer model, questionnaire design, questionnaire pre‐testing

What problem did the study address?

  • Respondents may understand and answer items of a questionnaire in different ways. These problems and an incomplete item pool threaten validity of an instrument.

What were the main findings?

  • Cognitive interviewing revealed problems regarding completeness of the item pool, comprehension of items, retrieval of information, judgment while finding answers and reporting the appropriate response. Rewording of 11 items reduced vagueness, adding of six items completed the item pool.

Where and on whom will the research have an impact?

  • Patients are more willing to participate in research if they feel that a questionnaire has face validity. Valid instruments are essential in nursing research and nursing practice to provide high‐quality inpatient care based on valid research findings.

1. INTRODUCTION

Nurses are interested in measuring the effectiveness of interventions that have been undertaken. Some of the related indicators can be addressed by using technical equipment such as weight scales or blood pressure monitors. However, there are indicators of health conditions that can only be measured by using health‐related, patient‐reported outcomes measurements (PROMs) (Polit & Yang, 2016; U.S. Department of Health and Human Services & Food and Drug Administration, 2009). One of these indicators is comfort which is a two‐dimensional construct pertaining to “the immediate experience of being strengthened by having needs for relief, ease, and transcendence met in four contexts (physical, psychospiritual, sociocultural, and environmental); much more than the absence of pain or other physical discomforts” (Kolcaba, 2013, p. 193). The experience of having high levels of comfort can help patients to overcome stressful situations (Kolcaba, 1995) like hospitalization in an epilepsy monitoring unit (EMU). In an EMU, patients with refractory epilepsy undergo long‐term video‐electroencephalography for about 5 days. For diagnostic purposes, it is necessary that the patients suffer several seizures, thus implying the risk of adverse events. To enhance safety, restricted ambulation of patients, guard rails, specially designed bathrooms, specific nursing, and technician support and standardized operating procedures for the treatment of status epilepticus are compulsory in many EMUs (Dobesberger et al., 2017; Kobulashvili et al., 2016; Rosenow et al., 2016; Spanaki et al., 2012; Spritzer et al., 2015). Former research findings indicate that EMU patients are tense and feel anxious, bored, depressive and less than clean (Andrewes, Camp, Kilpatrick, & Cook, 1999; Bristol, Natarajan, Lin, & Malow, 2001; Egger‐Rainer, Trinka, Höfler, & Dieplinger, 2017; Ozanne, Graneheim, Ekstedt, & Malmgren, 2016). Enhancing patient comfort is an essential outcome of nursing (Kolcaba, 1992) in this setting. However, nurses need an EMU‐setting specific comfort questionnaire to assess comfort levels before and after interventions.

1.1. Background

An instrument that aims to measure a subjective experience like comfort has to be constructed carefully. Potential users must be provided with information regarding the development process and psychometric properties, based on which they can determine the quality of the instrument (Polit & Yang, 2016). If translation of an already existing instrument is included, researchers have to perform a rigorous procedure (Beaton, Bombardier, Guillemin, & Ferraz, 2000; Sousa & Rojjanasrirat, 2011; Wild et al., 2005) following guidelines as edited by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) task force for translation and cultural adaptation (Wild et al., 2005). The working group on the Consensus‐based Standards for the selection of health Measurement Instruments lists three measurement properties—validity, reliability, and responsiveness. The validity domain comprises content and face validity. While content validity refers to an instrument that adequately reflects the construct, face validity means that the members of the target population consider all items of the instrument as relevant (Mokkink et al., 2010). If participants feel that a measurement has no face validity, they will probably decline to participate in a study (Polit & Yang, 2016). Therefore, it has been suggested that patients need to be included in the process of developing PROMs (U.S. Department of Health and Human Services, & Food and Drug Administration, 2009). Validity of self‐administered instruments may be threatened by three components—comprehension problems, validity problems, and processing difficulties (Oksenberg, Cannell, & Kalton, 1991). The more effort undertaken to generate validity, the greater may be the trustworthiness of the instrument. Therefore, a mixed methods approach should be chosen that includes a quantitative method (assessment of content validity index or CVI (Polit, Beck, & Owen, 2007) and a qualitative method (cognitive interviewing or CI) (Polit & Yang, 2016; Rothman et al., 2009).

1.2. Cognitive interviewing

Cognitive interviewing focuses on the questionnaire itself and not on the survey process. It is a method used to determine how respondents understand and answer questions and to assess which modifications are necessary to make questions easier to answer (Collins, 2015; Patrick et al., 2011; Willis & Miller, 2011). Since the layout and visual appearance of a self‐completion questionnaire may influence the question‐and‐answer process, a usability testing should be undertaken to identify processing difficulties and to make the questionnaire more user‐friendly (Blake, 2015). CI is based on Tourangeau's four‐stage question‐and‐answer model. While answering, a respondent must: (a) comprehend what is being asked; (b) retrieve prior knowledge from memory; (c) make a judgement to find the answer; and, (d) either select a pre‐defined category that fits the answer or find appropriate words to express the response (Tourangeau, 1984). Tourangeau's model implicates pitfalls in all four stages including misinterpretation, adding or loss of details, incomplete recalling, and social desirability (Collins, 2015; Tourangeau, 1984). By conducting semi‐structured interviews with members of the target population, researchers can detect problems inherent in the questionnaire. Different techniques are applicable, among which think‐aloud (as described by Ericsson and Simon 1980) and verbal probing (as described by Forsyth and Lessler 2004) are the most common (Collins, 2015; Willis & Miller, 2011). When conducting think‐aloud, participants report thoughts that lead them through the process of finding an appropriate answer to a question. Verbal probing can be performed concurrently or retrospectively. In concurrent probing probes are asked while the questionnaire is being filled in. Retrospective probing is conducted after the questionnaire has been completed and is the preferred way to test self‐completion questionnaires. Probes may be designed either prior to interviews (scripted probes) or during interviews based on emerging problems (spontaneous probes). Mixing the two types of probing is advisable (Willis, 1999; Willis & Artino, 2013).

1.3. Questionnaire

The Epilepsy Monitoring Unit Comfort Questionnaire (EMUCQ) has been conceptualized as a self‐completion questionnaire and aims to assess patient comfort in an EMU. It is based on Kolcaba's General Comfort Questionnaire (GCQ) that was translated from English to German following ISPOR guidelines (Wild et al., 2005). The GCQ consists of 24 positive worded items and 24 negative worded items. After literature review, 12 items were added for setting‐specific usage. Initial content validity was assessed by experts and based on the results, 14 items were omitted. Eight items were considered for further assessment by the patients. The resulting EMUCQ consisted of 38 items and reached an average CVI on scale level (S‐CVI/ave) of 0.90. The CVI indicates the agreement of experts on the relevance of items to measure an underlying construct (Lynn, 1986; Polit et al., 2007). High evidence of the content validity of an instrument is provided by a S‐CVI/ave of at least 0.90 (Waltz et al. as cited in Polit & Beck, 2006). Please find detailed reports elsewhere (Egger‐Rainer, 2018; Egger‐Rainer, Lorenzl, & Trinka, 2018).

1.4. Aim of the study

The aim of this study was to identify problematic items, assess completeness, and user‐friendliness and undertake modifications to enhance face validity of the newly developed EMUCQ.

2. THE STUDY

2.1. Design

When translating an instrument, cognitive debriefing with members of the target population is necessary to assess translation alternatives, comprehensibility, and cognitive equivalence of the translation. Furthermore, inappropriate items and other difficulties need to be detected (Wild et al., 2005). Therefore, a qualitative‐descriptive study was conducted as part of the author's PhD thesis which is dedicated to the development of the EMUCQ. Tourangeaus's four‐stage question‐and‐answer model served as the theoretical framework (Tourangeau, 1984).

2.2. Setting and participants

The study was conducted between July ‐ November 2017 in the four‐bed EMU at the Department of Neurology, Christian Doppler Medical Centre, Paracelsus Medical University, Salzburg, Austria. Patients were asked to participate if they were minimum 18 years of age, able to read and speak German, willing to participate in the study and did not have any cognitive impairment. A purposive sampling strategy was used to recruit a diverse sample (Polit & Yang, 2016). Participants of either gender needed to be from different age groups, have different professional backgrounds and diverse educational levels. This way, a holistic picture of patient interpretations could be provided. Researchers recommended conducting three to four iterative rounds of interviews with 5–15 participants in each round (Prüfer & Rexroth, 2005; Willis, 1999; Willis & Miller, 2011). It was reported that even after the 50th interview, some problems can remain undetected (Blair, Conrad, Ackermann, & Claxton, 2006). Under cost‐effectiveness aspects, more than 30 interviews seemed to be impracticable, while 20 interviews were considered reasonable (Lenzner, Neuert, & Wanda, 2015; Rothman et al., 2009). Interviews were conducted for as long as no additional problems could be identified and data saturation was reached (Patrick et al., 2011). This occurred after 25 interviews.

2.3. Data collection

Cognitive interviewing was performed by the author with inpatients of the Salzburg EMU. A guideline was created including an observation checklist, general questions regarding the questionnaire and scripted probes (Figure 1). These were based on anticipated problems that were pre‐identified by evaluating each item of the EMUCQ with the German version of the question appraisal system (Faulbaum, Prüfer, & Rexroth, 2009; Willis & Lessler, 1999). The semi‐structured interviews aimed to explore clarity and relevance of the items, completeness of the item pool, understandability of the instructions, and visual appearance of the questionnaire (d'Ardenne, 2015; Rothman et al., 2009). Observation, think‐aloud, and retrospective verbal probing were used. Since it was reported that think‐aloud can be burdensome and uncomfortable for respondents (d'Ardenne, 2015; Willis, 1999), they were free to participate solely in verbal probing. When participants had signed informed consent forms, the interviewer explained to them both the interview techniques and they decided if think‐aloud should be used. The interviewer observed the respondents while they were filling out the questionnaire and made notes regarding skipping items, changing answers, giving more than one answer, difficulties regarding the scale, and hesitating while answering. Interviews were audio‐recorded so that the interviewer could concentrate fully on the interviews and not be distracted by taking notes.

Figure 1.

Figure 1

Open in a new tab

Scripted probes for cognitive interviewing [Colour figure can be viewed at wileyonlinelibrary.com]

2.4. Ethical considerations

The study was submitted to the local ethics commission (reference number 415‐EP/73/700‐2016). All patients received written and verbal information and signed informed consent forms.

2.5. Data analysis

The author used Express Scribe 5.75 © NCH Software to transcribe the interviews following guidelines of Dresing and Pehl (2015). Qualitative data analysis was based on the framework method (Ritchie & Spencer, 1994) which has been advocated for analyzing cognitive interviews (d'Ardenne & Collins, 2015; Hall & Beatty, 2014). Five key stages are involved in this approach: (a) familiarization, where the researcher gains an overview of and a feeling for the material by listening to audio tapes and reading transcriptions; (b) identifying a thematic framework, where a framework is set up to sort the material and index categories are derived from the material by identifying themes and key issues; (c) indexing, where categories are applied and codes are attached to the transcriptions; (d) charting, where a matrix for each theme is created; (e) mapping and interpretation, where accounts are compared, patterns are searched, and explanations are developed (d'Ardenne & Collins, 2015; Ritchie & Spencer, 1994). Each item of the EMUCQ was identified as a theme and placed in a separate matrix with eight columns. Matrices were created in Microsoft Excel 2013. Column headings were borrowed from d'Ardenne and Collins (2015) and included respondent details, survey answers, findings from observations and think‐aloud sessions and findings from general probes, comprehension probes, retrieval probes and comfort probes, and other findings (see Figure 2). Basically, charting is a process of abstraction and synthesis; it does not include verbatim text but is a summary of respondents’ comments put in columns (Ritchie & Spencer, 1994). However, wording is important in the context of assessment of questionnaire items and therefore, it was decided to enter direct quotes. The author conducted the analysis and discussed the difficulties with her supervisors.

Figure 2.

Figure 2

Open in a new tab

Example of the unpopulated matrix of item 13 (matrix based on d'Ardenne & Collins, 2015) [Colour figure can be viewed at wileyonlinelibrary.com]

3. RESULTS

Five rounds of CI were conducted. Interviews lasted for 41 min and 6 s on an average and revealed problems related to general aspects of the questionnaire, interpretation of items, and missing items. Problems were addressed if they were detected in two or more interviews. According to the findings, 27 items remained unchanged. Eleven items underwent rewording while another six items were added. Findings are reported in terms of the four main stages of the question‐and‐answer process (Tourangeau, 1984).

3.1. Sample characteristics

A total number of 29 patients were asked to participate in the interviews and 25 of them agreed. Reasons for denial included visits by friends, premature cancellation of stay at the EMU, and discharge on the planned day of the interview. The final sample consisted of 13 women and 12 men between the ages of 19–74 years (mean: 35.21 years). Most of them were single (72%), had completed vocational training (48%), and were employed full‐time (48%) (Table 1).

Table 1.

Characteristics of participants (N = 25)

N %
Gender
Female 13 52%
Male 12 48%
Age (mean: 35.21 years)
≤20 years 1 4%
21–40 years 17 68%
41–60 years 6 24%
≥61 years 1 4%
Marital status
Single 18 72%
Married/Unmarried couple 7 28%
Education
Compulsory school 2 8%
High school (diploma) 4 (2) 16%
Vocational Training (Additional High School diploma/ Master Tradesman Certificate) 12 (1/1) 48%
University Graduate 7 28%
Employment status
Student 1 4%
Apprentice 2 8%
Full‐time employee 12 48%
Part‐time employee 4 16%
Housewife/Househusband 1 4%
Unemployed/sick leave 2 8%
Retired 3 12%
Open in a new tab

3.2. Comprehension

Some participants did not interpret item 11, “No one understands me,” in the intended manner. In the context of comfort, this item pertains to personal feelings that cannot be understood by others. Patients reported: “…nurses and doctors are foreigners and don't speak German” (P6, m) or “I find it difficult to think of appropriate words. It's a symptom of my seizures” (P9, m). It was helpful to emphasize the aspect of feeling in the item and to do rewording: “I feel misunderstood.” Item 21, “I would like to see a doctor more often,” was interpreted literally: “Actually, I do see them during their rounds in wards. However, they should take more time to talk with me or at least give me that feeling” (P7, f). As comfort is influenced by the possibility of talking to a physician, this item was clarified by replacing “see” with “talk to.” Item 15, “I do not like it here” and item 32, “I have enough information about my health,” were considered unclear. In the opinion of participants, item 15 could pertain to the hospital or to the EMU: “This question is woolly. What is meant by ‘here’? The hospital? The ward?” (P8, f). Replacing the word “here” with “in this ward” eliminated misinterpretation. In item 32, “I have enough information about my health,” participants referred not only to their admission diagnoses but also to other diseases: “I have some sort of myatrophy and the doctor told me that I should drink more water to spare my kidneys” (P5, m). Rewording to “current health condition” made this item more specific. In the context of an earlier content validity rating, it was decided to omit the item, “I would like to do daily exercises with a physiotherapist” and to retain, “I would like to walk around more,” instead (Egger‐Rainer, 2018). In CI, participants were critical of this item: “Everybody wants to walk more. It would be better to include physiotherapy as we are not allowed to walk” (P14, f). Therefore, the item pertaining to physiotherapy was included again and the other item was omitted. Since some participants confounded “physiotherapy” with “psychotherapy”, rewording was necessary. Furthermore, “daily” was replaced by “more often” because some EMUs may already offer daily physiotherapy. The final wording of the item was: “I would like to do physical exercises with a therapist more often.” In item 10, “The sounds keep me from resting,” patients reported that sounds bothered them in a general manner rather than keeping them from resting: “I am used to noise. However, here helicopters are coming and going” (P1, f). The item was changed to “There are sounds here that bother me”. With regard to item 40, “I am afraid of a seizure,” participants commented: “It's less the seizure itself. It's more the effects of the seizure and how I will be feeling during the following hours” (P22, m). Therefore, this aspect was added.

3.3. Retrieval of information

As the EMUCQ aims at comfort as a momentaneous experience, retrieval of information was not reported to be difficult. Participants who had earlier been in the EMU compared their present situation with the previous one. This applied to item 22, “The temperature in this room is fine,” when a participant reported: “Much better than before when there was no air conditioning” (P10, m). In item 42, “I feel clean,” participants knew that this feeling would change: “It is only the second day. Ask me on the fifth day, once more” (P23, w).

3.4. Judgment

Cognitive interviewing showed that respondents referred to their situation at home or to their general living conditions when answering items 2, 5, 7, and 26. Participants responded to item 2, “I have enough privacy,” “I thought about at home” (P6, m) or “Does this refer to the ward?” (P9, m). During probing of item 7, “I feel dependent on others,” participants commented: “It is concerned with driving a car” (P10, m), or “I need someone to accompany me to the swimming pool” (P14, f). Therefore, these items were reworded to clarify that they were related to the specific situation at the EMU. Additionally, in the instructions section, bold letters were used to emphasize the connection of comfort to “the moment” and “right now.” These changes turned out to be appropriate.

3.5. Response

In general, participants found a six‐point Likert‐type scale ranging from “strongly disagree” ‐ “strongly agree” appropriate for answering the questions. They appreciated that the labelling of the scale was done as a two‐line heading on both sides of the questionnaire. Additionally, guidelines and an example could be found in the instructions section. Some missed a midpoint option (“Reducing to five categories would offer a middle. It's easier for people if they find something of that kind” [P23, f]), others suggested reducing the scale to solely two or three options (“It is sufficient to ask ʻagree’ or ʻdisagree’. There is no need for ʻstrongly agree’ or ʻstrongly disagree’” [P9, m]) and yet others would have preferred a system similar to grades (“Something like ʻvery good’ or ʻgood’ might be easier” [P24, f]). All response options were selected during CI. However, the interviewer observed some participants having difficulties while answering negatively drafted items and items including “not” or “no”. This was especially seen in item 40, “I am not afraid of a seizure.” Therefore, this item was reworded after the second round.

3.6. Completeness

Participants rated five of the eight items, which were put aside in the earlier expert rating, as relevant for assessing patient comfort in the EMU. Three items were included in their unchanged forms in the questionnaire (1, 20, 30) and two items were reworded (6, 29). Although the item, “It is difficult to eat in this position,” was rated as very relevant by the participants, it was decided not to include it. This omission was justified by the fact that in some EMUs patients do not need to eat in their bed (which is compulsory in Salzburg) but are allowed to take their meals sitting at a table. One item was rated irrelevant, “I have experienced changes which make me feel uneasy” and one item was considered redundant, “I feel peaceful.” Item 14, “My faith helps me to not be afraid,” was omitted after the expert rating. However, participants missed an item pertaining to God. Therefore, it was decided to include this item again in the EMUCQ. An overview of the results may be found in Tables 2, 3, 4.

Table 2.

Modifications per interview round

Round 1 (N = 9) Seven items reworded
one item added
Instructions section refined
Round 2 (N = 5) Four items reworded
Three items added
Round 3 (N = 4) One item reworded (second time)
Two items added
Round 4 (N = 4) Two items reworded (second time)
Round 5 (N = 3) No modifications conducted
Open in a new tab

Table 3.

Overview of revised items

Original item Revised item Revision after interview round
2 I have enough privacy I have enough privacy in this ward 1
4 I would like to walk around more I would like to do exercises with a physiotherapist more often 2
I would like to do physical exercises with a therapist more often 4
5 My condition gets me down My present condition gets me down 2
7 I feel dependent on others I feel dependent on others right now 1
I feel dependent on others here 3
10 There are sounds here that keep me from resting There are sounds here that bother me 2
11 No one understands me I feel misunderstood 1
15 I do not like it here I do not like it in this ward 1
21 I would like to see a doctor more often I would like to talk to a doctor more often 1
26 I am content I am content with the situation here 1
32 I have enough information about my health I have enough information about my current health condition 1
40 I am not afraid of a seizure I am afraid of a seizure 2
I am afraid of a seizure/its aftermath 4
Open in a new tab

Table 4.

Added items due to patient rating

Original item Interview findings Added item Added after interview round
I feel confident

Patients assessed this item as very relevant for the EMUCQ. However, they rated the item as unclear: “I feel confident…in my situation or with the therapy? What do you mean?” (P6, m)

“This question needs an appendix: for example regarding therapy.” (P9, m)

 I am confident that an appropriate therapy will be found for me 1
My body is relaxed right now Participants linked relaxation strongly with comfort: “In the EMU, relaxation is a good index of comfort.” (P12, f). “I would say this is some sort of feeling at ease.” (P11, m) Unchanged 2
My faith helps me to not be afraid Most of the patients rated this item as very relevant: “For a religious person, this is very important. By asking this question, you feel understood.” (P 3, f) “I wonder, if this is relevant for assessing comfort […] However, if the doctors don't know, who else? Yes, it is.” (P 14, f) “We are talking about comfort and for some people this is certainly a part of it.” (P 21, m) Unchanged 2
My personal belongings are not here Participants reported that they had all their belongings with them. However, some were beyond arm's reach. “It is important that things are within reach. This should be included in the question.” (P10, m) I can reach my personal belongings easily 2
I am hungry

Mainly male patients wanted this item to be included in the EMUCQ: “If there is nothing on the TV you get hungry because you have got nothing else to do.” (P3, f)

“Absolutely, you get too little to eat here.” (P 17, m)

 Unchanged 3
I feel out of place here Participants rated this item as relevant because patients may be doubtful about their stay: “This investigation is useless because no seizure has occurred, by now.” (P3, f). “It is really important to include this question. There are always people like two of my fellow patients who don't want to be here” (P19, m) Unchanged 3
Open in a new tab

4. DISCUSSION

This study showed that finding appropriate answers involves a complicated process consisting of comprehension, retrieval, judgement, and response (Tourangeau, 1984). CI is a helpful method to identify problems that would otherwise remain undetected (Drennan, 2003; Patrick et al., 2011; Willis & Artino, 2013). The framework method for data analysis is suitable for organize and summarize data (d'Ardenne & Collins, 2015). This study showed that small changes in font or wording may lead to more clarity in understanding. Although reported in the introductions section, it was not clear to the patients that all items specifically referred to the situation at the EMU right now. Therefore, it was important to emphasize this aspect; first, by using bold letters for phrases that referred to this aspect in the introductions section and then, by rewording concerned items. Also, Knafl et al. (2007) observed a similar problem while probing items of the Family Management Measure when respondents did not know if an item referred to home or to school. The phenomenon of interpreting items literally could be observed not only in this study, but in others as well (Collins, 2015; Hall & Beatty, 2014). In the EMUCQ the problem may have been due to the translation. While it is common to talk about “seeing the doctor” in the English‐speaking world, it is less usual in Austrian language. Participants were aware of the concept of comfort. They rated the EMUCQ as incomplete and suggested the inclusion of five items, which were put aside after expert rating. Also, they expressed missing an item pertaining to God. Validity of the questionnaire could be compromised if these items remained excluded (Rothman et al., 2009). Some participants had difficulties in answering negatively worded questions. Mixing positive and negative items was critically discussed as it could lead to various problems. Due to the method effects, negatively worded items could be identified as a separate factor in factor analysis (Dunbar, Ford, Hunt, & Der, 2000; Marsh, 1996) and especially, young persons (Marsh, 1996; Polit & Yang, 2016) and people with lower intellectual or communication abilities (Dunbar et al., 2000; Marsh, 1996) could have problems finding the correct answers. This phenomenon was observed in this study independent of participants’ ages. However, some patients had undergone sleep‐deprivation the previous night or had suffered a seizure some hours before the interview. These two factors may have had a negative impact on their cognition. Nonetheless, it was decided to retain the mix of positive and negative items to emphasize that the EMUCQ is part of Kolcaba's Comfort Questionnaire family. If an item turned out to be problematic, it was reworked. For now, findings support high face and content validity of the EMUCQ. Additional testing is necessary to further address construct and convergent validity, reliability, and responsiveness; particularly, more clarity about a method effect may be found in a confirmatory factor analysis (Dunbar et al., 2000).

4.1. Limitations

This study was conducted in a single EMU; hence, patients belonged to the same language area. It is not certain if people of other language areas would have understood and interpreted the items in the same manner. Furthermore, participation was voluntary. The inclusion of other people may have resulted in different findings.

5. CONCLUSION

Cognitive interviewing offered the chance to identify both unproblematic and problematic items of the EMUCQ. Additionally, room for improvement in the layout of the questionnaire was revealed. Although time‐consuming, CI turned out to be an inevitable part of instrument development. Participants provided feedback about understandability of written words and practicability of the questionnaire, so that the researcher could immediately clarify problems. Implementation of findings leads to enhanced face validity. Hence, respondents may show more willingness to participate in a study and the instrument will be more useful in clinical practice. After further quantitative testing, the final EMUCQ will enable nurses to find a valuable tool for the assessment of experienced comfort of EMU patients. Made‐to‐measure interventions, based on the results of comfort assessment, will be meaningful quality indicators of person‐centered care and will lead to highly satisfied patients.

ACKNOWLEDGEMENTS

The author is grateful to Prof. Dr. Stefan Lorenzl and Prof. Dr. Eugen Trinka who provided scientific support; to the patients who participated in the interviews; to the EMU‐staff who provided organizational support; and to Prof. Dr. Katharine Kolcaba who allowed translation and modification of the General Comfort Questionnaire.

CONFLICT OF INTEREST

No conflict of interest has been declared by the author.

AUTHOR CONTRIBUTIONS

All authors have agreed on the final version and meet at least one of the following criteria (recommended by the ICMJE [http://www.icmje.org/recommendations/]):

  1. substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data;

  2. drafting the article or revising it critically for important intellectual content.

Egger‐Rainer A. Enhancing validity through cognitive interviewing. A methodological example using the Epilepsy Monitoring Unit Comfort Questionnaire. J Adv Nurs. 2019;75:224–233. 10.1111/jan.13867

Funding information

This research received no specific grant from any funding agency in the public, commercial, or not‐for‐profit sectors.

REFERENCES

  1. Andrewes, D. , Camp, K. , Kilpatrick, C. , & Cook, M. (1999). The assessment and treatment of concerns and anxiety in patients undergoing presurgical monitoring for epilepsy. Epilepsia, 40(11), 1535–1542. 10.1111/j.1528-1157.1999.tb02037.x [DOI] [PubMed] [Google Scholar]
  2. Beaton, D. E. , Bombardier, C. , Guillemin, F. , & Ferraz, M. B. (2000). Guidelines for the process of cross‐cultural adaptation of self‐report measures. Spine, 25, 3186–3191. 10.1097/00007632-200012150-00014 [DOI] [PubMed] [Google Scholar]
  3. Blair, J. , Conrad, F. , Ackermann, A. C. , & Claxton, G. (2006). The effect of sample size on cognitive interview findings. Retrieved from http://www.abtassoc.us/presentations/AAPOR06_Sample_Size_Cognitive_Interviews.pdf
  4. Blake, M. (2015). Other pretesting methods In Collins D. (Ed.), Cognitive interviewing practice (pp. 28–56). London: SAGE; 10.4135/9781473910102 [DOI] [Google Scholar]
  5. Bristol, K. , Natarajan, A. , Lin, X. , & Malow, B. (2001). Effects of long‐term video‐electroencephalographic monitoring on mood in epilepsy patients. Epilepsy & Behavior, 2(5), 433–440. 10.1006/ebeh.2001.0237 [DOI] [PubMed] [Google Scholar]
  6. Collins, D. (2015). Cognitive interviewing: Origin, purpose and limitations In Collins D. (Ed.), Cognitive interviewing practice (pp. 3–27). London: SAGE; 10.4135/9781473910102 [DOI] [Google Scholar]
  7. d'Ardenne, J. (2015). Developing interview protocols In Collins D. (Ed.), Cognitive interviewing practice (pp. 101–125). London: SAGE; 10.4135/9781473910102 [DOI] [Google Scholar]
  8. d'Ardenne, J. , & Collins, D. (2015). Data management In Collins D. (Ed.), Cognitive interviewing practice (pp. 142–161). London: SAGE; 10.4135/9781473910102 [DOI] [Google Scholar]
  9. Dobesberger, J. , Höfler, J. , Leitinger, M. , Kuchukhidze, G. , Zimmermann, G. , Thomschewski, A. , … Trinka, E. (2017). Personalized safety measures reduce the adverse event rate of long‐term video EEG. Epilepsia Open, 2(4), 400–414. 10.1002/epi4.12078 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Drennan, J. (2003). Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. Journal of Advanced Nursing, 42(1), 57–63. 10.1046/j.1365-2648.2003.02579.x [DOI] [PubMed] [Google Scholar]
  11. Dresing, T. , & Pehl, T. (2015). Praxisbuch Interview, Transkription & Analyse. Anleitungen und Regelsysteme für qualitativ Forschende [Practice Book Interview, Transcription & Analysis. Guides and control systems for qualitative researchers] (6th ed.). Marburg: Self Publishing Company. [Google Scholar]
  12. Dunbar, M. , Ford, G. , Hunt, K. , & Der, G. (2000). Question wording effects in the assessment of global self‐esteem. European Journal of Psychological Assessment, 16(1), 13–19. 10.1027//1015-5759.16.1.13 [DOI] [Google Scholar]
  13. Egger‐Rainer, A. (2018). Determination of content validity of the epilepsy monitoring unit comfort questionnaire using the content validity index. Journal of Nursing Measurement, 26(2), 398–410. 10.1891/1061-3749.26.2.398 [DOI] [PubMed] [Google Scholar]
  14. Egger‐Rainer, A. , Lorenzl., S. , & Trinka, E. (2018). Assessing comfort in the epilepsy monitoring unit: Development of an instrument. Epilepsy & Behavior. 10.1016/j.yebeh.2018.05.004 [DOI] [PubMed] [Google Scholar]
  15. Egger‐Rainer, A. , Trinka, E. , Höfler, J. , & Dieplinger, A. (2017). Epilepsy monitoring ‐ The patients’ views: A qualitative study based on Kolcaba's Comfort Theory. Epilepsy & Behavior, 68, 208–215. 10.1016/j.yebeh.2016.11.005 [DOI] [PubMed] [Google Scholar]
  16. Ericsson, K. A. , & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251. 10.1037/0033-295X.87.3.215 [DOI] [Google Scholar]
  17. Faulbaum, F. , Prüfer, P. , & Rexroth, M. (2009). Was ist eine gute Frage? Die systematische Evaluation der Fragenqualität [What is a good question? The systematic evaluation of the question quality]. Wiesbaden: VS Verlag für Sozialwissenschaften. [Google Scholar]
  18. Forsyth, B. , & Lessler, J. (2004). Cognitive laboratory methods: A taxonomy In Biemer P., Groves R., Lyberg L., Mathiowetz N., & Sudman S. (Eds.), Measurement errors in surveys (pp. 393‐418). Hoboken: Wiley. [Google Scholar]
  19. Hall, S. , & Beatty, S. (2014). Assessing spiritual well‐being in residents of nursing homes for older people using the FACIT‐Sp‐12: A cognitive interviewing study. Quality of Life Research, 23(6), 1701–1711. 10.1007/s11136-014-0627-6 [DOI] [PubMed] [Google Scholar]
  20. Knafl, K. , Deatrick, J. , Gallo, A. , Holcombe, G. , Bakitas, M. , Dixon, J. , & Grey, M. (2007). The analysis and interpretation of cognitive interviews for instrument development. Research in Nursing & Health, 30(2), 224–234. 10.1002/(ISSN)1098-240X [DOI] [PubMed] [Google Scholar]
  21. Kobulashvili, T. , Höfler, J. , Dobesberger, J. , Ernst, F. , Ryvlin, P. , Cross, J. H. , … Trinka, E. (2016). Current practices in long‐term video‐EEG monitoring services: A survey among partners of the E‐PILEPSY pilot network of reference for refractory epilepsy and epilepsy surgery. Seizure, 38, 38–45. 10.1016/j.seizure.2016.03.009 [DOI] [PubMed] [Google Scholar]
  22. Kolcaba, K. (1992). Holistic comfort: Operationalizing the construct as a nurse‐sensitive outcome. Advances in Nursing Science, 15(1), 1–10. 10.1097/00012272-199209000-00003 [DOI] [PubMed] [Google Scholar]
  23. Kolcaba, K. (1995). The art of comfort care. Image Journal of Nursing Scholarship, 27(4), 287–289. 10.1111/j.1547-5069.1995.tb00889.x [DOI] [PubMed] [Google Scholar]
  24. Kolcaba, K. (2013). Comfort In Peterson S. J. & Bredow T. S. (Eds.), Middle range theories (3rd ed, pp. 193–209). Philadelphia, PA: Wolters Kluwer|Lippincott Williams & Wilkins. [Google Scholar]
  25. Lenzner, T. , Neuert, C. , & Wanda, O. (2015). Kognitives pretesting. GESIS Survey Guidelines [Cognitive pretesting. GESIS survey guidelines]. Mannheim, Germany: GESIS‐Leibniz‐Institut für Sozialwissenschaften; 10.15465/gesis-sg_010 [DOI] [Google Scholar]
  26. Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. [PubMed] [Google Scholar]
  27. Marsh, H. W. (1996). Positive and negative global self‐esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810–819. 10.1037/0022-3514.70.4.810 [DOI] [PubMed] [Google Scholar]
  28. Mokkink, L. B. , Terwee, C. B. , Patrick, D. L. , Alonso, J. , Stratford, P. W. , Knol, D. L. , … de Vet, H. C. W. (2010). The COSMIN study reached international consensus on taxonomy, terminology and definitions of measurement properties for health‐related patient‐reported outcomes. Journal of Clinical Epidemiology, 63(7), 737–745. 10.1016/j.jclinepi.2010.02.006 [DOI] [PubMed] [Google Scholar]
  29. Oksenberg, L. , Cannell, C. , & Kalton, G. (1991). New strategies for pretesting survey questions. Journal of Official Statistics, 7(3), 349–365. [Google Scholar]
  30. Ozanne, A. , Graneheim, U. H. , Ekstedt, G. , & Malmgren, K. (2016). Patients’ expectations and experiences of epilepsy surgery—A population‐based long‐term qualitative study. Epilepsia, 57(4), 605–611. 10.1111/epi.13333 [DOI] [PubMed] [Google Scholar]
  31. Patrick, D. L. , Burke, L. B. , Gwaltney, C. J. , Leidy, N. K. , Martin, M. L. , Molsen, E. , & Ring, L. (2011). Content validity—Establishing and reporting the evidence in newly developed patient‐reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2—assessing respondent understanding. Value in Health, 14(8), 978–988. 10.1016/j.jval.2011.06.013 [DOI] [PubMed] [Google Scholar]
  32. Polit, D. F. , & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing and Health, 29(5), 489–497. 10.1002/nur.20147 [DOI] [PubMed] [Google Scholar]
  33. Polit, D. F. , Beck, C. T. , & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 459–467. 10.1002/nur.20199 [DOI] [PubMed] [Google Scholar]
  34. Polit, D. F. , & Yang, F. M. (2016). Measurement and the measurement of change. A primer for the health professions. Philadelphia: Wolters Kluwer. [Google Scholar]
  35. Prüfer, P. , & Rexroth, M. , & Zentrum für Umfragen, Methoden und Analysen‐ZUMA . (2005). Kognitive Interviews. GESIS‐How‐to 15 [Cognitive interviews. GESIS‐How‐to 15]. Retrieved from http://nbn-resolving.de/urn:nbn:de:0168-ssoar-201470
  36. Ritchie, J. , & Spencer, L. (1994). Qualitative data analysis for applied policy research In Bryman A., & Burgess R. G. (Ed.), Analyzing qualitative data (pp. 173–194). London: Routledge; 10.4324/9780203413081 [DOI] [Google Scholar]
  37. Rosenow, F. , Bast, T. , Czech, T. , Feucht, M. , Hans, V. H. , & Helmstaedter, C. , … Strzelczyk, A. (2016). Revised version of quality guidelines for presurgical epilepsy evaluation and surgical epilepsy therapy issued by the Austrian, German and Swiss working group on presurgical epilepsy diagnosis and operative epilepsy treatment. Epilepsia, 57, 1215–1220. 10.1111/epi.13449 10.1111/epi.13449 [DOI] [PubMed] [Google Scholar]
  38. Rothman, M. , Burke, L. , Erickson, P. , Kline Leidy, N. , Patrick, D. L. , & Petrie, C. D. (2009). Use of existing patient‐reported outcome (PRO) instruments and their modification: The ISPOR good research practices for evaluation and documenting content validity for the use of existing instruments and their modification PRO task force report. Value in Health, 12(8), 1075–1083. 10.1111/j.1524-4733.2009.00603.x [DOI] [PubMed] [Google Scholar]
  39. Sousa, V. D. , & Rojjanasrirat, W. (2011). Translation, adaptation and validation of instruments or scales for use in cross‐cultural health care research: A clear and user‐friendly guideline. Journal of Evaluation in Clinical Practice, 17(2), 268–274. 10.1111/j.1365-2753.2010.01434.x [DOI] [PubMed] [Google Scholar]
  40. Spanaki, M. V. , McCloskey, C. , Remedio, V. , Budzyn, D. , Guanio, J. , Monroe, T. , … Schultz, L. (2012). Developing a culture of safety in the epilepsy monitoring unit: A retrospective study of safety outcomes. Epilepsy & Behavior, 25(2), 185–188. 10.1016/j.yebeh.2012.06.028 [DOI] [PubMed] [Google Scholar]
  41. Spritzer, S. D. , Riordan, K. C. , Berry, J. , Corbett, B. M. , Gerke, J. K. , Hoerth, M. T. , … Noe, K. H. (2015). Fall prevention and bathroom safety in the epilepsy monitoring unit. Epilepsy & Behavior, 48, 75–78. 10.1016/j.yebeh.2015.05.026 [DOI] [PubMed] [Google Scholar]
  42. Tourangeau, R. (1984). Cognitive sciences and survey methods In Jabine T. B., Straf M. L., Tanur J. M., & Tourangeau R. (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington, D.C: National Academy Press. [Google Scholar]
  43. U.S. Department of Health and Human Services, & Food and Drug Administration . (2009). Guidance for Industry. Patient‐Reported Outcome Measures: Use in medical product development to support labeling claims. Retrieved from https://www.fda.gov/downloads/drugs/guidances/ucm193282.pdf [DOI] [PMC free article] [PubMed]
  44. Wild, D. , Grove, A. , Martin, M. , Eremenco, S. , McElroy, S. , & Verjee‐Lorenz, A. , … ISPOR Task Force for Translation and Cultural Adaptation . (2005). Principles of good practice for the translation and cultural adaptation process for patient‐reported outcomes (PRO) measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value in Health, 8(2), 94–104. 10.1111/j.1524-4733.2005.04054.x 10.1111/j.1524-4733.2005.04054.x [DOI] [PubMed] [Google Scholar]
  45. Willis, G. B. (1999). Cognitive interviewing. A “how to” guide. Retrieved from http://appliedresearch.cancer.gov/archive/cognitive/interview.pdf
  46. Willis, G. B. , & Artino, A. R. (2013). What do our respondents think we're asking? Using cognitive interviewing to improve medical education surveys. Journal of Graduate Medical Education, 5(3), 353–356. 10.4300/JGME-D-13-00154.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Willis, G. B. , & Lessler, J. T. (1999). Question appraisal system QAS‐99. Rockville: Research Triangle Institute. [Google Scholar]
  48. Willis, G. B. , & Miller, K. (2011). Cross‐cultural cognitive interviewing: Seeking comparability and enhancing understanding. Field Methods, 23(4), 331–341. 10.1177/1525822X11416092 [DOI] [Google Scholar]

Articles from Journal of Advanced Nursing are provided here courtesy of Wiley

RESOURCES