Selection and Implementation of Outcome Measurement

Introduction

Advancing our knowledge through research requires the development of a well-designed research protocol that aligns with the research question(s). The results or the outcomes of clinical research are also dependent on high quality and reliable data collection methods (Saczynski, McManus, & Goldberg, 2013). Careful consideration is needed to ensure that the data collected matches the aims of the research, feasible to obtain and able to evaluate the outcome (phenomena of interest) of the investigation.

As discussed in the fourth column of this series, “Planning for Measurement in Nursing Research” (Curtis & Keeler, submitted), data may also be called a variable. A variable has measurable traits that can change during a study and evaluated for associations between other variables and used to predict outcomes and draw conclusions about cause and effect (Hulley et al., 2013). Approaches used to obtain data include indirect and direct measures (Curtis & Keeler, submitted). Interviews and questionnaires are examples of the indirect method to quantify abstract concepts such as mood and pain, serving as a proxy measure for the phenomenon (Curtis & Keeler, submitted). The measurement of physiologic data, such as blood pressure, waist circumference, and biological assays, are examples of direct measurements.

In quantitative studies, statistical techniques are used to analyze numerical data to evaluate associations between research variables and to evaluate effects (outcomes) (Curtis & Keeler, submitted). In comparison, qualitative studies analyze non-numerical data sources, such as text-based sources (Curtis & Keeler, submitted), to assess for experiences or to understand processes for a population under study (Capili, 2020). This paper in the Step by Step Research series continues the discussion on outcome measures published in the fourth installment of this series. This paper focuses on questionnaire formats, direct methods for collecting data, and guidelines for selecting and administering these approaches in clinical research. Examples of direct methods for data collection are discussed briefly since the process and procedures for this approach are vast and continually evolving with science and technology.

Guidelines for Selecting and Implementing Measurement Approaches:

Before selecting the measurement instrument, investigators need to create a list of variables (data/information) and concepts that require measurement in the study (Hulley et al., 2013). Consider the role each item serves in answering the research question(s) (i.e., independent variable, dependent variable/outcome variable, potential confounding variables), and review approaches to measure (evaluate) the variables. Please refer to, “Planning for Measurement in Nursing Research” (Curtis & Keeler, submitted) an article in this series for further information on variables.

In selecting the measurement questionnaires or instruments, identify existing questionnaires (i.e., Insomnia Severity Index, SF-36 Health Survey) or candidate questions that can serve to measure the variable(s) under study (Hulley et al., 2013). It is critical to use the best instrument to measure the independent and dependent variables. Review the literature for existing questionnaires used to measure the variables of interest. The methods section of published studies should report the psychometric properties (validity, internal consistency, reliability) of the selected instrument. Please refer to the article “Planning for Measurement in Nursing Research” for information discussing psychometric properties. Additionally, it is best to use an existing tool without modifications because deleting question items might change the meaning of scores or the tool’s ability to detect changes (Hulley et al., 2013).

Furthermore, select an existing instrument validated for a specific language (i.e.g, instrument validated in English or Spanish). It is not enough to hire a professional to translate a tool developed for a Chinese population into English. Translated instruments require validation by experts and require additional psychometric testing in the new language (Sperber, 2004). The rationale for new psychometric testing is because the original questionnaire might use cultural references or norms not relevant or understandable in the translated language. For example, the concept of ‘adolescent’ or ‘family’ might differ between the cultures. The challenge is to adapt an existing tool that is relevant and understandable while keeping the original meaning and intent of the instrument (Sperber, 2004). Developing a new instrument is always an option, but that often takes several years to validate and is beyond the scope of this article.

To clarify the concepts presented above, reducing insomnia will be used as an example. If an investigator wants to examine the efficacy of an herbal preparation compared to a placebo to reduce insomnia, the independent variables are the types of treatment (herbal preparation treatment and the placebo). The dependent variables are the level of insomnia (measured by the Insomnia Severity Index – indirect measure), the hours of sleep per night (measured by a wrist actigraph – direct measure), and the level of fatigue (measured by a salivary fatigue biomarker – direct measure). The actigraph is a small, wearable device that can yield an objective and more precise estimation of sleep duration compared to using a questionnaire that relies on recall, which can be faculty when participants do not accurately remember previous events or experiences. Measuring a salivary biomarker for fatigue (Michael, Valle, Cox, Kalns, & Fogt, 2013) can also serve as a biomarker for sleep deprivation. The results from the processed biological assays (saliva, serum, hair, urine, stool) are the ‘direct measure’ and not the actual sample.

Confounding variables are variables that influence both the dependent and independent variables. The confounder is an extra variable that affects both the dependent and independent variables under study, so the results do not reflect the actual relationship between the variables under investigation (Pourhoseingholi, Baghestani, & Vahedi, 2012). One method to identify potential confounding variables is to review published studies using similar interventions. For instance, in a study examining older adults with chronic insomnia, some participants might take medications associated with increased alertness or drowsiness identified in the literature. One method to reduce the possible effects of confounding is to use a randomized clinical trial design because it assists in creating study groups that are relatively equal concerning known and unknown confounding variables (Pourhoseingholi et al., 2012). For additional methods to control confounding variables, please consult a statistician. A future article in this series will focus on experimental approaches, including the randomized clinical trial design and inferential statistics.

Administration/Implementation

Regardless of which method is used to collect data and measure research outcomes, study protocols should develop an operations manual or a set of specific instructions on how to implement the study. Instructions can include how to teach the participants to complete questionnaires, the order of completion, how to conduct an interview, how to use biometric instruments, and how to collect specimens. The written guidelines will assist in maintaining a uniform data collection process and reduce systematic errors that are often associated with incorrect administration of an instrument, participant instruction, or calibration of equipment that is repeated throughout a study (Hulley et al., 2013).

Indirect Questionnaire Method

Much of the data collected in clinical research uses participant self-report on standardized questionnaires, wherein everyone answers the same questions (Saczynski et al., 2013). Collecting self-report data allows the investigator to collate information from a sample of individuals through their responses (Ponto, 2015). Questionnaires can use a qualitative approach (open-ended questions), a quantitative approach (close-ended questions with numerically rated items), or both (mixed methods) (Ponto, 2015). Factors often collected include socio-demographic characteristics, medical history, medication use, and lifestyle practices (Saczynski et al., 2013), which can serve as independent or confounding variables in the data analysis. Other questions commonly asked focus on human behavior, healthcare knowledge, quality of life, and functional status (Saczynski et al., 2013). Methods to complete the questionnaires include interviews conducted in-person, via telephone, or by using virtual teleconferencing platforms (e.g., Skype, WhatsApp, Zoom). Administering questionnaires, on paper, or electronically is also another approach to gather the data (Hulley et al., 2013). (See the administration section for additional information).

Open-ended Questions

The open-ended style of question has a specific purpose and limitations. The open-ended form provides the opportunity to capture a participant’s exact words (Hulley et al., 2013). For example, what does good sleep mean to you? Open-ended questions allow the participant to respond freely, without limits imposed by the investigator or by the discrete answer choices from standardized questionnaires (Hulley et al., 2013). However, responses provided by a participant might not adequately capture or completely answer the question(s). It is vital to review the items with study team members and, if possible, by an individual similar to the target population to be studied before implementation to ensure they are clear and understood by the participants.

Answers can be handwritten by the participant or audio-taped by the investigator and transcribed verbatim. The text obtained is analyzed using qualitative methods to identify themes and meaningful associations. Open-ended questions are helpful for exploratory phases of research to assist investigators in understanding concepts as reported by participants (Hulley et al., 2013). A limitation of using open-ended questions is that more time is required to analyze the text-based data than responses with discrete answers. In continuing with the sleep study example, if the target population are adults who are 65 years and older with a history of chronic insomnia, the open-ended question presented above might provide insight into the perceived benefit of getting enough sleep for an older adult. Their responses might reveal information not previously recognized by researchers. An upcoming paper in this series will focus on qualitative research methods.

Administration: Open-ended Questions

Using interviews to obtain responses from an open-ended question is an excellent format to collect answers to complicated questions. However, the relationship between the interviewer and participant can influence the responses. The interviewing skills of the interviewer also impact the quality of the answers. The development of a standardized structure to implement the interviews can increase the consistency among all the participant interviews. Additionally, interviewers need to prevent introducing their preconceptions into the responses by changing the words of the questions or tone of their voice (Hulley et al., 2013). Interviewers must read the study questions precisely as written. Developing standard phrases to “probe” participants to clarify their responses can be written beneath the questions (Hulley et al., 2013). For example, asking, “What does good sleep mean to you?” the standard probe phrase might be, “Is this what you said?” to clarify the response.

For studies with multiple research personnel, conducting mock interviews before the start of the study is a method that allows for training, feedback, and consistency. Audio-taping all interview sessions and systematically auditing a portion of the interviews also permits ongoing supervision and the ability to assess adherence to the interview structure. An upcoming article in this series will address qualitative studies and the associated methods with the approach.

Closed-ended Questions

Close-ended questions provide a list of response choices and allow participants to select their responses faster. Additionally, the responses are easily entered into a spreadsheet for tabulation and analysis. The response options also assist in clarifying the meaning of the question and often used in questionnaires that produce a single score with an interpretation (i.e., Insomnia Severity Index, a score of 22 = severe clinical insomnia) (Hulley et al., 2013; Morin, Belleville, Bélanger, & Ivers, 2011). The close-ended style of questions has several limitations. Participants do not have the option to express their responses, possibly missing a more detailed response. The available choices might not include all possible responses (e.g., type of sleep aid used). “One option to consider is to add “Other (please specify)” or “None of the above” “(Hulley et al.,20013 p. 24). According to Hulley et al. (2013), if a question allows for more than one answer, the better format is to develop several questions that address the topic area and to ask participants to mark each response as either “yes,” or “no” to ensure the responses are adequately captured. See table 1A Question Styles: Close Ended for an example.

Table 1A:

Questions Styles: Close-Ended

Which of the following do you believe increases the chance of you falling asleep faster? Circle your response to each item.
Exercise daily	yes	no	do not know
Use a relaxing bedtime ritual	yes	no	do not know
Use your bedroom only for Sleep	yes	no	do not know
Sleep and wake at the same time every day	yes	no	do not know
Avoid bright light in the evening	yes	no	do not know

Another form of the close-ended question is the visual analog scale (VAS). VAS is a method used to measure a characteristic or attitude believed to exist along a continuum of values, such as the severity of insomnia (NCI, 2018). The VAS uses a straight line with word anchors on each side, describing the extremes of the characteristic (i.e., not worried to very much worried). To use a VAS, participants mark an “x” on the line that best represents their perception/response. The lines are usually 10 cm in length, providing a score range from 0 to 100, and the score is determined by measuring the distance (mm) from the lowest extreme to the center of the “x” using a ruler. If a participant marks an “x” at the 41mm point, the score is 41 (Hulley et al., 2013). See table 1B Visual Analog Scale.

Table 1B.

Visual Analog Scale:

The VAS is easy to use and rate characteristics on a continuous scale. This method is also more sensitive to slight changes in ratings compared to the nominal (data used for labeling or categorizing variables without a quantitative value such as gender) level of measurement (Hulley et al., 2013). A limitation of this format is the inability to administer via interview or telephone. Additionally, if using a paper version of a VAS and the questionnaire is photo-copied, the length of the 10 cm line may change. The length of the scale must be consistent throughout a study; therefore, a careful review of copied questionnaires for proper VAS line length is essential.

The Likert Scale is another close-ended question format used in clinical research to quantify abstract concepts like insomnia, attitudes, beliefs, and quality of life. It uses multiple questions or statements organized into a scale to measure a concept since it is challenging to evaluate concepts like insomnia or quality of life from a single item (Hulley et al., 2013). The word scales provide the participants with a list of questions or statements and instruct them to select the best response that represents the rank of their answer. Each word choice is assigned a specific number of points, and each instrument using a Likert Scale will have a scoring guide and associated meaning. A limitation of the multi-item questionnaire is that the results might be difficult to understand intuitively (i.e., the composite score for quality of life among older adults with insomnia = 48.2) (Hulley et al., 2013). Please see the article in this series titled Planning for Measurement in Nursing Research to review a questionnaire’s reliability and validity. See table 1C for the Likert Scale for Insomnia.

Table 1C.

Likert Scale for Insomnia

For each question, select the number that best describes your response. In the past seven days, please rate the severity of your insomnia.
Question	None	Mild	Moderate	Severe	Very Severe
Difficult time falling asleep	0	1	2	3	4
Difficult time staying asleep	0	1	2	3	4

Administration: Close-ended Questionnaires

Participants can complete questionnaires in person, through the mail, email, or the study website. Regardless of the method of administration, providing clear written instructions on how to complete the questionnaire is imperative. Additionally, the font size and color of the text may affect the legibility of the questionnaire and impact the completeness (Edwards, 2010). Moreover, pilot testing the questionnaire(s) among a similar population that will be studied can provide the investigator(s) with insight about the clarity and readability of the questions.

The advantage of completing questionnaires in person allows the investigator to provide verbal instructions and to review the completeness of the responses. For mailed questionnaires, it is vital to obtain the correct mailing address, provide a self-addressed stamped envelope for return, and include a return date for the responses. For investigators that are part of a research center wherein the name of the center could disclose a participant’s health status to members of their household, when mailed documents are received, for example, the Center for Addictive Behaviors, precautions are needed to protect their privacy. One approach to prevent accidental disclosures is for the investigator to only have their name on the return address of the envelope without the center’s name. Another option is to email questionnaires instead of mailing. Emailed questionnaires have advantages over mailed ones because they can be sent quickly and returned smoothly to the investigators. However, participants without emails, internet access, or lack familiarity with the internet can not use this method (Edwards, 2010).

Other options for delivering questionnaires include using online survey platforms such as REDCap, Survey Monkey, and Qualtrics. REDCap, Survey Monkey, and Qualtrics are HIPAA-compliant. Access to REDCap is free to members of the REDCap consortium, and nonprofit organizations can join for free (link for further details about joining https://projectredcap.org/about/faq/). Survey Monkey (www.surveymonkey.com) and Qualtrics (www.qualtrics.com) are commercial products available for use with a fee.

The advantages of online platforms are the ability to build the questions into the systems using developmental tools with functions to create matrix fields and items with branching logic. Questionnaires can be emailed to study participants or survey links created and posted onto study websites and accessible via smartphones, and tablets, making it convenient for participants to complete. The survey platforms can also tabulate the responses and the results downloaded to spreadsheets, removing the need for manual data entry.

Direct Measurement Methods

Alternatives to using questionnaires and interviews to obtain data and measure outcomes include using biometric measurement instruments, such as body weight scales and blood pressure monitors, collecting biological materials, and various imaging modalities (Gray, Grove & Sutherland, 2017; Saczynski et al., 2013). These assessments can directly measure common conditions or exposures more precisely versus asking about the number of hours of sleep from the previous evening.

Biological samples commonly collected include saliva, urine, stool, serum, and hair. These samples can assist in profiling study participants’ metabolic, proteomic, or genomic status, and understand underlying pathology or response to treatment or disease (Saczynski et al., 2013). Imaging technologies such as magnetic resonance imaging (MRI) and computed tomographic (CT) scanners can visualize underlying anatomic or pathologic mechanisms involved with the development of a disease, response to treatment, or prognosis (Saczynski et al., 2013). Similarly to biological assays, images from MRI or CT scans are read by a qualified expert such as a radiologist, and the results of the reading are the outcome and the direct measure.

A vital factor to consider is how often biological materials are collected during a study. The frequency will depend on the study design, the research question, the cost, feasibility, participant safety, and participant burden (i.e., invasiveness of the procedure (spinal taps), time requirement from the participant). The timing of specimen collection also requires knowledge if a sample exhibits circadian variability (early morning specimen collection versus evening collection) (Saczynski et al., 2013).

In developing the methods to collect data and measure outcomes, it is essential to consider the intensity of data collection (e.g., multiple blood draws, frequent visits to study site) with the ability to retain study participants in multi-session, longitudinal studies. Investigators also need to be alert to new technologies that can directly measure characteristics previously assessed only through questionnaires and interviews (Hulley et al., 2013). A future article in this series will address experimental designs in more detail.

Administration: Direct Methods

There are various direct approaches (e.g., physiological measurements, biological assays, imaging modalities) to collect data and measure research outcomes. The implementation will depend on the method selected. Regardless of the method chosen, written instructions with images (if appropriate) on how to perform the measurement is essential. Training and observing research staff and the study participant to conduct the procedure will assist in reducing systematic errors.

For the insomnia study, provide the study participants with written instructions and images on how to use the wrist actigraph properly. The directions should include when to wear and when to remove. Provide information on activating and turning off the actigraph, possible solutions to technical difficulties, when to contact the study team with technical issues, and when and how to return the actigraph. Actigraph instructions for research staff might include downloading the actigraph data to a computer for data storage and calibration, as well as observing them teach a mock study participant on using the actigraph.

Conclusion

The results of clinical research studies depend on the quality and appropriate use of measurement instruments to collect data that matches the aims of the research and answers the research question(s). Developing an operations manual or set of instructions to ensure research staff and study participants consistently follow the same methods to complete study instruments is crucial in reducing systematic study errors and standardizing the approach.