How to use Outcome Questionnaires: Pearls and Pitfalls

Abstract

Outcome questionnaires are increasingly used in the current medical practice. Patient reported outcomes serve as an essential and perhaps more relevant means for assessing patients’ response to treatment than clinical measures alone. Many of the procedures performed in plastic surgery are associated with aesthetic outcomes, therefore it is pertinent to thoroughly understand the patient’s perspective of achieved results. Surgeons need to possess the knowledge and skills about outcomes assessments and understand how to apply them to improve quality of care delivered based on evidence. This paper discusses the appropriate use of outcome questionnaires to rigorously evaluate treatment methods based on patient satisfaction and the outcome measurement instruments frequently used in plastic surgery.

OVERVIEW

Outcomes assessment is now an integral component of evaluating the success of various medical and surgical procedures in the evidence-based era. Rather than relying on traditional “hard” outcomes data such as how far one can walk after lower leg reconstruction or how much breast tissue is resected for in breast reduction surgery, physicians and patients are much more interested in patients’ perception of their functional improvement, quality of life, and satisfaction with treatment. Such appraisal is vital not only for clinicians but also to patients. Patients are constantly trying to derive maximum information from their surgeon with regards to the outcomes of the procedures they undergo. These inquiries extend beyond recovery and functional restoration. In plastic surgery, patients want to be reassured of other critical aspects of care such as, satisfaction, physical and social wellbeing, and aesthetic appearances as a result of an intervention.

Traditionally, outcomes would be measured in the form of assessments made by the treating plastic surgeon through photographs, anatomic measurements, and complications. However, the perception of results by a surgeon and patient differ. A plastic surgeon may be contented with the results obtained from his or her treatment but the patient may not be similarly pleased with the outcome achieved. Therefore, outcomes measured from patient’s view point are highly relevant because majority of the procedures performed in plastic surgery aim at improving physical appearance, body image, psychosocial function, and quality of life.¹ Acceptance by friends and family, emotional and mental satisfaction, confidence and happiness with appearance after an intervention influence quality of life outcomes.² The volume of plastic surgery procedures is huge, and ever increasing. Procedural statistics from the American Society of Plastic Surgeons (ASPS) showed that 5.5 million reconstruction procedures and 1.6 million cosmetic surgical procedures were performed in the year 2011 with an increase of 5% and 2% respectively over the year 2010.³ Therefore, subsequent assessment of outcomes from patients’ perspective is relevant in plastic surgery.

Measures to quantify the results in plastic surgery did not exist earlier, but in the last two decades several outcome questionnaires or surveys in the form of patient reported outcomes (PRO) were developed and used. However, all of these outcomes tools are not validated. Encouragingly, last decade has seen much progress in this area and attempts to develop more robust measurement tools continue. Plastic surgery is a unique field in which outcomes are not assessed by mortality and morbidity alone, therefore patient satisfaction and quality of life components take prime importance.² The future and success of this specialty depends heavily on the patients’ perception of their outcomes. The ultimate goal is to have outcome measures that incorporate patient satisfaction and all the quality of life measures that can potentially reflect the real effect of a surgical intervention. We intend to educate our readers about how to use these tools to measure patient satisfaction and outcomes achieved in a more meaningful and coherent manner. We also want to inform our readers about the common pearls and pitfalls encountered during the use of these questionnaires.

PROMIS and its development

Patient Reported Outcomes help to associate the outcomes achieved with the care provided from the patient’s perspective. Rising costs of health care and restricted funding environments lead surgeons to find cost-effective measures to sustain health care delivery for present and future. Outcome assessments with the aid of patient questionnaires can partially achieve this task. The federal government has devoted substantial funding for the initiative Patient Reported Outcomes Measurement Information System (PROMIS) under the National Institutes of Health guidance in 1994. The primary goal of this multicenter (12 sites) research project is to develop valid, reliable and standardized tools to assess PRO.⁴ PROMIS uses item banks to generate instruments that can be used as primary or secondary endpoints in clinical studies that evaluate treatment effectiveness. These outcome measures will help assess various chronic conditions so outcomes can be comparable across studies.

Types of available outcome questionnaires

PRO are obtained from patient interviews or questionnaires completed by patients during several follow-ups in the treatment process.⁵ Two types of questionnaires are available to use; generic questionnaires and disease specific questionnaires. Each questionnaire will have certain advantages and disadvantages associated with them because they were originally designed for different purposes. As a result it is important to differentiate between them before proceeding with their use.

Generic questionnaires are designed to assess the disease effect on the whole person irrespective of the medical condition. They are broad and can be used for an overall health assessment after an intervention, as an accompaniment to disease-specific questionnaires, and when disease specific questionnaires are not yet designed and available. For instance, Short Form 36 (SF-36) and Sickness Impact Profile (SIP) can be used in a variety of conditions. SF-36 is a widely used generic measure along with specific measures to assess 8 health domains.⁶ Generic measures incorporate various qualitative and quantitative aspects of human life. ⁷ Each questionnaire is unique, so they provide the researchers an opportunity to work with one or few questionnaires simultaneously and an ability to compare outcome results across different conditions.⁸ However, they lack the precision and sensitivity to detect specific changes after an intervention.

Disease specific questionnaires are designed to assess interventions in patient populations identified by a particular disease. They are more responsive than general questionnaires as they are sensitive to detect changes due to focused questions. They are useful to evaluate specific interventions and differences between two similar treatments. For example, Nasal Appearance and Function Evaluation Questionnaire (NAFEQ) can be used to assess functional and aesthetic outcomes after nasal reconstruction.⁹ Similarly Carpal Tunnel Questionnaire is a valid and reliable tool to assess symptom and functional changes after carpal tunnel surgery.¹⁰ MHQ is another valid questionnaire with 6 health domains that is used all over the world to evaluate outcomes in patients with hand conditions.¹¹ It also collects the data on unaffected hand to be used as a control for comparison of outcomes.

A disease-specific instrument is designed to assess specific interventions. However, when a specific instrument addresses all aspects of intervention but fails to consider quality of life domains such as psychosocial and sexual functioning, a generic instrument should be used as an accompaniment. For example, Breast-related Symptoms Questionnaire used to evaluate outcomes after breast reduction assesses breast symptoms only.¹² A generic questionnaire can be used to evaluate a specific treatment when a disease specific instrument is not available. For instance, Dolan et al. used SF-36 to assess health related quality of life outcomes after micro vascular free flap reconstruction.¹³ But the use of specific questionnaire to assess general health cannot accomplish the expected purpose because it fails to incorporate the items beyond the specific condition. A list of available outcome questionnaires in plastic surgery with their component scales and specific use is outlined in Table 1.

Table 1.

Available Questionnaires for Outcomes Assessment in Plastic Surgery

Name	Purpose	Component Scales/Items	Developed by
Breast
BREAST-Q1,23	To assess impact and effectiveness of breast surgery	3 modules-augmentation, reduction, and reconstruction. 6 scales: Psychosocial well-being, physical, and sexual well-being, satisfaction with breasts, satisfaction with outcome, and satisfaction with care	Pusic et al.
Breast Evaluation Questionnaire (BEQ)28,29	To assess patient satisfaction with breast attributes and quality of life outcome after breast surgery	55 items-Degree of comfort with size, appearance of the breasts, and satisfaction level achieved	Anderson et al.
Outcomes of Plastic Surgery (OPS)-hand/arm questionnaire4	To assess outcomes of plastic surgery of hand and arm	Symptoms, limitation of daily activities, psychological functioning/cosmetic appearance, and patient satisfaction
European Organization for Research and Treatment of Cancer Quality of Life questionnaire C301	Breast module (Br 23) items assess disease symptoms, side effects of treatment, body image, sexual functioning, and future perspectives	9 scales: 5 functional, 3 symptom and one global health-related quality-of-life scale Used to assess cancer patients
Face
FACE-Q30	To assess impact and effectiveness of facial aesthetic procedures	4 scales: Satisfaction with facial appearance, health related quality of life, negative sequelae, and satisfaction with process of care	Klassan et al.
Facial Injectables, Longevity, Late and Early Reactions and Satisfaction Questionnaire (FILLERS-Q)31	Physical and social experiences after treatment with injectable facial soft tissue fillers.	43 items: Patient demographics (4), Patient satisfaction with treatment (10), Procedure related events (3 to 7), impact on relationships (9–15), and economic considerations (3–7)	Sclafani et al.
Facial Clinimetric Evaluation Scale17	Measures Facial impairment and disability	!5 items, 6 domains: facial movement, facial comfort, oral function, eye comfort, lacrimal control, and social function Used to assess patients with facial paralysis	Baylor college of Medicine, Houston, Texas
Facial Disability index17	Measures disability and social and emotional well-being of facial paralysis patients	10 items, 2 domains-social/wellbeing function and physical function
Facial Lines Treatment Satisfaction15	To assess patient satisfaction with facial line treatment	14 items measuring facial line appearance, procedure satisfaction and patient confidence	Allergan
Facial Lines Outcome Questionnaire15	to measure hyperfunctional facial lines of the upper face	7 items	Allergan
Derriford Appearance scale (DAS)1,24	Physical and psychosocial aspects of facial and bodily appearance	6 measures of psychological distress and dysfunction and one measure of physical distress and dysfunction
Rhinoplasty Outcomes Evaluation (ROE)15,27	Used to assess patients after rhinoplasty surgery	6 items. 3 domains: Appearance, functional outcome and social acceptance	Alsaraff
Blepharoplasty Outcomes Evaluation (BOE)15,27	Used to assess patients after blepharoplasty surgery	3 domains: Appearance, functional outcome and social acceptance	Alsaraff
Facelift Outcomes Evaluation (FOE)15,27	Used to assess patients after facelift surgery	3 domains: Appearance, functional outcome and social acceptance	Alsaraff
Skin Rejuvenation outcomes Evaluation (SROE)15,27	Used to assess patients after skin resurfacing surgery	3 domains: Appearance, functional outcome and social acceptance	Alsaraff
Louisville Instrument For Transplantation (LIFT)32 Cunningham	Used to assess patients after composite tissue allo-transplantation surgery	Quality of life improvement, aesthetic and functional outcomes	Cunningham et al.
Glassgow Benefit Inventory(GBI)15,33	Measures general perception of well-being, social and physical well-being	18 items, General benefit scale, social support scale and physical health status scale Used to assess patients after head and neck surgery, especially functional and cosmetic rhinoplasty
Aesthetic Surgery
Multidimensional Body-States Relations Questionnaire (MBSRQ)11,34	To assess body image	Has psychological, body image and general questions 10 subscales to assess individuals satisfaction with five dimensions of body image
Body Dysmorphic Disorder Questionnaire (BDDQ)25	To assess body dysmorphic disorder	4 sets of questions	Phillips et al.
Dysmorphic Concern Questionnaire-DCQ25	Used to assess patients with concern on physical appearance	7 questions	Oosthuizen et al.

Factors affecting the selection of an appropriate questionnaire

Most clinicians are not aware of clinical usefulness of questionnaire to be used in their study, so user preference served as a guide to choose the instrument.^{7, 14} Several factors such as study sample, type of disease, and type of intervention need to be taken into account when selecting a questionnaire.⁵ The purpose of a questionnaire use should be clearly defined before its use to assess outcomes in plastic surgery. Quality and content of the instruments are other factors considered important in the selection for clinical use.¹⁵ To evaluate the outcome of a specific treatment performed on a single patient at different points of time or on a group of patients, a disease specific questionnaire is more applicable because it is more responsive to small changes with time. For example, BREAST-Q can be used for outcomes after breast reconstruction in breast cancer patients. To examine quality of care delivered and cost effectiveness of interventions in different scenarios, a generic questionnaire is more applicable such as, SF-36 or SIP. To compare outcomes among different studies or to estimate the use of resources, then a disease specific or generic questionnaires can be used respectively.⁷ Overall, the purpose of research or the outcome of interest to surgeons or an outcome important to patient determines the choice of a generic or disease specific or sometimes both questionnaires in a given situation.¹⁶ Table 1 provides a list of outcome instruments commonly used in plastic surgery.

Strategies to identify an ideal questionnaire

An ideal instrument used will yield accurate results in terms of demonstrating the true effect of an intervention. An instrument that demonstrates good reliability, validity, and responsiveness to change will be considered ideal to perform assessments. During the design and testing stages of an instrument, certain criteria need to be fulfilled for an ideal instrument to possess the above mentioned attributes. The criteria include, item development, item reduction, scale development, field testing, and psychometric evaluation.¹ Previous instruments developed for similar conditions can be referred to guide the development of items and scales in a new instrument while adapting to the patient population and condition in context. Guidelines established by the Scientific Advisory Committee of the Medical Outcomes Trust and U.S. F.D.A. regarding criteria evaluation can be used in the instrument development.^17–19 It is therefore critical to ensure that the instrument selected to use in the study has incorporated these steps in the design process. This can be achieved by reviewing at the literature regarding the development of the questionnaire. Reliability of an instrument refers to the ability to produce similar results upon repeat testing. Intraclass correlation coefficient measures this test-retest reliability, values greater than 0.9 are considered acceptable (range 0.0 to 1.0).²⁰ Individual items within a domain or scale are expected to correlate with one another referred to as internal consistency reliability. Minimum standard for this reliability coefficient is greater than 0.7 as measured by Cronbach’s α (range 0.0 to 1.0).^1,20

Additionally it would be advantageous if the instrument possesses all the domains of the PRO it is intended to measure. This is referred to as content validity, one of the two components of validity, which constitute the psychometric property of an instrument.¹ Involving patients in the item generation stage and field or pilot testing stage through interviews will provide a stronger content validity than just referring to literature or expert opinion as these are patient reported outcome measurement tools.²¹ Patient interviews help the surgeons learn about the information most important to patients that may be overlooked by surgeons. The results obtained with the use of a new instrument should then be compared with an existing standard or other widely used similar instrument to assess its performance. This component is the construct validity of an instrument. Although difficult to establish, well established construct validity adds to the value of a tool.

Responsiveness of an instrument is the ability to detect clinical changes in outcome. In evaluating treatments, responsiveness refers to the ability to identify the changes from preoperative to postoperative follow-up periods. It is commonly expressed in terms of effect size and minimal clinically important difference (MCID) of an instrument for that particular condition. In addition, responsiveness as determined by MCID helps in establishing the clinical significance of a study; if the study outcome scores are at or above the MCID, the study findings becomes clinically significant. As a result evaluating the responsiveness indirectly determines the clinical usefulness of an instrument.

The reliability and validity of ad-hoc questionnaires that are sometimes used cannot be ensured in evidence based practice because they are not scientifically developed and psychometrically tested.^21,22 Readers can perform a literature search to identify the psychometric properties of an instrument that were established in the studies conducted earlier and thus select an instrument that possesses good attributes.

Pearls

Although it is a laborious task to develop a questionnaire and establish its psychometric properties and at times practically difficult to incorporate its use into busy clinical practice, the use of outcome questionnaires endows plastic surgeons with several advantages. Scientifically devised and psychometrically tested instruments offer evidence based results in response to the outcome assessments. Often plastic surgeons encounter situations when several treatment methods for a condition are associated with similar outcomes and similar complications, and literature fails to recommend a procedure. PRO through questionnaires will guide the surgeons to choose a treatment method with which patients are more satisfied in these circumstances.¹ They also help to evaluate different treatments, differentiate between various approaches, and potentially compare the results obtained by different surgeons in a systematic way.²

Outcome measurement tools can be used to perform a cost effective analysis. Costs of health care are continually increasing, and in order to provide quality care without creating a huge burden on society, comparing the cost effectiveness of alternate procedures will help arrive at treatments that provide greater relief at less cost. For example, one of the commonly performed procedures in plastic reconstructive surgery is breast reconstruction with different types of flaps, TRAM flap, DIEP flap, and latissmus dorsi flap. Using BREAST-Q questionnaire to compare the outcomes achieved and cost incurred with the three types of flap will help identify the less expensive method to achieve quality reconstruction.

It is a good practice to administer the instruments that have been used in prior studies, or that have their development details elucidated in the articles. In the event of inability to find the conceptual background of a tool, reference into a study cited in the article for the development process of that specific tool can be done to obtain additional information. Each outcome questionnaire is unique and distinct from others, thereby permitting the use of an additional questionnaire when one questionnaire does not seem to cover all the domains of treatment outcomes.

Overall, a PRO measure developed based on the guidelines will allow the surgeon to compare techniques, quantify the positive effects, and identify potential candidates for appropriate procedures from a group of patients.¹⁵ It will function as a standard for future clinical trials. In addition, it will help the surgeons to have important patient feedback about the entire treatment experience that includes aspects beyond the procedure itself, such as, patient education, communication before and after the procedure.¹⁵

Pitfalls

As an accompaniment to the numerous advantages of an outcome questionnaire use, surgeons need to be cautious about pitfalls encountered during their use. Most important would be the selection of an instrument; an inappropriate tool used to make assessments will not be able to accomplish the purpose of its use. It is challenging to choose the suitable measurement tool among the myriad of existing ones that pertains to the target population being implemented. However, the goal can be fulfilled if the choice is based on selecting one that has been designed on conceptual framework and scientific background.

The use of instruments not specifically designed for certain type of population will not yield meaningful results when they were not involved in the pilot testing stage.² For instance, BREAST-Q is used to evaluate outcomes in women who had undergone breast reconstruction, but its use in women who had undergone lumpectomy or radiation may not be valid because these women were not represented in the initial design and development of the instrument.²³ It is important to ensure that the tools are used before, during and after the intervention is performed so that the true effect is captured.⁵ Specific attention should be made to the after treatment use because the time to follow-up differs based on the intervention and disease condition, and restricting to too short time period may fail to measure the real outcome.

Poor design, incorrect use, and misinterpretation of scores will lead to false inferences from a study; therefore well designed tools should be used according to the recommendations made by the developers of the instrument. Modification of questionnaires, such as adding or deleting items, rephrasing of questions and translation to other languages may affect its validity, hence the psychometric properties of the modified questionnaire should be tested again with the new items before it can be used for the intended purpose.⁵

Use of multiple questionnaires or too many questions in one instrument may add burden on patients, reduce compliance and require additional analysis.^7,8 Therefore it is necessary to inquire if the disease specific questionnaires are necessary for every plastic surgery condition or similar results can be obtained from a more generalized questionnaire before their use. For example, in a cohort of carpal tunnel syndrome patients, Kotsis and Chung had used only two questionnaires, MHQ and DASH and found that both questionnaires were responsive in measuring outcomes after carpal tunnel surgery.⁸ Therefore, in future studies, use of either the MHQ or the DASH should provide sufficiently valid data to evaluate outcomes of carpal tunnel surgery.

Pediatric population who may have different requirements with regards to item content and language need distinct consideration in choosing an appropriate outcome tool.²⁴ The Derriford Appearance scale used for measuring the physical and psychosocial aspects of facial and bodily appearance is one such questionnaire that is not applicable to the pediatric patients. Likewise, SF36 is a frequently used generic measure but cannot be used for patients under 14 years because it is not designed for that population.⁶

Conclusions

The process of instrument development is complex and requires rigorous methods to ensure that it possesses the necessary psychometric properties essential for the intended clinical purpose.²⁰ A systematic review by Pusic et al. found that only 7 of the 223 (3%) patient reported outcomes measures available in breast surgery were psychometrically tested for their use.¹² They identified the necessity to develop reliable and valid measures in cosmetic and reconstruction breast surgery. Similar measures in other subspecialties of cosmetic surgery were found to be lacking.^15,25–27 Outcome instruments that are systematically developed and validity tested need to be used in patients to assess surgical and nonsurgical interventions in a meaningful and responsive manner. Such an initiative will help to improve the understanding of the effectiveness of interventions and quality of care delivered to the patients.

The concept of using a psychometric scale in the form of questionnaires/surveys in routine clinical practice by a plastic surgeon is novel, but it is becoming standard practice due to its associated benefits. Such an increased use will help plastic surgeons in appropriate patient selection for procedures and also to evaluate outcomes after treatment and for research.²⁴ Identifying the appropriate instruments that can be applied in clinical practice also helps to compare between studies and treatments. An example of a well-designed outcomes instrument in plastic surgery is BREAST-Q by Pusic et al which adheres to the guidelines on outcome instrument development.²³

If well developed and validated tools are available for a condition, there is little need to develop new questionnaires except when they are simpler or provide enhanced information. In such an instance, it needs to be developed in accordance with the scientific structure. The recent shift in trend is to develop more region or disease specific as the general questionnaires are too broad and imprecise for specific conditions. So there should be a balance on how specific the questionnaires are developed. Instead, future efforts should focus on enabling the process of data collection and analysis through questionnaires simple enough to facilitate the regular use of these tools in clinical practice. Normative data need to be established for validated questionnaires to establish a reference when interpreting the scores from these tools. In the current era of outcomes assessment and evidence-based medicine, it is essential for plastic surgeons to keep well-informed about the latest developments in understanding the assessment tools available achieve enhanced patient satisfaction and quality of care.

Key Points.

• If well developed and validated tools are available for a condition, there is little need to develop new questionnaires except when they are simpler or provide enhanced information.

• Future efforts should focus on enabling the process of data collection and analysis through questionnaires simple enough to facilitate the regular use of these tools in clinical practice.

• In the current era of outcomes assessment and evidence-based medicine, it is essential for plastic surgeons to keep well-informed about the latest developments in understanding the assessment tools available achieve enhanced patient satisfaction and quality of care.

Abbreviations

DASH

MCID

Minimal clinically important difference

MHQ

NAFEQ

Nasal Appearance and Function Evaluation Questionnaire

PRO

Patient reported outcomes

PROMIS

Patient Reported Outcomes Measurement Information System

SF-36

Short Form 36

SIP

Sickness Impact Profile