Abstract
Background
Forehead lines (FL) are one of the main signs of aging. Traditional tools to measure FL have limited ability to evaluate the multidimensional impact of these lines on appearance, confidence, and psychological and social relationships.
Objectives
We developed and validated the Facial Line Distress Scale–Forehead Lines (FINE-FL) to evaluate the severity and psychosocial distress associated with FL.
Methods
We conducted a cross-sectional survey for FINE-FL psychometric validation at a tertiary hospital and 2 local clinics in Korea. First, a preliminary item pool for the FINE-FL was developed through a qualitative interview based on literature reviews and expert consultations. Second, cognitive interviews and a pilot test were conducted to evaluate comprehension, ease of response, acceptability of terminology, phrasing, and response options. FINE-FL consisted of 26 items. In this study, exploratory factor analysis was conducted to identify the underlying factor structure of the FINE-FL, and internal consistency and test-retest reliability were also examined.
Results
We found 21 items in 4 domains. The model fit was good. Coefficient αs ranged from 0.89 to 0.95 for subdomains and 0.96 for the total. The FINE-FL was moderately correlated with the appearance appraisal score. On the test-retest, the range of the intraclass correlation coefficient was 0.65 to 0.79.
Conclusions
FINE-FL is a reliable, valid, and comprehensive patient-reported outcome measure for assessing FL severity and distress. This will be helpful in determining a patient's eligibility for inclusion in a study and measuring primary or secondary effectiveness endpoints for forehead line treatment.
Forehead lines (FL) are horizontal transverse wrinkles that can run through the entire forehead. The horizontal forehead lines are one of the first manifestations of wrinkles on an aging face.1 Therefore it is one of the areas in which patients are most likely to be treated first, particularly females ages 30 to 34.2 Various treatments, such as injection procedures with fillers or botulinum toxin, have been developed to improve forehead lines.
With the increasing treatment of forehead lines, various tools for the accurate evaluation of the efficacy of these treatments have been developed. The traditional methods for quantitatively measuring the severity of forehead lines include skin imitation and 3D imaging.3,4 However, although forehead lines are related to changes in facial expressions, such as frowning or opening the eyes wide, or to individual psychological and social factors, traditional tools have limited ability to evaluate the multidimensional impact of these lines on appearance, confidence, and psychological and social relationships.5,6 Patient-reported outcome measurements (PROMs) recently been accepted as a suitable solution for evaluating multidimensional treatment satisfaction and success.7
Several PROMs, such as the Facial Line Satisfaction Questionnaire, have been developed to evaluate facial lines’ severity and psychosocial impact.4 Facial Line Outcomes and FACE-Q have been developed to assess both the severity of facial lines and their psychosocial impact.7-9 However, these measures are for general face lines. Although the Forehead Lines Severity Scale (FWS) was developed to classify the severity of forehead lines, it is insufficient for a comprehensive evaluation that considers various psychosocial factors.10 Therefore, we developed and validated the Facial Line Distress Scale–Forehead Lines (FINE-FL) to evaluate the severity and psychosocial distress associated with FL.
METHODS
Study Participants
This cross-sectional study was conducted at a tertiary hospital and 2 local clinics in Korea between April 18 and June 17, 2022. The eligibility criteria for study participation were as follows: (1) adults aged 18 years or older, (2) adults concerned or distressed about FL, and (3) adults able to read and write Korean. We excluded participants with severe dermal scarring on the upper face, severe cognitive impairment, or mental disorders. The study was advertised with posters online and on offline information boards. We provided the participants with $10 as an incentive to participate.
The study survey was conducted on paper. One-third of the participants inquired about retesting 3 to 7 days after their initial test. The study was approved by the Institutional Review Board of the Samsung Medical Center, Seoul, Republic of Korea (SMC-2021-07-166). All study participants provided informed consent.
Measurement
In this study, we developed the FINE-FL according to the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) checklist and the US FDA guidelines for PROMs.11,12 The development of the FINE-FL involved several steps. First, an expert group of 2 clinicians and 4 behavioral scientists conducted an extensive literature review and semistructured in-depth interviews with 25 adults who were more than18 years old and had concerns about FL. Based on the literature review and qualitative study, a total of 54 items were prepared for the initial version of the Facial Line Distress Scale (iFINE). Second, cognitive interviews were conducted with participants who met the eligibility criteria to evaluate comprehension, ease of response, acceptability of terminology, phrasing, and response options. Based on cognitive interviews and expert review, 26 items in 2 domains and 6 subscales (general, facial expression, certain situations, self-consciousness, psychological distress, and social distress) were prepared for the FINE-FL. A pilot test was conducted with 5 participants for in-depth interviews. During the test, the participants completed the questionnaire and were subsequently interviewed for feedback on content. Third, the FINE-FL was psychometrically validated to evaluate its reliability, validity, and responsiveness. In addition, the convergent and discriminant validity of the FINE-FL was tested. Participants who consented to participate in the quantitative survey completed it either on paper or online. The PROMs for health-related quality of life (HRQoL), body image, and appearance appraisal were compared with the FINE-FL. For HRQoL, the World Health Organization quality of life assessment instrument (WHOQOL-BREF) was included.13 The WHOQOL-BREF comprises 26 questions in 4 domains: physical (health), psychological, social relationships, environmental. The domain scores were calculated by multiplying the mean score of each domain by 4, according to the scoring manual.13 Questions in WHOQOL-BREF ask about the past 2 weeks.
The Body Image Scale (BIS) was utilized to measure body image. The BIS is a 10-item scale that evaluates affective, behavioral, and cognitive body image distress.14 On the 4-point rating scale, a higher total score indicated a higher level of body image distress. One example was “Have you been feeling self-conscious about your appearance?” with response options of (0) not at all, (1) a little, (2) quite a bit, and (3) very much. The time span covered the previous week. The item scores were obtained by summing the responses for all items, and the total score ranged from 0 to 30. The Merz Scale (MS) for aesthetics measured line severity. The MS is a 2-item numerical rating scale for evaluation of FL severity at rest and dynamic.15 The items were rated on a 5-point Likert scale, with answers ranging from no FL (0) to 4 (very severe). Higher scores indicated more severe FL.
Additional sociodemographic and clinical information, including age, sex, marital status, education, monthly family income, employment, and FL treatment experience, was obtained.
Statistical Analysis
Participant characteristics were reported with descriptive statistics. The mean and standard deviation (SD) were reported for each item on the FINE-FL scale. An exploratory factor analysis (EFA) was performed to reduce the number of items. Varimax rotation was utilized for the EFA. Maximum likelihood methods determined whether or not the Tucker-Lewis Index (TLW) of factoring reliability was over 0.9 to confirm the adequacy of the number of factors.
The item response theory (IRT) graded response model (GRM) was employed for the distress domain. Differential item functioning (DIF) analyses were performed to evaluate whether participants from different groups, given similar levels of distress due to FL, had different probabilities of providing a response after conditioning at the level was measured.
After extracting the factor structure, we performed a confirmatory factor analysis (CFA) of the final items with the maximum likelihood without missing values to test whether our factor structure fit the data.16 Several goodness-of-fit indices evaluated the model fit, including the comparative fit index (CFI) and the standardized root mean squared residual (SRMR). A CFI >0.9 and SRMR <0.08 indicated a good fit to the data.6,17
To assess the internal consistency of the FINE-FL, we calculated the internal consistency of each domain with Cronbach's α and the item-total correlation of each domain. An α value of 0.8 or higher indicated very good reliability.18
To test the criterion validity, we calculated the area under the curve (AUC) of the FINE-FL for severe FL evaluated by a physician with the Merz Scale. For the analysis, “severe forehead line” and “very severe forehead line” were recorded as the severe FL group. We also applied the Youden index to determine the cutoff values.19
The FINE-FL test-retest reliability was measured with the intraclass correlation coefficient (ICC), with a 2-way mixed model with absolute agreement specified during repeated measurements. A questionnaire was considered reliable if its ICC value was > 0.70.20 With the repeated measure data set, the standard error of measurement (SEM) was also calculated by creating a variable for the difference between the scores obtained during the first and second administrations (test score-retest score = difference).21 Next, we calculated the smallest detectable change (SDC), which was the change in the instrument's score beyond the measurement error from the SEM. The SDC can be calculated by SEM × 1.96×√2.22
To test the hypothesis of construct validity, we calculated convergent and discriminant validity with Pearson's correlations between the FINE-FL and the WHOQOL-BREF, BIS, and Merz scales. We hypothesized that FINE-FL would exhibit negative correlations with WHOQOL-BREF scores within the range of −0.70 to −0.30, indicating convergent validity. Additionally, we expected FINE-FL to demonstrate positive correlations with both the BIS and Merz Scale scores within the range of 0.30 to 0.70, further supporting convergent validity. To account for multiple comparisons, we applied the Holm-Bonferroni method to adjust the obtained P values.
The significance level was set at P < .05 (2-sided), and all statistical analyses were performed with STATA version 16 (StataCorp LP, College Station, TX) and R 4.1.2 (R Foundation for Statistical Computing, Vienna, Austria).
RESULTS
Study Participants
Among the 202 patients, we excluded 1 patient who did not complete the questionnaires. Participants’ mean age was 47.41 (SD 13.82) years (range 24-83); 74.3% (n = 150) were female. Most participants had mild or more severe FL, and 33.7% had experience with facial aesthetic treatments (Supplemental Table 1, available at www.aestheticsurgeryjournal.com).
Item Reduction: Exploratory Factor Analysis
All 26 items satisfied Bartlett's test for sphericity (P < .01) and the Kaiser-Meyer-Olkin (KMO) test for sampling adequacy (P = .94). Exploratory factor analysis indicated a 5-factor solution with an eigenvalue >1.0, although it was initially designed as a 6-factor solution.
For the individual items, one item loaded significantly on the interpretable factor solution. Two items (22 and 23) with low loading values (r < 0.5) were excluded. Although 2 items (13 and 16) met the factor-loading criteria, the expert review led to their exclusion due to their similarity in meaning to other items. Additionally, item 22 was excluded based on an expert review because it was deemed to have potential differences in interpretation by sex. Items 9 and 10 were initially thought to belong to a specific situational domain. However, the factor-loading analysis revealed that they also had higher loadings in the domain of psychosocial distress. After extensive discussion among the researchers, it was decided to keep these items in the original domain, because they were intended to specifically address the severity of forehead lines in certain situations (Table 1).
Table 1.
Factor Loadings From the Exploratory Factor Analysis and Reliability of the FINE–FLa Dimensions (n = 202)b
| Item | Factor | Decision | ||||||
|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||||
| Appraisal of FL | ||||||||
| General | ||||||||
| #1 A lot of forehead lines. | 0.77 | General | ||||||
| #2 Forehead lines stand out. | 0.72 | General | ||||||
| #3 The depth of the forehead lines. | 0.83 | General | ||||||
| #4 The length of the forehead lines. | 0.83 | General | ||||||
| Facial expression | ||||||||
| #5 A lot of forehead lines. | 0.78 | 0.40 | Facial expression | |||||
| #6 Forehead lines stand out. | 0.74 | 0.46 | Facial expression | |||||
| #7 The depth of the forehead lines. | 0.82 | Facial expression | ||||||
| #8 The length of the forehead lines. | 0.79 | Facial expression | ||||||
| Certain situation | ||||||||
| #9 I am concerned about the forehead lines when talking. | 0.49 | 0.40 | Certain situation | |||||
| #10 I am concerned about the forehead lines under bright light. | 0.58 | 0.44 | Certain situation | |||||
| #11 I am concerned about the forehead lines when I look in the mirror. | 0.66 | Certain situation | ||||||
| #12 I am concerned about the forehead lines when raising my eyebrows and opening my eyes widely. | 0.82 | Certain situation | ||||||
| #13 I am concerned about the forehead lines when I frown. | 0.66 | Deletedc | ||||||
| Distress due to FL | ||||||||
| Self-consciousness | ||||||||
| #14 I look older than my peers because of my forehead lines. | 0.40 | 0.54 | Self-consciousness | |||||
| #15 I look angry because of my forehead lines. | 0.73 | Self-consciousness | ||||||
| #16 I look depressed because of my forehead lines. | 0.81 | Deletedc | ||||||
| #17 I look worried because of my forehead lines. | 0.74 | Self-consciousness | ||||||
| #18 I look like I have a strong impression or character because of my forehead lines. | 0.66 | Self-consciousness | ||||||
| Psychological distress | ||||||||
| #19 I have low self-confidence because of my forehead lines. | 0.59 | 0.48 | Psychosocial distress | |||||
| Social distress | ||||||||
| #20 I am concerned about meeting new people because of my forehead lines. | 0.76 | 0.46 | Psychosocial distress | |||||
| #21 I am concerned that forehead lines will affect my social life. | 0.73 | 0.46 | Psychosocial distress | |||||
| #22 I am concerned about going out without makeup because of my forehead lines. | 0.70 | 0.46 | Deletedc | |||||
| #23 I avoid sitting close to others because of my forehead lines. | 0.79 | Psychosocial distress | ||||||
| #24 I use cosmetics to improve forehead lines. | 0.46 | Deleted | ||||||
| #25 I don't make facial expressions that might make forehead lines. | 0.46 | Deleted | ||||||
| #26 I can't have different hairstyles because of the lines on my forehead. | 0.51 | Psychosocial distress | ||||||
aFINE-FL, Facial Line Distress Scale-Forehead Lines; FL, Forehead Lines.
bThe values are greater than 0.4, and the highest loading value is highlighted in bold.
cDeleted based on the expert review.
Confirmatory Factor Analysis
Further examination of the factor structure of the 21-item FINE-FL was evaluated with the CFA, which revealed high loadings (0.69-0.93) in general. The fit indices for the model were good: CFI = 0.902; SRMR = 0.061 (Figure 1).
Figure 1.
Confirmatory factor analysis.
The final version of the FINE-FL was confirmed with 21 items in 2 domains (appearance appraisal: general, facial expression, and certain situations; and distress due to FL: self-consciousness and psychosocial distress) and 5 subscales. The FINE-FL score was calculated and ranged from 0 to 4 by summing the responses of the items in each domain. Higher scores indicated higher levels of distress due to FL.
Internal Consistency
With the final 21 items of the FINE-FL, the possible score range was 0-100 points. The mean total score for the FINE-FL was 47.62 (SD 20.41). In our study participants, floor and ceiling effects were 3.0% and 0.5%, respectively. Cronbach's α coefficients of the 5 subscales ranged from 0.89 to 0.95, indicating satisfactory internal consistency. Cronbach's α coefficient of the total score was 0.96 (Supplemental Table 2 available at www.aestheticsurgeryjournal.com).
Criterion Validity
When the physician evaluation was performed with the MS, 44.1% of the participants had severe FL. A difference of 18.0 points on the FINE-FL was noted between participants with severe FL and those without, as evaluated by the physician (39.7 vs 57.7; P < .01) (data not shown). The accuracy of the FINE-FL score in predicting severe FL was characterized by an AUC of 0.76 (Figure 2). The cutoff value for severe FL was 52.0 out of 100.
Figure 2.
Criterion validity: receiver-operating characteristic (ROC) curves of FINE-FL for severe forehead lines evaluated by physicians. AUC, area under the curve; FINE-FL, Facial Line Distress Scale–Forehead Lines.
Test-retest
There were 86 patients who returned responses for the retest. There were no patients who experienced events that affected the retest results. When we performed the analysis to test reliability, the range of the ICC was 0.65 to 0.79, which was satisfactory consistency. The SEM of the total score was 9.41, and the SDC was 18.44 (Table 2).
Table 2.
Reliability of the FINE–FL (n = 86)
| Subscales | ICC (95% CI) | SEM | SDC |
|---|---|---|---|
| Appraisal of FL | 0.65 (0.53-0.74) | 6.72 | 13.17 |
| General | 0.68 (0.58-0.77) | 2.48 | 4.87 |
| Facial expression | 0.55 (0.42-0.67) | 2.88 | 5.65 |
| Certain situation | 0.59 (0.46-0.70) | 2.58 | 5.05 |
| Distress due to FL | 0.79 (0.70-0.85) | 3.87 | 7.58 |
| Self-consciousness | 0.74 (0.65-0.81) | 2.17 | 4.55 |
| Psychosocial distress | 0.76 (0.66-0.84) | 2.17 | 4.26 |
| Total FINE–FL | 0.74 (0.65-0.81) | 9.41 | 18.44 |
CI, confidence interval; FINE-FL, Facial Line Distress Scale–Forehead Lines; FL, forehead lines; ICC, intraclass coefficient; SDC, smallest detectable change; SEM, standard error of measurement.
Hypothesis Testing for Construct Validity
In the correlations, the self-evaluation of FL at rest and dynamic in the FINE-FL was moderately correlated with the Merz Scale at rest (general, r = 0.72; facial expression, r = 0.58) and the Merz Scale dynamic (general, r = 0.61; facial expression, r = 0.73). The FINE-FL subscales of the distress domain were weakly correlated with body image. In addition, WHO quality of life was moderately correlated with the psychosocial distress subscale in the FINE-FL, and there was a weak correlation with other subscales in the FINE-FL (Table 3).
Table 3.
Correlation of the FINE–FL With Legacy Measures (n = 202)
| Legacy measures | ||||
|---|---|---|---|---|
| Subscales | WHOQOL-BREF | BIS | MS at rest | MS at dynamic |
| Appraisal of FL | −0.269a | 0.151b | 0.638a | 0.670a |
| General | −0.294a | 0.175b | 0.716a | 0.606a |
| Facial expression | −0.153b | 0.051 | 0.584a | 0.732a |
| Certain situation | −0.278a | 0.185a | 0.402a | 0.447a |
| Distress due to FL | −0.469a | 0.324a | 0.514a | 0.391a |
| Self-consciousness | −0.395a | 0.285a | 0.501a | 0.406a |
| Psychosocial distress | −0.489a | 0.327a | 0.472a | 0.335a |
| Total FINE–FL | −0.376a | 0.238a | 0.634a | 0.601a |
a P < .01; bP < .05. BIS, Body Image Scale; FINE-FL, Facial Line Distress Scale–Forehead Lines; FL, forehead lines; MS, Merz Scale; WHOQOL-BREF, World Health Organization quality of life assessment instrument.
DISCUSSION
In this study, we developed the FINE-FL, a reliable and valid tool for evaluating the severity of psychosocial distress due to FL, following the COSMIN and US FDA guidelines for PROM development. The content validity of the FINE-FL was confirmed. The goodness-of-fit indices of the FINE-FL were also high. Moderate correlations demonstrated the convergent validity of the FINE-FL with the relevant questionnaires.
The FINE-FL consists of 2 subparts: an appraisal of lines (general, facial expression, and certain situations) and the impact of lines on patient distress (public self-consciousness and psychosocial distress). Although the internal consistency reliability of the measure was high, some items were loaded in a different domain than we initially hypothesized. In the appraisal domain, there was an initial attempt to separate these items into general and facial expression subscales. However, the exploratory factor analysis results indicated that they loaded onto a single factor. FL often exhibited deep furrows even at rest, indicating a strong correlation between facial expressions and their baseline state.2,5 Additionally, items 9 and 10 were initially thought to belong to a certain situation domain. However, the factor loading analysis revealed that they also had higher loadings in the domain of psychosocial distress. The appraisal of FL included items to assess the severity of FL not only generally but also in situations in which people experienced concerns or distress due to FL, as during facial expressions or in certain situations, such as talking, looking in the mirror, and concentrating. These reflected real situations in which people were concerned about the FL, associated with the high content validity of the FINE-FL. We performed in-depth qualitative interviews, cognitive interviews, and pilot tests to develop the FINE-FL. In fact, we discovered that the participants had varying interpretations of general questions regarding facial lines, leading to inconsistent responses. Measures were taken to design FL-related problems and improve the ability to respond to the questionnaire.
Recognizing FL severity suggests a close association between psychological and social stress.23 The accuracy of the FINE-FL score in predicting severe FL was characterized by an AUC of 0.76. Therefore, specialized tools that assess both psychological and social distress specific to forehead wrinkles, such as the FINE-FL, are expected to be useful. Moderate correlations were demonstrated, indicating the convergent validity of the FINE-FL with the relevant questionnaires. Specifically, the self-evaluation of the FINE-FL showed a moderate correlation with the Merz Scale, whereas the FINE-FL subscales of the distress domain showed a weak correlation with body image. This is because the FINE-FL is a specific tool designed to measure appraisal and distress related to FL, not general body image or overall quality of life. When we conducted qualitative interviews, we found that we could not measure FL-specific distress with a general question on the upper facial lines. Therefore, the FINE-FL can measure unique aspects of FL-related distress, and FL severity is a useful tool.
In this study, all participants completed all questions, which is a higher completion rate than that reported in other studies. Because more than 25.2% of the study participants were over 60 years of age, the FINE-FL seems to be a feasible instrument for evaluating the impact of FL, regardless of literacy.
This study had several limitations. First, because we recruited only Koreans, these results cannot be generalized to patients from other settings. However, in this study, we recruited people from various groups who responded that they were concerned about forehead lines, including those who visited the hospital for Botox, general office workers, and housewives. The recruitment of participants at various locations was intended to capture a wider range of potential participants for the FINE-FL. Considering the characteristics of the study participants, the FINE-FL has acceptable measurement properties for use in patients from diverse backgrounds. Second, the study did not include an existing questionnaire on distress due to lines to confirm convergent validity. Third, because the recruitment of participants occurred in a local community outside the hospital, the recruited researchers may have introduced some bias. Researchers may have missed uninterested participants during patient recruitment even though they were trained on the protocol before patient recruitment.
CONCLUSIONS
In conclusion, this study adds to the evidence supporting the FINE-FL's reliability and validity in assessing Korean speakers’ quality of life. The FINE-FL is an efficient means of assessing FL severity and its impact on a broad range of quality of life domains. Because the FINE-FL is a comprehensive tool for measuring patients’ forehead line distress, not specific to the effectiveness of treatment for the lines, researchers and clinicians can utilize it to evaluate the current status and effectiveness of treatment.
Supplementary Material
Acknowledgments
Dr E Kang and Dr D Kang contributed equally to this work as first coauthors.
Supplemental Material
This article contains supplemental material located online at www.aestheticsurgeryjournal.com.
Disclosures
The authors declared no potential conflicts of interest with respect to the research, authorship, and publication of this article.
Funding
This study was supported by Medytox Inc. (Seoul, South Korea).
REFERENCES
- 1. Luebberding S, Krueger N, Kerscher M. Quantification of age-related facial wrinkles in men and women using a three-dimensional fringe projection method and validated assessment scales. Dermatol Surg. 2014;40(1):22–32. doi: 10.1111/dsu.12377 [DOI] [PubMed] [Google Scholar]
- 2. Narurkar V, Shamban A, Sissins P, Stonehouse A, Gallagher C. Facial treatment preferences in aesthetically aware women. Dermatol Surg. 2015;41(Supplement_1):S153–S160. doi: 10.1097/dss.0000000000000293 [DOI] [PubMed] [Google Scholar]
- 3. Hersant B, Abbou R, SidAhmed-Mezi M, Meningaud JP. Assessment tools for facial rejuvenation treatment: a review. Aesthetic Plast Surg. 2016;40(4):556–565. doi: 10.1007/s00266-016-0640-y [DOI] [PubMed] [Google Scholar]
- 4. Pompilus F, Burgess S, Hudgens S, Banderas B, Daniels S. Development and validation of a novel patient-reported treatment satisfaction measure for hyperfunctional facial lines: facial line satisfaction questionnaire. J Cosmet Dermatol. 2015;14(4):274–285. doi: 10.1111/jocd.12166 [DOI] [PubMed] [Google Scholar]
- 5. Heckmann M, Teichmann B, Schröder U, Sprengelmeyer R, Ceballos-Baumann AO. Pharmacologic denervation of frown muscles enhances baseline expression of happiness and decreases baseline expression of anger, sadness, and fear. J Am Acad Dermatol. 2003;49(2):213–216. doi: 10.1067/s0190-9622(03)00909-5 [DOI] [PubMed] [Google Scholar]
- 6. Ximénez C. Recovery of weak factor loadings in confirmatory factor analysis under conditions of model misspecification. Behav Res Methods. 2009;41(4):1038–1052. doi: 10.3758/BRM.41.4.1038 [DOI] [PubMed] [Google Scholar]
- 7. Kosowski TR, McCarthy C, Reavey PL, et al. A systematic review of patient-reported outcome measures after facial cosmetic surgery and/or nonsurgical facial rejuvenation. Plast Reconstr Surg. 2009;123(6):1819–1827. doi: 10.1097/PRS.0b013e3181a3f361 [DOI] [PubMed] [Google Scholar]
- 8. Yaworsky A, Daniels S, Tully S, et al. The impact of upper facial lines and psychological impact of crow's feet lines: content validation of the Facial Line Outcomes (FLO-11) Questionnaire. J Cosmet Dermatol. 2014;13(4):297–306. doi: 10.1111/jocd.12117 [DOI] [PubMed] [Google Scholar]
- 9. Pusic AL, Klassen AF, Scott AM, Cano SJ. Development and psychometric evaluation of the FACE-Q satisfaction with appearance scale: a new patient-reported outcome instrument for facial aesthetics patients. Clin Plast Surg. 2013;40(2):249–260. doi: 10.1016/j.cps.2012.12.001 [DOI] [PubMed] [Google Scholar]
- 10. Carruthers A, Carruthers J, Hardas B, et al. A validated grading scale for forehead lines. Dermatol Surg. 2008;34(supplement_2):S155–S160. doi: 10.1111/j.1524-4725.2008.34364.x [DOI] [PubMed] [Google Scholar]
- 11. Mokkink LB, Prinsen CA, Patrick DL, et al. COSMIN study design checklist for patient-reported outcome measurement instruments. 2019:4-–16..
- 12. Services USDoHaHAdministration FaD(CDER) CfDEaR(CBER) CfBEaR(CDRH) CfDaRH . Guidance for industry: patient-reported outcome measures—Use in medical product development to support labeling claims. 2009:7–20. doi: 10.1186/1477-7525-4-79 [DOI]
- 13. Development of the World Health Organization WHOQOL-BREF quality of life assessment. The WHOQOL Group. Psychol Med. 1998;28(3):551–558. doi: 10.1017/s0033291798006667 [DOI] [PubMed] [Google Scholar]
- 14. Hopwood P, Fletcher I, Lee A, Al Ghazal S. A body image scale for use with cancer patients. Eur J Cancer. 2001;37(2):189–197. doi: 10.1016/s0959-8049(00)00353-1 [DOI] [PubMed] [Google Scholar]
- 15. Flynn TC, Carruthers A, Carruthers J, et al. Validated assessment scales for the upper face. Dermatol Surg. 2012;38(2 Spec No.):309–319. doi: 10.1111/j.1524-4725.2011.02248.x [DOI] [PubMed] [Google Scholar]
- 16. Bryant FB, Yarnold PR. Principal-components analysis and exploratory and confirmatory factor analysis. In: Reading and Understanding Multivariate Statistics. American Psychological Association; 1995:99–136. [Google Scholar]
- 17. Rubinshtein R, Halon DA, Gaspar T, et al. Usefulness of 64-slice cardiac computed tomographic angiography for diagnosing acute coronary syndromes and predicting clinical outcome in emergency department patients with chest pain of uncertain origin. Circulation. 2007;115(13):1762–1768. doi: 10.1161/CIRCULATIONAHA.106.618389 [DOI] [PubMed] [Google Scholar]
- 18. Ursachi G, Horodnic IA, Zait A. How reliable are measurement scales? External factors with indirect influence on reliability estimators. Proc Econ Finance. 2015;20:679–686. doi: 10.1016/S2212-5671(15)00123-9 [DOI] [Google Scholar]
- 19. Ruopp MD, Perkins NJ, Whitcomb BW, Schisterman EF. Youden index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J. 2008;50(3):419–430. doi: 10.1002/bimj.200710415 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. de Souza JA, Yap BJ, Wroblewski K, et al. Measuring financial toxicity as a clinically relevant patient-reported outcome: the validation of the comprehensive score for financial toxicity (COST). Cancer. 2017;123(3):476–484. doi: 10.1002/cncr.30369 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Kenaszchuk C, MacMillan K, van Soeren M, Reeves S. Interprofessional simulated learning: short-term associations between simulation and interprofessional collaboration. BMC Med. 2011;9(1):29. doi: 10.1186/1741-7015-9-29 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Ries JD, Echternach JL, Nof L, Gagnon Blodgett M. Test-retest reliability and minimal detectable change scores for the timed “up & go” test, the six-minute walk test, and gait speed in people with Alzheimer disease. Phys Ther. 2009;89(6):569–579. doi: 10.2522/ptj.20080258 [DOI] [PubMed] [Google Scholar]
- 23. Gupta MA, Gilchrest BA. Psychosocial aspects of aging skin. Dermatol Clin. 2005;23(4):643–648. doi: 10.1016/j.det.2005.05.012 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.