Skip to main content
PMC home page
Health and Quality of Life Outcomes logoLink to Health and Quality of Life Outcomes
. 2007 Jul 30;5:46. doi: 10.1186/1477-7525-5-46

Injection Drug User Quality of Life Scale (IDUQOL): Findings from a content validation study

Anita M Hubley 1,✉, Anita Palepu 2,3
PMCID: PMC1994946  PMID: 17663783

Abstract

Background

Quality of life studies among injection drug users have primarily focused on health-related measures. The chaotic life-style of many injection drug users (IDUs), however, extends far beyond their health, and impacts upon social relationships, employment opportunities, housing, and day to day survival. Most current quality of life instruments do not capture the realities of people living with addictions. The Injection Drug Users' Quality of Life Scale (IDUQOL) was developed to reflect the life areas of relevance to IDUs. The present study examined the content validity of the IDUQOL using judgmental methods based on subject matter experts' (SMEs) ratings of various elements of this measure (e.g., appropriateness of life areas or items, names and descriptions of life areas, instructions for administration and scoring).

Methods

Six SMEs were provided with a copy of the IDUQOL and its administration and scoring manual and a detailed content validation questionnaire. Two commonly used judgmental measures of inter-rater agreement, the Content Validity Index (CVI) and the Average Deviation Mean Index (ADM), were used to evaluate SMEs' agreement on ratings of IDUQOL elements.

Results

A total of 75 elements of the IDUQOL were examined. The CVI results showed that all elements were endorsed by the required number of SMEs or more. The ADM results showed that acceptable agreement (i.e., practical significance) was obtained for all elements but statistically significant agreement was missed for nine elements. For these elements, SMEs' feedback was examined for ways to improve the elements. Open-ended feedback also provided suggestions for other revisions to the IDUQOL.

Conclusion

The results of the study provided strong evidence in support of the content validity of the IDUQOL and direction for the revision of some IDUQOL elements.

Background

In the health and medical fields, quality of life (QoL) is widely used to evaluate social and clinical interventions, treatment side effects, and disease impact over time [1,2]. Most of these QoL instruments tend to focus on health-related quality of life (HRQOL) or the functional effects of respondents' perceived mental and physical health [3]. Gill and Feinstein [3], however, defined QoL as a reflection of respondents' perceptions and reactions to not only their mental and physical health, but also to non-health related aspects of their lives (e.g., family, friends, work). Thus, measurement of QoL needs to encompass more than just the health-related aspects of respondents' lives.

Nearly all studies of QoL in injection drugs users (IDUs) use measures of HRQOL [4-8]. Commonly used measures of HRQOL with this population include the Opiate Treatment Index [9], Nottingham Health Profile [10,11], the Berlin Quality of Life Profile [12] and the MOS surveys (including the SF-36 and the SF-12) [13-16]. Many studies have shown that IDUs experience significantly lower HRQOL relative to the general population [16-20] but, as noted by Fernández Miranda [21], remarkably little published research has examined QoL as an outcome variable in the treatment of drug addiction [22,23].

Although previous research with IDUs and related populations (e.g., illicit drug users, HIV/AIDS) has considered the effects of non-health related aspects of respondents' lives on their HRQOL [4,7,15,24-28] or even on the initiation or maintenance of drug use [29-35], rarely has published research with IDUs used a broadly-defined QoL measure (i.e., one that captures various social, psychological, physical, geographic, and occupational domains of QoL). Two exceptions would be Wasserman and colleagues [36], who examined the psychometric properties of Lehman's [37] Quality of Life Interview – Brief Version with IDUs, and Dunaj and Kovác [38], who compared convicted drug addicts and controls on broadly-defined QoL using the WHOQOL-BREF [39] and ComQol-A5 [40]. Dunaj and Kovác reported that addicts scored significantly lower than controls in their subjective ratings of areas such as health, emotional well-being, safety, and social standing.

Although broader measures of QoL are beginning to be used with IDUs [36,38] and are certainly an improvement over the use of strictly HRQOL measures, measures developed specifically for the IDU population and using a context sensitive approach that considers the many life areas deemed by IDUs as critical to their QoL, are still needed. QoL, as defined by the World Health Organization Quality of Life (WHO-QOL) group, refers to "an individual's perceptions of their position in life in the context of the culture and value systems in which they live, and in relation to their goals, expectations, standards, and concerns" (pp. 1–2) [41]. The item content and methods of administration for most available QoL instruments do not measure the QoL of drug users in a culturally-sensitive fashion [42]. IDUs live in a distinct environment characterized by a high prevalence of infectious disease, crime, violence, and lack of stable housing. Many IDUs cannot depend on basic necessities and experience considerable instability in many aspects of their lives.

A recently developed broadly-defined QoL measure, the Injection Drug User Quality of Life (IDUQOL) scale, was designed to capture the health and non-health related aspects of IDUs' lives that would be important components of their quality of life, particularly given their individual circumstances and environment [43,44]. This measure has also been adapted for use in Spanish with injection and non-injection drug users [45]. To use an instrument with confidence, it is important that there be evidence of validity – that is, the meaningfulness, usefulness, and appropriateness of an instrument for a given population in a given context [46-48]. Previous research has examined the factor structure, internal consistency, and test-retest reliability of scores from the IDUQOL as well as the criterion-related, convergent, and discriminant validity of inferences made from the measure [44]. Content validity, a critical step in the test development and validation process [49-51], refers to the degree to which elements of an assessment tool are representative of the construct of interest and appropriate for a given population [52]. Importantly, the elements of interest in a content validation study are not just the content or items of the measure, but all elements of the instrument including the instructions, response format, and scoring procedures [53].

The purpose of the present study was to examine the content validity of the IDUQOL using judgmental methods based on subject matter experts' (SMEs) ratings of the IDUQOL title, items, instructions, response format, scoring procedures, and record form.

Methods

Participants

The sample consisted of a panel of six subject matter experts (SMEs; 50% male), all of whom were researchers working in the area of drug use in the United States or Canada with an average of 10 years experience in the field. Two SMEs were epidemiologists and four SMEs were physicians who also provided addiction and medical care to drug users in their clinical practice. As noted by others [54], there is no set number of SMEs required for content validation studies. Typically, somewhere between three to ten experts is recommended, although a minimum of five SMEs is recommended to control for chance agreement; furthermore, the larger the number of experts, the greater the confidence in the ratings and the easier it is to detect rater outliers [53,55].

Measures

The subject matter experts were provided with a copy of the IDUQOL and its administration and scoring manual and a detailed content validation questionnaire. Ethics approval for this study was obtained from the University of British Columbia and Providence Health Care Research Ethics Boards.

Injection Drug User Quality of Life Scale (IDUQOL)

The original IDUQOL, which includes both health and non-health related aspects of QoL [3] and is based on the WHO-QOL group definition of QoL [41], consisted of 20 life areas. Several of these areas (e.g., Drugs, Drug Treatment, Harm Reduction and Neighbourhood Safety) were included in the measure precisely because of their particular relevance to the social and physical reality of IDUs as confirmed by focus groups during the development phase [43]. Each IDUQOL life area is represented on a 4 by 4 inch card, with the name of the area printed on the front along with a simple picture. A description of the life area is presented on the back (see Table 1 for a list of all 20 life areas and descriptions). Graphic representation of the life areas is intended to make the instrument more accessible to individuals who have low literacy skills or do not speak English as a first language. When administering the IDUQOL, the interviewer starts by showing the respondent each of the 20 life area cards and describes the area. The participant selects those areas that he/she deems important to his/her quality of life and any remaining cards are set aside. The cards representing important areas are laid out and the participant is given three poker chips for each card. The total number of chips can, therefore, range from 0 (no life areas are important) to 60 (all 20 life areas are important). The participant then distributes the chips across the cards to indicate the level of importance of each life area, with more chips indicating greater importance. Next, the participant provides a satisfaction rating for each area, using a 6-point Likert-type scale anchored by 1 (very dissatisfied) and 6 (very satisfied) and illustrated with six stylised frowning and smiling faces.

Table 1.

Injection Drug User Quality of Life (IDUQOL) life area names and descriptions

Life Area Description
Being Useful e.g., volunteering, employment, participating in the community, helping others
Community Resources e.g., food bank, soup kitchen, shelters, outreach programs, social service agencies
Drugs drug use – e.g., alcohol, heroin, cocaine, crack
Drug Treatment e.g., detox, recovery house, residential treatment, methadone, abstinence
Education e.g., formal schooling, literacy programs
Family e.g., parents, children, siblings, foster families (not friends)
Feeling Good about Yourself e.g., self-esteem, self-worth
Friends anyone you consider a friend (but not family)
Harm Reduction access to, and experience with: e.g., methadone treatment, needle exchange, safe injection programs, prescription heroin
Health mental and physical health, including HIV, AIDS, Hepatitis C, disability, schizophrenia
Health Care access to, and experience with: physicians, nurses, hospitals, clinics, ER
Housing e.g., owning, renting, house, apartment, hotel room, shelters, homeless
Independence and Free Choice e.g., making your own decisions, autonomy, being able to do things on your own, having individual rights
Leisure Activities e.g., music, sports, movies, books, parties
Money e.g., income, welfare, cash flow, meeting your needs
Neighborhood Safety e.g., crime, violence, police harassment
Partner(s) e.g., spouse, common-law partner, same-sex partner, girlfriend or boyfriend (not casual partners)
Sex e.g., sexual intimacy, quantity or quality of sex, sex in exchange for money or drugs, sexual abuse
Spirituality e.g., religion, faith, belief in a higher being or spiritual world (or not)
Transportation e.g., car, taxi, public transportation, getting to places you need to go
Open in a new tab

When scoring the IDUQOL, the importance rating (number of chips) of each area is divided by the total number of chips used by that participant and then multiplied by the satisfaction rating for that area. This produces an area score. Finally, all area scores are summed to obtain an overall quality of life score ranging from 1 (very dissatisfied) to 6 (very satisfied).

IDUQOL content validation questionnaire

The questionnaire was divided into seven sections covering the clarity of the instrument title, ease of administration procedure instructions, clarity of the names and descriptions of the 20 IDUQOL life areas, whether each of the 20 IDUQOL life areas should be included in the measure (including whether any life areas need to be added, revised, or deleted), ease of the response formats used for each of importance and satisfaction ratings, clarity of scoring procedure instructions, and the ease of use of the record form. Experts were also given the opportunity to provide open-ended commentary in each section. As recommended by Lynn [55], a four-point Likert type scale was used in most cases. For questions involving clarity, the following four response options were used: 0 = not at all clear, 1 = somewhat clear, 2 = mostly clear, 3 = very clear. For questions involving ease, the following four response options were used: 0 = not at all easy to follow/use, 1 = somewhat easy to follow/use, 2 = mostly easy to follow/use, 3 = very easy to follow/use. For questions involving inclusion of items, the following three response options were used: 0 = no, 1 = unsure, 2 = yes. Two questions asked about how helpful the provided examples were in the manual; for these, the following four response options were used: 0 = not at all helpful, 1 = somewhat helpful, 2 = mostly helpful, 3 = very helpful.

Procedures

The six SMEs were identified through the second author's professional contacts with nationally and internationally recognized experts in the area of substance abuse epidemiology and treatment. They were sent a letter of invitation and agreed to take part in the study. None of the SMEs were associated with the development of the IDUQOL. The SMEs were mailed a copy of the IDUQOL (which included the 20 life area cards, poker chips, satisfaction rating card, and record form), the administration and scoring manual, and the IDUQOL content validation questionnaire. As suggested by Grant and Davis [56], SMEs were provided with the conceptual basis for the IDUQOL via the brief introduction in the manual in which the definition of QoL underlying this measure, the target population, and how the measure is intended to be used was provided. The SMEs completed the content validation questionnaire at their leisure and independently of one another. All SMEs returned usable questionnaires.

Two commonly used judgmental measures of inter-rater agreement, the Content Validity Index (CVI) [55,57] and the Average Deviation Mean Index (ADM) [58-60], were used to evaluate SMEs' agreement on ratings of the various IDUQOL elements. The two measures provide very different types of information, however, and should be viewed as complementary. Generally, the CVI indicates the proportion of SMEs that endorse an element as content valid whereas the ADM indicates the degree of disagreement among SMEs in the response option selected regardless of whether they, as a group, endorsed an element or not. Thus, one should first examine the CVI values to determine whether the SMEs endorsed an item or not and then consider the level of agreement among the SMEs by examining the ADM.

The CVI can be computed at the individual item level (I-CVI) and at the level of the overall scale or subscale (S-CVI). I-CVI is computed as the proportion of SMEs that endorse an item. Following standard procedures for four response options [49,55], ratings of 2 or 3 were combined and treated as endorsements by SMEs whereas ratings of 0 or 1 were combined and treated as non-endorsements in the present study. When three response options were used, a rating of 2 was treated as an endorsement by SMEs whereas ratings of 0 or 1 were combined and treated as non-endorsements. A minimum of five out of the six SMEs (I-CVI ≥ .83) had to endorse an item to achieve significant evidence (α = .05) of content validity for any given item on the IDUQOL content validity questionnaire and to provide confidence that agreement was not occurring by chance alone [55]. Elements that were not endorsed by a minimum of five SMEs were examined further to determine if appropriate revisions could be made.

The S-CVI may be defined and computed a number of different ways, but Polit and Beck [49] recommend using the average proportion of items endorsed by the SMEs (what they refer to as S-CVI/Ave) and computing this as the average of the I-CVI values. This is the approach that will be used in the present study in conjunction with Lynn's [55] description of S-CVI as "the proportion of total items judged content valid" (p. 384). For S-CVI/Ave, the minimum acceptable value is recommended to be .90 [49].

The ADM Index measures dispersion of ratings about the mean rating; thus, it is actually a measure of disagreement so lower values indicate higher levels of agreement among SMEs. An advantage of the ADM Index is that it provides a measure of dispersion that is directly interpretable in terms of the original rating scale units. The general cut-off for determining acceptable ADM values is based on c/6, with c referring to the number of response options [59]. Thus, using this guideline for practical significance, acceptable ADM values are .50 or less for ratings with three response options and .69 or less for ratings with four response options. Critical values that can be used to evaluate whether an obtained ADM could have been achieved by chance can also be computed. Critical values for ADM at the 5% level of significance, taking into account the number of SMEs and the number of response options in the present study, would be .28 or less for three response options and .44 or less for four response options [58]. ADM values that are equal to or below these critical values are unlikely to have been obtained by chance. Elements for which ADM values were neither practically nor statistically significant or were only practically significant were examined further to determine if appropriate revisions could be made.

Results

Content validity evidence for IDUQOL life areas

Table 2 presents the I-CVI and ADM results for each of the 20 IDUQOL life areas indicating whether (a) each life area was appropriate for a QoL measure for IDUs, (b) the name of the life area was clear, and (c) the description of the life area was clear. I-CVI results showed that all individual life areas, including the name used and the description provided, were endorsed by a minimum of five SMEs.

Table 2.

Item level CVI (I-CVI) and ADM index values for each of the 20 IDUQOL life areas

Appropriate?a Nameb Descriptionb
Life Area I-CVI ADM I-CVI ADM I-CVI ADM
Being Useful 1.00 .00 1.00 .44 0.83 .67
Community Resources 1.00 .00 1.00 .28 1.00 .28
Drugs 1.00 .00 0.83 .55 0.83 .67
Drug Treatment 0.83 .28 1.00 .00 1.00 .28
Education 0.83 .28 1.00 .00 1.00 .28
Family 1.00 .00 1.00 .00 1.00 .44
Feeling Good about Yourself 0.83 .28 1.00 .28 1.00 .50
Friends 1.00 .00 1.00 .00 1.00 .28
Harm Reduction 1.00 .00 1.00 .00 0.83 .33
Health 1.00 .00 1.00 .00 1.00 .28
Health Care 1.00 .00 1.00 .00 1.00 .28
Housing 1.00 .00 1.00 .00 1.00 .00
Independence and Free Choice 0.83 .28 1.00 .50 1.00 .00
Leisure Activities 1.00 .00 1.00 .00 1.00 .00
Money 1.00 .00 1.00 .00 1.00 .28
Neighborhood Safety 1.00 .00 1.00 .00 1.00 .44
Partner(s) 1.00 .00 1.00 .28 1.00 .00
Sex 1.00 .00 1.00 .00 1.00 .44
Spirituality 1.00 .00 1.00 .00 1.00 .00
Transportation 1.00 .00 1.00 .00 1.00 .00
Open in a new tab

a Ratings were made on a 3-point scale. b Ratings were made on a 4-point scale.

Note. I-CVI: 1.00 = endorsement by all six subject matter experts (SMEs); 0.83 = endorsement by five of six SMEs. ADM Index: with a 3-point scale, acceptable values are .50 or less and statistically significant values are .28 or less; with a 4-point scale, acceptable values are .69 or less and statistically significant values are .44 or less. Values that are acceptable, but not statistically significant, are bolded.

The ADM results provide additional information about the extent to which the SMEs agreed on the exact rating (e.g., not at all clear, somewhat clear, mostly clear, very clear) for each life area. In all cases, acceptable agreement (i.e., practical significance) was obtained. In terms of the appropriateness of the life areas, each life area also showed statistically significant agreement among the SMEs. When considering whether the name of the life area was clear, there were only two cases (i.e., Drugs, Independence & Free Choice) in which agreement was acceptable, but not statistically significant – meaning that agreement could have occurred by chance. In the case of Drugs, five SMEs rated this name as 'very clear' whereas one SME thought it was only 'somewhat clear' because drugs could be confused with medications. Independence & Free Choice was rated as 'mostly clear' by half of the SMEs and 'very clear' by the other half of SMEs with the problem being different interpretations of the word "independence". When considering whether the description of the life area was clear, there were three cases (i.e., Being Useful, Drugs, Feeling Good about Yourself) in which agreement was acceptable, but not statistically significant. The description for Being Useful was rated as 'somewhat clear' by one SME, 'mostly clear' by one SME and 'very clear' by four SMEs. Most SME comments were focused on the visual depiction provided on the card and no suggestions for changes or additions to the description were offered. The description for Drugs (i.e., "drug use – e.g., alcohol, heroin, cocaine, crack") was rated as 'somewhat clear' by one SME, 'mostly clear' by two SMEs and 'very clear' by three SMEs. There were two concerns raised by SMEs. The first and most prominent concern was that we limited this life area to use of drugs; the second concern was that we only listed four drugs. The description for Feeling Good about Yourself (i.e., "e.g., self-esteem, self-worth") was rated as 'mostly clear' by half of the SMEs and 'very clear' by the other half of the SMEs. No suggestions were made for changes to the description.

SMEs were also asked if there were any life areas that they would recommend deleting or adding to the IDUQOL. No life areas were recommended for deletion from the IDUQOL. The following additional life areas were suggested: food, pets, personal safety, sense of future (e.g., hopefulness, aspirations), employment (as its own life area separate from Being Useful), and pain.

Table 3 shows that the S-CVI/Ave for the element groupings of Appropriateness, Name clarity, and Description clarity of the IDUQOL life areas ranged from .97 to .99, which exceeded the minimum value of .90 and is also strong evidence of content validity.

Table 3.

Scale level CVI (S-CVI/Ave) for elements of the IDUQOL measure and manual

IDUQOL Element S-CVI/Ave
Appropriateness of IDUQOL Life Areas (20 items)a 0.97
Clarity of IDUQOL Life Area Names (20 items)b 0.99
Clarity of IDUQOL Life Area Descriptions (20 items)b 0.98
Clarity of Title and Target Population (2 items)b 0.92
Ease of Administration Procedure (4 items)b 1.00
Ease of Response Formats (2 items)b 0.83
Ease of Scoring Procedure (3 items)b 0.94
Helpfulness of Provided Examples (3 items)b 1.00
Ease of the Record Form to Use (1 item)b 1.00
Open in a new tab

a Ratings were made on a 3-point scale. b Ratings were made on a 4-point scale.

Note. Ease of Record Form result is the same as the individual level findings; this element is included here for completeness. For S-CVI/Ave, the minimum acceptable value is .90. A scale level ADM Index could not be applied here as it requires scales comprised of essentially parallel items.

Content validity evidence for other IDUQOL elements

Table 3 also shows that the S-CVI/Ave for the element groupings of Clarity of Title and Target Population, Ease of Administration Procedure, Ease of Scoring Procedure, and Helpfulness of Provided Examples ranged from .92 to 1.00. Only Ease of Response Formats produced a S-CVI/Ave (.83) that was below the minimum recommended value of .90, although both items were endorsed by five out of six SMEs.

I-CVI and ADM results for other individual elements (e.g., title, administration instructions, scoring instructions, record form) of the IDUQOL measure and manual are presented in Table 4. I-CVI results showed that all individual elements were endorsed by a minimum of five SMEs. The ADM results show that acceptable agreement (i.e., practical significance) was obtained in all cases, although there were four cases in which agreement was not statistically significant (i.e., Clarity of Title, Response Format – Chips, Response Format – Smiley Faces, Scoring Procedure – Summed Score).

Table 4.

Item level CVI (I-CVI) and ADM index values for IDUQOL measure and manual elements

IDUQOL Content Validity Questionnaire Itema I-CVI ADM Index
Clarity of Title 0.83 .67
Clarity of Intended Population 1.00 .00
Administration Procedure – Introduction 1.00 .28
Administration Procedure – Respondent Selects Life Areas 1.00 .44
Administration Procedure – Respondent Rates Importance 1.00 .00
Administration Procedure – Respondent Rates Satisfaction 1.00 .00
Response Format – Easy for Respondent to Use Chips 0.83 .67
Response Format – Easy for Respondent to Use Smiley Faces 0.83 .67
Scoring Procedure – Relative Importance Score 1.00 .44
Scoring Procedure – Importance × Satisfaction Score 1.00 .44
Scoring Procedure – Obtain Summed Score 0.83 .67
Example – Relative Importance Score 1.00 .00
Example – Importance × Satisfaction Score 1.00 .00
Example – Completed Sample Record Form 1.00 .00
Ease of the Record Form to Use 1.00 .28
Open in a new tab

a Ratings were made on a 4-point scale.

Note. I-CVI: 1.00 = endorsement by all six subject matter experts (SMEs); 0.83 = endorsement by five of six SMEs. ADM Index: with a 4-point scale, acceptable values are .69 or less and statistically significant values are .44 or less. Values that are acceptable, but not statistically significant, are bolded.

The name or title of the IDUQOL was rated as 'somewhat clear' by one SME, 'mostly clear' by one SME and 'very clear' by four SMEs, with no suggestions made for how to make the title clearer. The Response Format – Chips was rated as 'somewhat easy to use' by one SME, 'mostly easy to use' by one SME, and 'very easy to use' by four SMEs. Suggestions were made for simplifying the instructions given to IDUs about how to use the poker chips to indicate the importance of the different life areas. In addition, it was suggested that poker chips might act as a trigger for IDUs with gambling issues. The Response Format – Smiley Faces was rated as 'somewhat easy to use' by one SME and 'very easy to use' by five SMEs. It was suggested that we consider using an odd-numbered Likert-type scale (rather than our 6-point Likert-type scale) for the smiley faces that would permit a neutral response. The Scoring Procedure – Summed Score was rated as 'somewhat easy to follow' by one SME, 'mostly easy to follow' by one SME, and 'very easy to follow' by four SMEs. It was suggested that we clarify the headings on the record form so they would better match the terms used in the manual.

Suggested revisions to the IDUQOL based on SME feedback

Open-ended feedback and comments from the SMEs resulted in several other suggestions for changes to the IDUQOL measure, materials, and manual. These may be grouped into three points. First, suggestions were made to expand the descriptions for (a) Education, (b) Family, and (c) Sex. Second, it was pointed out that we needed to show greater diversity in our cards involving people – specifically Family and Friends. Third, we were advised to revise the cards depicting (a) Being Useful, and (b) Independence and Free Choice to make them clearer.

Discussion

Test development and validation are ongoing processes designed to ensure measures and the inferences made from them remain appropriate, relevant, and useful for the target population and context of use [47]. The IDUQOL was developed as a measure of broadly defined subjective QoL that incorporates both health and non-health related aspects of IDUs' lives. Administration of the IDUQOL was designed to be sensitive to the diversity of literacy levels, English language skills, attention levels, and cognitive abilities of the target population. An important step in test development and validation is the evaluation of content validity. The purpose of the present study was to examine the content validity of various elements of the IDUQOL measure and manual using SMEs and two commonly used judgmental methods (i.e., CVI and ADM). The CVI indicates the proportion of SMEs that endorse an element as content valid whereas the ADM indicates the degree of disagreement among SMEs in the response option selected. Overall, the results of this study provide strong evidence for the content validity of the elements of the IDUQOL measure and manual. Specifically, the I-CVI results supported the content validity of each of the individual elements. These elements include the appropriateness, name, and description of each of the 20 life areas, clarity of the name of the measure, clarity of the target population, ease of each step of the administration procedure, ease of each response format (i.e., chips and smiley face scale), ease of each step of the scoring procedure, helpfulness of each of the provided example boxes in the manual, and ease of use of the IDUQOL record form.

The S-CVI/Ave results also supported the content validity of all of the grouped elements of the IDUQOL measure and manual (e.g., Appropriateness of Life Areas, Ease of Administration Procedure), with the exception of Ease of Response Formats. Two points are worth noting about the Ease of Response Formats case. First, when examined individually using I-CVI, each of the two items under Ease of Response Formats was endorsed by five of the six SMEs, supporting their content validity. In fact, the one SME who supposedly did not endorse either of these items actually circled both 'somewhat easy to use' and 'mostly easy to use' (responses that fell into the 'not endorsed' and 'endorsed' categories, respectively) in each case and indicated that the ease of each response format for the target population was an empirical question that should be piloted instead. Using a conservative approach, we treated this SME's response as a non-endorsement, although it could be argued to be more ambiguous. Second, it should be noted that when there are only two or three items making up a grouping of elements (as is the case for the two-item Ease of Response Formats grouping), the minimum acceptable level of .90 for S-CVI/Ave cannot be reached unless all but one item in the grouping achieves endorsement by all six of the SMEs. Taking each of these points into account, we would argue that the content validity of the response formats should not be discounted, but that care should be taken to ensure, through further study or pilot testing, that these response formats are appropriate to the group of IDUs with whom a researcher or practitioner wishes to use the IDUQOL. Based on interviews we conducted with the experienced staff who administered the IDUQOL to participants in another study [44,61], that sample of IDUs did not have difficulty using either response format, although some respondents expressed the desire for a neutral response option on the Likert-type smiley face satisfaction scale.

In terms of the ADM results, practical significance was obtained for all 75 elements of the IDUQOL and manual, indicating a high level of agreement among the SMEs in their ratings. However, statistical significance was not reached for the following nine elements: (a) names for the Drugs and Independence & Free Choice life areas, (b) descriptions for the Being Useful, Drugs, and Feeling Good about Yourself life areas, and (c) clarity of title, ease of response format – chips, ease of response format – smiley faces, and ease of scoring procedure – summed score. Thus, for these elements, it is possible that SME agreement on the ratings may have occurred by chance alone.

Both the CVI and ADM results and the open-ended feedback from the SMEs were used to make seven main revisions to IDUQOL elements. First, the names for the Drugs and Independence & Free Choice life areas were changed to Drugs & Alcohol and Free Choice, respectively. We also decided to take one SME's suggestion to change the Education life area to Education & Training.

Second, descriptions for the life areas of Drugs & Alcohol, Education, Family, and Sex were expanded. The original and revised descriptions for each of these life areas are provided in Figure 1. The descriptions for some other life areas (e.g., Being Useful, Feeling Good about Yourself) remained unchanged because SMEs had only commented on the visual image used or had not provided any suggestions for changes.

Figure 1.

Figure 1

Open in a new tab

Samples of original and revised IDUQOL card descriptions.

Third, we added Sense of Future as a life area to the IDUQOL with the following description: "e.g., hopefulness, aspirations, dreams, goals". The SMEs made several suggestions for life areas that could be added to the IDUQOL (i.e., food, pets, personal safety, sense of future, employment, pain). After considerable thought and given previous IDU focus group discussions about important life areas, we ultimately decided to only add Sense of Future as its own life area to the measure. However, we recognize that other researchers and practitioners may want to consider including these suggestions as additional life areas in their own work. Three points about our decision are worth noting: (a) following discussion, we decided that 'personal safety' was an implicit part of Neighborhood Safety, (b) we decided to incorporate 'pain' under our description of the Health life area, and (c) we resisted adding employment as its own life area because IDUs in a concurrent focus group study strongly opposed viewing employment as separate from, or more important than, other aspects of 'being useful in society'.

Fourth, we made revisions to the visual depictions for Friends, Family, Being Useful, and Free Choice. Based on SMEs' comments, we increased the diversity of people in the cards for Friends and Family (see Figure 2). For the Being Useful and Free Choice life areas, SMEs recommended using different images to make these concepts clearer to respondents. The old and new cards are shown in Figure 3.

Figure 2.

Figure 2

Open in a new tab

Original and revised cards to improve diversity of people.

Figure 3.

Figure 3

Open in a new tab

Original and revised cards to improve poor graphics on cards.

Fifth, we were particularly struck by one SME's suggestion that the poker chips used in the response format for importance ratings might act as a trigger for IDUs with gambling issues and so we changed these to unmarked chips. We also incorporated suggestions made by SMEs for simplifying the instructions given to IDUs in the manual about how to use the chips to indicate the importance of the different life areas.

Sixth, we changed the Likert-type smiley face scale used to rate satisfaction from a 6-point scale to a 7-point scale that would permit a neutral response. This was based not only on SMEs' suggestions but also on our own concurrent experiences in administering the IDUQOL to IDUs [44,61]. We found that it was particularly appropriate to have a neutral option available when the respondent was rating satisfaction with a life area that had not been rated as particularly important.

Seventh, we revised the headings used in the IDUQOL record form so they would better match the terms used in the manual and would make obtaining the summed score easier (see Figure 4.)

Figure 4.

Figure 4

Open in a new tab

Original and revised IDUQOL record form headings.

Given the strong support provided by this study for the content validity of the IDUQOL and its manual, the revisions made to improve them further, and previous research supporting the validity of inferences made from the IDUQOL [44,61], the IDUQOL is a viable instrument for assessing broad-based QoL in IDUs and potentially in non-injection drug users [45]. The majority of published research in which QoL is examined with IDUs focuses on HRQOL. Future research is needed that examines the impact of drug use and various treatment options on QoL using broadly defined subjective QoL measures such as the IDUQOL. Future research on the IDUQOL needs to further examine its appropriateness and usefulness with non-injection drug users, its sensitivity to change, and its relationship with other broad-based QoL measures such as the Quality of Life Interview – Brief Version, which was originally developed for use with the mentally ill, or the Personal Wellbeing Index, which is the successor to the ComQol-A5 and was developed for use with the general population [62].

Conclusion

The results from the present study provided strong support for the content validity of the elements of the IDUQOL measure and manual. Further revisions were made based on the CVI and ADM results as well as the open-ended feedback from the SMEs. These revisions included (a) revised names for three life areas (now Free Choice, Drugs & Alcohol, and Education & Training), (b) an expanded description for the Drugs & Alcohol, Education & Training, Family, and Sex life areas, (c) the addition of Sense of Future as a life area, (d) revisions to the visual depictions for Family, Friends, Being Useful and Free Choice, (e) the use of unmarked chips rather than poker chips in the response format for importance ratings, (f) the inclusion of a neutral point on the Likert-type scale of smiley faces for rating satisfaction, and (g) revisions to the headings used in the IDUQOL record form to make obtaining the summed score clearer. These revisions to the IDUQOL resulted in an instrument that is even easier for researchers, practitioners, and program evaluators to use as a way of assessing and tracking changes in QoL over time or as a result of interventions in IDUs.

List of abbreviations

ADM Average Deviation Mean Index

CVI Content Validity Index

HRQOL health-related quality of life

IDUs injection drug users

IDUQOL injection drug user quality of life scale

QoL quality of life

SMEs subject matter experts

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

AH obtained funding, designed the study, directed the statistical analyses, prepared the initial draft of the manuscript and conducted revisions. AP conceived of the study, obtained funding, coordinated data collection, and conducted revisions of the manuscript. Each author read and approved the final manuscript.

Acknowledgments

Acknowledgements

This research was supported by an operating grant from the Canadian Institutes of Health Research (CIHR) to Dr. Anita Palepu and Dr. Anita Hubley. Additional support was provided through a Canadian Institutes for Health Research New Investigator Award and a Michael Smith Foundation for Health Research Senior Scholar Award to Dr. Anita Palepu. We thank Dr. Bruno D. Zumbo and Lara Russell for their comments on an earlier draft of this manuscript.

Contributor Information

Anita M Hubley, Email: anita.hubley@ubc.ca.

Anita Palepu, Email: anita@hivnet.ubc.ca.

References

  1. Globe DR, Hays RD, Cunningham WE. Associations of clinical parameters with health-related quality of life in hospitalized persons with HIV disease. AIDS Care. 1999;17:71–86. doi: 10.1080/09540129948216. [DOI] [PubMed] [Google Scholar]
  2. Koch T. Life quality vs the 'quality of life': Assumptions and underlying prospective quality of life instruments in health care planning. Social Science and Medicine. 2000;51:419–427. doi: 10.1016/S0277-9536(99)00474-8. [DOI] [PubMed] [Google Scholar]
  3. Gill TM, Feinstein AR. A critical appraisal of the quality of quality-of-life measurements. Journal of the American Medical Association. 1994;272:619–626. doi: 10.1001/jama.272.8.619. [DOI] [PubMed] [Google Scholar]
  4. Burgess AP, Carretero M, Elkington A, Pasqual-Marsettin E, Lobaccaro C, Catalán J. The role of personality, coping style, and social support in health-related quality of life in HIV infection. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care & Rehabilitation. 2000;9:423–437. doi: 10.1023/a:1008918719749. [DOI] [PubMed] [Google Scholar]
  5. Rooney S, Freyne A, Kelly G, O’Connor J. Differences in the quality of life of two groups of drug users. Irish Journal of Psychological Medicine. 2002;19:55–59. doi: 10.1017/S0790966700006960. [DOI] [PubMed] [Google Scholar]
  6. Sherbourne CD, Hays RD, Fleishman JA, Vitiello B, Magruder KM, Bing EG, McCaffrey D, Burnam A, Longshore D, Eggan F, Bozzette SA, Shapiro MF. Impact of psychiatric conditions on health-related quality of life in persons with HIV infection. American Journal of Psychiatry. 2000;157:248–254. doi: 10.1176/appi.ajp.157.2.248. [DOI] [PubMed] [Google Scholar]
  7. te Vaarwerk MJE, Gaal EA. Psychological distress and quality of life in drug-using and non-drug-using HIV-infected women. European Journal of Public Health. 2001;11:109–115. doi: 10.1093/eurpub/11.1.109. [DOI] [PubMed] [Google Scholar]
  8. Turner J, Page-Shafer K, Chin DP, Osmond D, Mossar M, Markstein L, Huitsing J, Barnes S, Clemente V, Chesney M, the Pulmonary Complications of HIV Infection Study Group Adverse impact of cigarette smoking on dimensions of health-related quality of life in persons with HIV infection. AIDS Patient Care and STDs. 2001;15:615–624. doi: 10.1089/108729101753354617. [DOI] [PubMed] [Google Scholar]
  9. Darke S, Hall W, Wodak A, Heather N, Ward J. Development and validation of a multidimensional instrument for assessing outcome of treatment among opiate users: Opiate Treatment Index. British Journal of Addiction. 1992;87:733–742. doi: 10.1111/j.1360-0443.1992.tb02719.x. [DOI] [PubMed] [Google Scholar]
  10. Torrens M, San L, Martinez A, Castillo C, Domingo-Salvany A, Alonso J. Use of the Nottingham Health Profile for measuring health status of patients in methadone maintenance treatment. Addiction. 1997;92:707–716. doi: 10.1111/j.1360-0443.1997.tb02937.x. [DOI] [PubMed] [Google Scholar]
  11. Puigdollers E, Domingo-Salvany A, Brugal MT, Torrens M, Alvaros J, Castillo C, Magri N, Martin S, Vazquez JM. Characteristics of heroin addicts entering methadone maintenance treatment: Quality of life and gender. Substance Use & Misuse. 2004;39:1353–1368. doi: 10.1081/JA-120039392. [DOI] [PubMed] [Google Scholar]
  12. Giacomuzzi SM, Riemer Y, Ertl M, Kemmler G, Rossler H, Hinterhuber H, Kurz M. Gender differences in health-related quality of life on admission to a maintenance treatment program. European Addiction Research. 2005;11:69–75. doi: 10.1159/000083035. [DOI] [PubMed] [Google Scholar]
  13. Ryan CF, White JM. Health status at entry to methadone maintenance treatment using the SF-36 health survey questionnaire. Addiction. 1996;91:39–45. doi: 10.1111/j.1360-0443.1996.tb03160.x. [DOI] [PubMed] [Google Scholar]
  14. Stein MD, Mulvey KP, Plough A, Samet JH. The functioning and well being of persons who seek treatment for drug and alcohol use. Journal of Substance Abuse. 1998;10:75–84. doi: 10.1016/S0899-3289(99)80142-4. [DOI] [PubMed] [Google Scholar]
  15. Préau MP, C., Spire B, Sobel A, Dellamonica P, Moatti JP, Carrieri MP. Health related quality of life among both current and former injection drug users who are HIV-infected. Drug and Alcohol Dependence. 2007;86:175–182. doi: 10.1016/j.drugalcdep.2006.06.012. [DOI] [PubMed] [Google Scholar]
  16. Costenbader EC, Zule WA, Coomes CM. The impact of illicit drug use and harmful drinking on quality of life among injection drug users at high risk for hepatitis C infection. Drug and Alcohol Dependence. 2007. pp. Feb 21 [E–pub ahead of print]. [DOI] [PMC free article] [PubMed]
  17. Carretero MD, Burgess AP, Soler P, Soler M, Catalan J. Reliability and validity of an HIV-specific health-related quality-of-life measure for use with injecting drug users. AIDS. 1996;10:1699–1705. doi: 10.1097/00002030-199612000-00015. [DOI] [PubMed] [Google Scholar]
  18. Dalgard O, Egeland A, Skaug K, Vilimas K, Steen T. Health-related quality of life in active injecting drug users with and without chronic hepatitis C virus infection. Hepatology. 2004;39:74–80. doi: 10.1002/hep.20014. [DOI] [PubMed] [Google Scholar]
  19. Perez IR, Baño JR, Lopez Ruz MA, Jiminez AA, Prados MC, Liaño JP, Rico RM, Lima JT, Pardal JLP, Gomez ML, Muñoz N, Morales D, Marcos M. Health-related quality of life of patients with HIV: Impact of sociodemographic, clinical, and psychosocial factors. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care, & Rehabilitation. 2005;14:1301–1310. doi: 10.1007/s11136-004-4715-x. [DOI] [PubMed] [Google Scholar]
  20. Smith MY, Feldman J, Kelly P, DeHovitz JA, Chirgwin K, Minkoff H. Health-related quality of life of HIV-infected women: Evidence for the reliability, validity, and responsiveness of the Medical Outcomes Study Short-Form 20. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care, & Rehabilitation. 1996;5:47–55. doi: 10.1007/BF00435968. [DOI] [PubMed] [Google Scholar]
  21. Fernández Miranda JJ. La calidad de vida en adicciones: Una medida de la efectividad de los tratamientos [Quality of life in addictions: A measure of treatment effectiveness] Anales de Psiquiatría. 2003;19:377–384. [Google Scholar]
  22. Amato L, Davoli M, Perucci CA, Ferri M, Faggiano F, Mattick RP. An overview of systematic reviews of the effectiveness of opiate maintenance therapies: Available evidence to inform clinical practice and research. Journal of Substance Abuse Treatment. 2005;28:321–329. doi: 10.1016/j.jsat.2005.02.007. [DOI] [PubMed] [Google Scholar]
  23. Reno RR, Aiken LS. Life activities and life quality of heroin addicts in and out of methadone treatment. . International Journal of Addictions. 1993;28:211–232. doi: 10.3109/10826089309039624. [DOI] [PubMed] [Google Scholar]
  24. Fleming CA, Christiansen D, Nunes D, Heeren T, Thornton D, Horsburgh Jr. CR, Koziel MJ, Graham C, Craven DE. Health-related quality of life of patients with HIV disease: Impact of hepatitis C coinfection. Clinical Infectious Diseases. 2004;38:572–578. doi: 10.1086/381263. [DOI] [PubMed] [Google Scholar]
  25. Gielen AC, McDonnell KA, Wu AW, O’Campo P, Faden R. Quality of life among women living with HIV: The importance of violence, social support, and self care behaviors. Social Science & Medicine. 2001;52:315–322. doi: 10.1016/S0277-9536(00)00135-0. [DOI] [PubMed] [Google Scholar]
  26. McDonnell KA, Gielen AC, O’Campo P, Burke JG. Abuse, HIV status, and health-related quality of life among a sample of HIV positive and HIV negative low income women. Quality of Life Research. 2005;14:945–957. doi: 10.1007/s11136-004-3709-z. [DOI] [PubMed] [Google Scholar]
  27. Millson P, Challacombe L, Villeneuve PJ, Strike CJ, Fischer B, Myers T, Shore R, Hopkins S. Determinants of health-related quality of life of opiate users at entry to low-threshold methadone programs. European Addiction Research. 2006;12:74–82. doi: 10.1159/000090426. [DOI] [PubMed] [Google Scholar]
  28. Mrus JM, Leonard AC, Yi MS, Sherman SN, Fultz SL, Justice AC, Tsevat J. Health-related quality of life in veterans and nonveterans with HIV/AIDS. Journal of General Internal Medicine. 2006;21:S39–47. doi: 10.1111/j.1525-1497.2006.00644.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. DeWit DJ. Frequent childhood geographic relocation: Its impact on drug use initiation and the development of alcohol and other drug-related problems among adolescents and young adults. Addictive Behaviors. 1998;23:623–634. doi: 10.1016/S0306-4603(98)00023-9. [DOI] [PubMed] [Google Scholar]
  30. Fuller CM, Borrell LN, Latkin CA, Galea S, Ompad DC, Strathdee SA, Vlahov D. Effects of race, neighborhood, and social network on age at initiation of injection drug use. American Journal of Public Health. 2005;95:689–695. doi: 10.2105/AJPH.2003.02178. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Higgins K, McElrath K. The trouble with peace: The cease-fires and their impact on drug use among youth in Northern Ireland. Youth & Society. 2000;32:29–59. doi: 10.1177/0044118X00032001003. [DOI] [Google Scholar]
  32. March JC, Oviedo-Joekes E, Romero M. Injection and non-injection drug use related to social exclusion indicators in two Andalusian cities. Drugs: Education, Prevention & Policy. 2005;12:437–447. doi: 10.1080/09687630500281998. [DOI] [Google Scholar]
  33. Martín LL, Hernández SO, Carrobles JA. Variables psicosociales en la adherencia al tratemiento antiretroviral en pacientes adscritos a un programa de mantenimiento con metadona [Psychosocial variables in the antiretroviral treatment adherence in methadone maintenance patients] Psicothema. 2005;17:575–581. [Google Scholar]
  34. Neaigus A, Gyarmathy A, Miller M, Frajzyngier VM, Friedman SR, Des Jarlais DC. Transitions to injecting drug use among noninjecting heroin users: Social network influence and individual susceptibility. Journal of Acquired Immune Deficiency Syndromes. 2006;41:493–503. doi: 10.1097/01.qai.0000186391.49205.3b. [DOI] [PubMed] [Google Scholar]
  35. Newcomb MD, Bentler PM. The impact of family context, deviant attitudes, and emotional distress on adolescent drug use: Longitudinal latent-variable analyses of mothers and their children. Journal of Research in Personality. 1988;22:154–176. doi: 10.1016/0092-6566(88)90012-8. [DOI] [Google Scholar]
  36. Wasserman DA, Sorensen JL, Delucchi KL, Masson CL, Hall SM. Psychometric evaluation of the Quality of Life Interview, Brief Version, in injection drug users. Psychology of Addictive Behaviors. 2006;20:316–321. doi: 10.1037/0893-164X.20.3.316. [DOI] [PubMed] [Google Scholar]
  37. Lehman AF. Evaluating quality of life for persons with severe mental illness: Assessment toolkit. Cambridge, MA , Evaluation Center at Health Services Research Institute; 1995. [Google Scholar]
  38. Dunaj R, Kovác D. Quality of life of convicted drug addicts; Preliminary report. Studia Psychologica. 2003;45:357–360. [Google Scholar]
  39. WHOQOL Group Development of the World Health Organisation WHOQOL-BREF quality of life assessment. Psychological Medicine. 1998;28:551–558. doi: 10.1017/S0033291798006667. [DOI] [PubMed] [Google Scholar]
  40. Cummins RA. Comprehensive Quality of Life Scale-Adult: ComQol-A5. 5th. Melbourne, Australia , School of Psychology, Deakin University; 1997. [Google Scholar]
  41. Bonomi AE, Patrick DL, Bushnell DM, Martin M. Validation of the United States’ version of the World Health Organization Quality of Life (WHOQOL) instrument. Journal of Clinical Epidemiology. 2000;53:1–12. doi: 10.1016/S0895-4356(99)00123-7. [DOI] [PubMed] [Google Scholar]
  42. Metzger DS, O'Brien CP. Substance Abuse: The Challenge of Assessment. In: Spilker B, editor. Quality of Life Assessments in Clinical Trials. New York , Raven Press, Ltd.; 1990. pp. 237–246. [Google Scholar]
  43. Brogly S, Mercier C, Bruneau J, Palepu A, Franco E. Towards more effective public health programming for injection drug users: Development and evaluation of the Injection Drug User Quality of Life Scale. Substance Use & Misuse. 2003;38:965–992. doi: 10.1081/JA-120017619. [DOI] [PubMed] [Google Scholar]
  44. Hubley AM, Russell L, Palepu A. Injection Drug Users Quality of Life (IDUQOL) scale: A validation study. Health and Quality of Life Outcomes. 2005;3 doi: 10.1186/1477-7525-3-43. http://www.hqlo.com/content/3/1/43 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Morales-Manrique CC, Valderrama-Zurian JC, Castellano-Gomez M, Aleixandre-Benavent R, Palepu A. Cross cultural adaptation of the Injection Drug User Quality of Life Scale (IDUQOL) in Spanish drug dependent population, with or without injectable consumption : Drug User Quality of Life Scale - Spanish (DUQOL - Spanish) Addictive Behaviors. 2007;32:1913–1921. doi: 10.1016/j.addbeh.2006.12.006. [DOI] [PubMed] [Google Scholar]
  46. APA. AERA. NCME . Standards for educational and psychological testing. Washington, DC , American Psychological Association, American Educational Research Association, National Council on Measurement in Education; 1985. [Google Scholar]
  47. Hubley AM, Zumbo BD. A dialectic on validity: Where we have been and where we are going. The Journal of General Psychology. 1996;123:207–215. [Google Scholar]
  48. Messick S. Validity of test interpretation and use. Research Report No. 90-11. Educational Testing Service; 1990. [Google Scholar]
  49. Polit DF, Beck CT. The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health. 2006;29:489–497. doi: 10.1002/nur.20147. [DOI] [PubMed] [Google Scholar]
  50. Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: Conducting a content validity study in social work. Social Work Research. 2003;27:94–104. [Google Scholar]
  51. Sireci SG. The construct of content validity. Social Indicators Research. 1998;45:83–117. doi: 10.1023/A:1006985528729. [DOI] [Google Scholar]
  52. Beck CT, Gable RK. Ensuring content validity: An illustration of the process. Journal of Nursing Measurement. 2001;9:201–215. [PubMed] [Google Scholar]
  53. Haynes SN, Richard DR, Kubany ES. Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment. 1995;7:238–247. doi: 10.1037/1040-3590.7.3.238. [DOI] [Google Scholar]
  54. Cramer ME, Atwood JR, Stoner JA. Measuring community coalition effectiveness using the ICE© instrument. Public Health Nursing. 2006;23:74–87. doi: 10.1111/j.0737-1209.2006.230111.x. [DOI] [PubMed] [Google Scholar]
  55. Lynn MR. Determination and quantification of content validity. Nursing Research. 1986;35:382–385. doi: 10.1097/00006199-198611000-00017. [DOI] [PubMed] [Google Scholar]
  56. Grant J, Davis L. Selection and use of content experts for instrument development. Research in Nursing and Health. 1997;20:269–274. doi: 10.1002/(SICI)1098-240X(199706)20:3<269::AID-NUR9>3.0.CO;2-G. [DOI] [PubMed] [Google Scholar]
  57. Lawshe CH. A quantitative approach to content validity. Personnel Psychology. 1975;28:563–575. doi: 10.1111/j.1744-6570.1975.tb01393.x. [DOI] [Google Scholar]
  58. Dunlap WP, Burke MJ, Smith-Crowe K. Accurate tests of statistical significance for r WG and average deviation interrater agreement indexes. Journal of Applied Psychology. 2003;88:356–362. doi: 10.1037/0021-9010.88.2.356. [DOI] [PubMed] [Google Scholar]
  59. Burke MJ, Dunlap WP. Estimating interrater agreement with the Average Deviation Index: A user's guide. Organizational Research Methods. 2002;5:159–172. doi: 10.1177/1094428102005002002. [DOI] [Google Scholar]
  60. Burke MJ, Finkelstein LM, Dusig MS. On average deviation indices for estimating interrater agreement. Organizational Research Methods. 1999;2:49–68. doi: 10.1177/109442819921004. [DOI] [Google Scholar]
  61. Russell LB, Hubley AM, Palepu A, Zumbo BD. Does weighting capture what's important? Revisiting subjective importance weighting with a quality of life measure. Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement. 2006;75:141–167. [Google Scholar]
  62. International Wellbeing Group . Personal Wellbeing Index. Melbourne, Australia , Australian Centre on Quality of Life, Deakin University.; 2006. http://www.deakin.edu.au/research/acqol/instruments/wellbeing_index.htm [Google Scholar]

Articles from Health and Quality of Life Outcomes are provided here courtesy of BMC

RESOURCES