Abstract
Introduction
Academic detailing (AD) is an educational outreach strategy to provide clinicians with current evidence-based information, which has been shown to change prescribing behaviours. The overall effectiveness of AD interventions is associated with prescriber satisfaction; however, most approaches use single items or non-validated measures. This study aims to develop and validate an instrument to assess prescriber satisfaction with AD interventions.
Methods
A group of candidate items was generated and refined based on constructs identified through a literature review and in consultation with an expert panel. The initial instrument was piloted with 183 primary care providers who participated in an AD intervention on opioid-related pain management. To support the validity and reliability of the measure, psychometric properties were examined.
Results
Ten candidate items were developed based on the following themes: acceptability, feasibility of implementation, usefulness, perception of efficacy, overall satisfaction, willingness to repeat and willingness to change. One item related to willingness to change did not contribute to assessing an individual’s ability and lowered the measure’s internal consistency and was therefore dropped.
Conclusion
Results supported the validity and reliability of a refined 9-item measure of Provider Satisfaction with Academic Detailing (the PSAD). This measure should be considered for broad use across educational outreach programmes as a standardized measure to assess provider satisfaction and provide continuous quality improvement.
Keywords: academic detailing, instrument development, prescriber satisfaction, psychometric, validity
Introduction
Academic detailing (AD) is an educational outreach strategy used to provide healthcare professionals with up-to-date, unbiased, evidence-based information that can improve patient care and health outcomes.1,2 AD interventions have been shown to facilitate the dissemination and implementation of clinical protocols and therapeutic guidelines, positively impact the quality of care and successfully change prescribing behaviour across several clinical areas.3–7 AD interventions are commonly delivered in one-on-one interactions between a trained ‘academic detailer’ (usually, a healthcare professional) and a healthcare provider, which are typically referred to as ‘visits’.3 AD visits usually take place in the setting where the participants conduct their practices (e.g. hospital, clinic, office) and a single AD programme can be implemented in one or more visits, contingent on its purposes, resources and other logistic considerations.3
Arguably, healthcare providers’ receptiveness to AD is key to the prospects of successful implementation of AD programmes. The Implementation Research Outcomes Model proposed by Proctor et al.8 identifies several outcomes critical to evaluating successful implementation: acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration and sustainability. This conceptual model proposes that the level of analysis for several of these outcomes is the individual provider being targeted by the programme, namely for the outcomes of acceptability, adoption, appropriateness, feasibility and fidelity. These same constructs appear in the implementation research literature under the canopy of the ‘satisfaction’ construct.9,10 The effectiveness of AD interventions has also been shown to be associated with overall prescriber satisfaction but typically single item measures or non-psychometrically validated surveys are used.11–13 Several studies also reported having used prescriber satisfaction to monitor the implementation of AD programmes and inform continuous quality improvement.10,12 Given the importance of satisfaction as a process outcome, as well as a potential predictor of the programme’s effectiveness, there is a need for a valid and reliable measure of prescriber satisfaction with AD. This study aimed to develop a new measure to assess prescriber satisfaction with AD programmes and test its psychometric properties.
Methods
The measure of Prescriber Satisfaction with Academic Detailing (PSAD) was developed under the umbrella of a Centers for Disease Control and Prevention (CDC)-funded AD programme targeting opioid prescribers in the state of Illinois. Briefly, this programme entailed assessing the impact of providing AD to primary care providers (PCPs) on opioid prescribing patterns. Further details of the original study were published elsewhere.14 Although this measure was first developed for an opioid-specific AD programme, our goal was to develop a measure that could be used to assess the satisfaction of recipients of AD across a wide range of topics and clinical specialties. The development of the PSAD instrument involved 6 stages. The first three stages of the process pertain to the development of the initial group of candidate items, stages 4 and 5 relate to the assessment of the psychometric properties of the instrument and, finally, stage 6 pertains to the refinement of the final measure. The study protocol was reviewed and approved by the UIC Institutional Review Board. Participants provided written informed consent at the beginning of the first visit.
Development of the initial group of candidate items
The need for the development of this new measure arose organically in the first stages of the development of the protocol for the overarching study. According to the conceptual model for the assessment of implementation research outcomes, outcomes of feasibility, acceptability, adoption, appropriateness and fidelity are relevant to the measurement of the success of the implementation of AD programmes such as ours. After an extensive literature search failed to identify any validated instruments suitable to the assessment of providers’ satisfaction with AD programmes, we decided to develop a novel instrument for this purpose.
Stage 1. Literature review
The first step of this endeavour consisted of a literature review to identify and describe themes related to prescribers’ satisfaction with AD and educational outreach interventions in the literature. A structured literature search was developed to retrieve manuscripts published until April 2018 that reported having assessed physician’s satisfaction with AD and educational outreach interventions. The search strategy included a combination of the following terms, as text words and MeSH terms: “academic detailing”, “educational outreach”, “experience”, “satisfaction” and “acceptability”. We included qualitative, quantitative or mixed-methods studies. Reports that included the assessment of constructs related to satisfaction (e.g. acceptability, experience) were also eligible. In addition, the references of articles that met the inclusion criteria were scanned to identify additional sources. Two independent reviewers (ST and AR) screened the titles identified, and discrepancies were resolved by a third researcher (AM).
Stage 2. Development of an initial set of candidate items
The development of candidate items was based on the following elements: (1) instruments or questions used to measure the same construct in other AD studies; (2) themes identified in the literature review. An initial set of candidate items was developed in a core-group meeting based on collective input and consensus. The core group of researchers includes members with expertise in educational measurement, psychometrics and survey development. At this stage, an ordinal rating response scale, ranging from ‘not at all’ = 1, ‘slightly’ = 2, ‘moderately’ = 3, ‘very’ = 4 and ‘extremely’ = 5, was selected to accompany these items.
Stage 3. Content-experts consultation (face and content validity)
Face and content validity of the items as well as response scales included in the initial instrument were evaluated by an expert panel consisting of one pharmacist, two physicians and one advanced nurse practitioner with extensive clinical expertise in chronic pain management. The content experts were asked to comment on whether the items were adequately worded and covered the content of interest. During this stage, the experts consulted were also invited to propose new themes and items, if they thought it was relevant.
Psychometric assessment
Stage 4. Study sample and data collection
At this stage, a set of 10 candidate items was field tested. Data were collected from a sample of PCPs within a large healthcare system located in the greater Chicagoland area. Participants received an AD intervention on safe and appropriate opioid prescribing (implemented in a two-visit format) administered by trained detailers with a pharmacy background. Detailers were trained to deliver key messages from the CDC’s Guidelines for Prescribing Opioids for Chronic Pain.15 Respondents were invited to complete the survey by themselves at the end of each visit and instructed to seal it in a manila envelope before returning it to the detailer.
Stage 5. Analysis
Reliability
A scale’s reliability refers to the proportion of the variance in the scale scores that can be attributed to the true score in the latent variable rather than to random error.16 In this paper, we report the examination of the PSAD’s internal consistency reliability. The internal consistency reliability assesses the homogeneity of the items that comprise a scale; scales are said to be internally consistent if the items are highly correlated with each other. To investigate the PSAD internal consistency reliability, we looked at inter-item correlations, item–total correlations (crude and corrected) and at Cronbach’s α. Due to the categorical nature of the items’ response scale, inter-item correlation matrix and item–total correlations were estimated using Spearman rank correlation coefficient.
Validity
An instrument is considered valid if it measures what it proposes to measure, or in other words, if the subject’s position in the latent trait is the underlying cause of the observed item variance.
Construct validity
The examination of construct validity assesses how well an instrument measures the underlying construct of interest. It involves the specification of factors/constructs that account for the variance of the item’s responses as well as the hypothesized relationships amongst them.16–18
In this paper, we report the assessment of three aspects of construct validity: factor analysis, convergent validity and criterion validity. However, it should be noted that, to date, there is no published gold standard for the assessment of the satisfaction of subjects involved in AD interventions. Accordingly, for our instrument, the psychometric validity can arguably only be implied rather than proved.
Factor analysis
Exploratory factor analysis (EFA) was used to explore the factor structure of the measure. In preparation for the EFA, the Kaiser–Meyer–Olkin measure of sampling adequacy test (KMO) was used to evaluate the suitability of the respondent data for factor analysis and the extent of the interrelation of various variables.19 Briefly, the KMO index ranges from 0 to 1, with values above 0.50 being considered acceptable for factor analysis.19 According to our literature search, we hypothesized that all items of the questionnaire would load in a single underlining dimension. The factor analysis also served as a basis to corroborate the unidimensionality of the scale. A polychoric correlation matrix was used to account for the categorical nature of the item data.20 The final model was developed using principal axis factoring, rather than maximum likelihood, due to the non-parametric distribution of our data. Regarding the rotation method applied to the correlation matrix, we first attempted to use an oblique rotation (Promax) that allows for correlations between factors, which is expected when dealing with data of this nature.21,22 After obtaining a single factor, we switched to an orthogonal rotation (Varimax) to allow for a simpler interpretation.21
Convergent validity
Convergent validity measures the degree of overlap between measures of theoretically related constructs.23 For this purpose, we estimated the correlation between our instrument and a measure of detailers’ perceived efficacy of the visit (DAVE).24 Whilst the constructs assessed by these two instruments do not perfectly overlap, both instruments measure aspects related to the acceptability and feasibility of AD interventions. The convergence between the overall score of the PSAD and the overall score of the DAVE was estimated using Pearson product-moment correlation coefficient. The overall scores were calculated by summing the scores obtained in each item. Correlation coefficients were considered strong if ≥0.7, moderate if <0.7 and ≥0.3, or weak if <0.3.25 High and moderate correlations indicate the measures are assessing similar constructs.
Criterion-related validity (predictive validity)
Criterion validity assesses whether measures have an empirical association with an external criterion or gold standard.16 In this study, we hypothesized that high reported satisfaction with the intervention would be positively associated with individual’s likelihood to participate in a second visit. To test this hypothesis, we used independent samples t-test to compare the mean PSAD baseline scores of individuals that accepted participating in a second visit with those who did not.
Item response theory
Item response theory models were applied to further examine the dimensionality of the measure and to explore the performance of the items in terms of the amount of information that each item yields for the estimation of the individuals’ ability levels. To achieve this purpose, a Graded Response Model was fitted to the data. Item and test information functions were estimated and graphically plotted to assess the amount of information yielded by each item for the estimation of the latent trait (Theta). In item response theory, the term ‘information’ is used to designate the precision with which the latent trait is measured across different levels of ability when using a given item. Therefore, a high information value is associated with a standard error of measurement.26 The test information function is given by the sum of item information functions.
Results
Literature review
The search strategy retrieved 66 unique titles. After applying the inclusion and exclusion criteria defined in the review protocol, seven titles were retained for further inspection. The following six constructs were identified in this stage: acceptability, feasibility, usefulness, perception of efficacy, overall satisfaction with the quality of the interviews, and willingness to repeat the experience.
Item development
During the first stage of item development, eight candidate items were proposed based on the constructs identified in the literature review (Figure 1). The content experts enrolled in the consultation stage confirmed the face and content validity of the candidate items. Two additional items related to the prescribers’ willingness to change their practice and the consistency of the message with their practice were generated after experts’ consultation. Figure 1 shows the list of candidate items that were carried out to the field-testing stage.
Figure 1.
Final pilot instrument for AD programme.
Sample population and survey administration
Data were collected between June and August 2018. In total, 183 PCPs participated and completed the PSAD at the first visit, whilst 157 participated and completed the PSAD at both the first and second visits (84.9% retention rate). Table 1 shows that the study sample, at the time of the first visit, had a mean time of practice experience of 14.9 (SD, 11.9) years. Approximately 55% of the healthcare providers recruited were women; in terms of their type of practice, most respondents were Doctors of Medicine (52%) and Doctor of Osteopathic Medicine (33.3%). First visits had an average length of 14.3 (SD, 0.2) minutes, whilst second visits lasted for 11.6 (SD, 0.2) minutes on average. More information about our study sample can be found in Table 1.
Table 1.
Provider characteristics for visits 1 and 2.
| Provider characteristics | Visit 1 (n=183) | Visit 2 (n=157) |
|---|---|---|
| Gender, n (%) | ||
| Women | 101 (55.19) | 85 (54.14) |
| Men | 82 (44.81) | 72 (45.86) |
| Provider type, n (%) | ||
| DO | 61 (33.33) | 51 (32.48) |
| MD | 96 (52.46) | 83 (52.87) |
| NP | 18 (9.84) | 16 (10.19) |
| PA | 8 (4.37) | 7 (4.46) |
| Resident, n (%) | 31 (16.94) | 19 (12.10) |
| Years of practice, mean (SD) | 14.92 (11.88) | 15.31 (11.59) |
DO, Doctor of Osteopathic Medicine; MD, Doctor of Medicine; NP, Nurse Practitioner; PA, Physician Assistant; SD, standard deviation.
Psychometric analysis
Table 2 shows the distribution of responses to each of the candidate items for visits 1 and 2. All items, except for item 9 (likelihood to change), present a left-skewed distribution with more than half of participants selecting the option ‘extremely’, the most positive extreme of the response scale. This pattern of response could be indicative of the presence of a ceiling effect. Another aspect of response patterns that were inspected in this analysis was the missingness, given its theoretical association with the acceptability of items. Notably, all the candidate items displayed a low proportion of missing values (≤1.64%). The mean total score was 43.8 (SD, 6.1) for visit 1 and 44.2 (SD, 6.0) for visit 2.
Table 2.
Descriptive statistics for visits 1 and 2.
| Visit 1 (n=182) | 1: Not at all | 2: Slightly | 3: Moderately | 4: Very | 5: Extremely | Missing | Item mean score (SD) |
|---|---|---|---|---|---|---|---|
| n (%) | n (%) | n (%) | n (%) | n (%) | n (%) | ||
| 1: The detailer was knowledgeable | 0 (0) | 1 (0.6) | 11 (6.0) | 67 (36.6) | 103 (56.3) | 1 (0.6) | 43.8 (6.13) |
| 2: The detailer was an effective communicator | 0 (0) | 1 (0.6) | 5 (2.7) | 58 (31.7) | 118 (64.5) | 1 (0.6) | |
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 1 (0.6) | 5 (2.7) | 26 (14.2) | 52 (28.4) | 98 (53.6) | 1 (0.6) | |
| 4: The printed material was useful | 1 (0.6) | 7 (3.8) | 26 (14.2) | 52 (28.4) | 95 (51.9) | 2 (1.1) | |
| 5: I would be receptive to future visits | 4 (2.2) | 4 (2.2) | 30 (16.4) | 59 (32.2) | 84 (45.9) | 2 (1.1) | |
| 6: This topic was relevant to my practice | 0 (0) | 3 (1.64 | 16 (8.7) | 55 (30.1) | 107 (58.5) | 2 (1.1) | |
| 7: This is an important topic | 0 (0) | 0 (0) | 4 (2.2) | 48 (26.2) | 130 (71.0) | 1 (0.6) | |
| 8: The key messages are feasible to implement in my practice | 1 (0.6) | 2 (1.2) | 14 (7.7) | 53 (28.9) | 110 (60.1) | 3 (1.6) | |
| 9: My practice is likely to change as a result of this visit | 11 (6.0) | 27 (14.8) | 52 (28.42 | 35 (19.1) | 55 (30.1) | 3 (1.6) | |
| 10: The key messages were consistent with my practice | 0 (0) | 0 (0) | 14 (7.7) | 53 (28.9) | 115 (62.8) | 1 (0.6) | |
| Visit 2 ( n =158) | |||||||
| 1: The detailer was knowledgeable | 0 (0) | 0 (0) | 8 (5.1) | 44 (28.0) | 104 (66.2) | 1 (0.6) | 44.2 (6.01) |
| 2: The detailer was an effective communicator | 0 (0) | 1 (0.6) | 6 (3.8) | 42 (26.8) | 108 (68.8) | 1 (0.6) | |
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 1 (0.6) | 1 (0.6) | 17 (10.8) | 43 (27.4) | 94 (59.9) | 1 (0.6) | |
| 4: The printed material was useful | 0 (0) | 4 (2.6) | 19 (12.1) | 42 (26.8) | 92 (58.6) | 0 (0) | |
| 5: I would be receptive to future visits | 6 (3.8) | 8 (5.1) | 32 (20.4) | 37 (23.6) | 73 (46.5) | 1 (0.6) | |
| 6: This topic was relevant to my practice | 1 (0.6) | 2 (1.3) | 17 (10.8) | 47 (29.9) | 89 (56.7) | 1 (0.6) | |
| 7: This is an important topic | 0 (0) | 0 (0) | 5 (3.2) | 42 (26.8) | 110 (70.1) | 0 (0) | |
| 8: The key messages are feasible to implement in my practice | 0 (0) | 3 (1.9) | 17 (10.8) | 47 (29.9) | 90 (57.3) | 0 (0) | |
| 9: My practice is likely to change as a result of this visit | 2 (1.3) | 24 (15.3) | 39 (24.8) | 31 (19.8) | 61 (38.9) | 0 (0) | |
| 10: The key messages were consistent with my practice | 0 (0) | 3 (1.9) | 15 (9.6) | 45 (28.7) | 92 (58.6) | 2 (1.3) |
m= mean, SD = Standard Deviation
Crude and corrected item–total correlations as well as Cronbach’s α estimates for visits 1 and 2 are presented in Table 3. In this instance, high positive values for the item–total correlation indicate high internal consistency suggesting that the items are discriminating well between highly and meagrely satisfied participants. Results for both visits consistently show that all items display strong correlations with the scale total, even when the variance of the item is removed from the total (corrected item–total correlations). The differences between crude and corrected item–total correlations are small for all items except for item 9 (likelihood to change). In respect to Cronbach’s α calculated for the full set of candidate items, high values were obtained in both visit 1 (α=0.9265) and visit 2 (α=0.9316), indicating the instrument’s internal consistency reliability. Having the values obtained for the full set of items as reference, we verified that Cronbach’s α values decrease slightly when each one of the items was omitted, with the exception of the omission of item 9 (α=0.9285 for visit 1; α=0.9282 for visit 2), suggesting that it may be more appropriate to report it separately from the other items.
Table 3.
Item–total correlation, corrected item–total correlation and Cronbach’s α for visits 1 and 2.
| Item–total | Item–total omitting item | Cronbach’s α | |
|---|---|---|---|
| Spearman’s rho (ρ) | Spearman’s rho (ρ) | Total=0.927 | |
| Visit 1 ( n =182) | Omitting each item | ||
| 1: The detailer was knowledgeable | 0.75 | 0.70 | 0.917 |
| 2: The detailer was an effective communicator | 0.73 | 0.69 | 0.919 |
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 0.82 | 0.77 | 0.916 |
| 4: The printed material was useful | 0.82 | 0.76 | 0.917 |
| 5: I would be receptive to future visits | 0.83 | 0.76 | 0.918 |
| 6: This topic was relevant to my practice | 0.78 | 0.73 | 0.919 |
| 7: This is an important topic | 0.68 | 0.65 | 0.918 |
| 8: The key messages are feasible to implement in my practice | 0.76 | 0.73 | 0.915 |
| 9: My practice is likely to change as a result of this visit | 0.74 | 0.54 | 0.929 |
| 10: The key messages were consistent with my practice | 0.73 | 0.69 | 0.922 |
| Visit 2 ( n =158) | Total=0.932 | ||
| Satisfaction | Omitting each item | ||
| 1: The detailer was knowledgeable | 0.70 | 0.67 | 0.925 |
| 2: The detailer was an effective communicator | 0.70 | 0.66 | 0.926 |
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 0.78 | 0.74 | 0.923 |
| 4: The printed material was useful | 0.78 | 0.72 | 0.923 |
| 5: I would be receptive to future visits | 0.84 | 0.77 | 0.924 |
| 6: This topic was relevant to my practice | 0.82 | 0.79 | 0.923 |
| 7: This is an important topic | 0.66 | 0.63 | 0.929 |
| 8: The key messages are feasible to implement in my practice | 0.85 | 0.82 | 0.920 |
| 9: My practice is likely to change as a result of this visit | 0.81 | 0.67 | 0.928 |
| 10: The key messages were consistent with my practice | 0.79 | 0.76 | 0.923 |
Table 4 displays item–item correlations for visits 1 and 2. The correlation matrix shows that, in both visits, correlations between items were of moderate or strong strength, which reinforces the notion that these items are measuring a single underlying latent construct. Once again, item 9 (likelihood to change) is the single exception, having displayed week correlations with items 7 (ρ=0.36) and 10 (ρ=0.38) in the first visit.
Table 4.
Item–item correlation for visits 1 and 2.
| Visit 1 (n=182) | Spearman’s rho (ρ) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1: The detailer was knowledgeable | 1 | |||||||||
| 2: The detailer was an effective communicator | 0.79 | 1 | ||||||||
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 0.67 | 0.67 | 1 | |||||||
| 4: The printed material was useful | 0.57 | 0.53 | 0.73 | 1 | ||||||
| 5: I would be receptive to future visits | 0.54 | 0.57 | 0.68 | 0.65 | 1 | |||||
| 6: This topic was relevant to my practice | 0.54 | 0.57 | 0.64 | 0.63 | 0.70 | 1 | ||||
| 7: This is an important topic | 0.65 | 0.53 | 0.60 | 0.60 | 0.53 | 0.65 | 1 | |||
| 8: The key messages are feasible to implement in my practice | 0.65 | 0.59 | 0.66 | 0.64 | 0.58 | 0.62 | 0.72 | 1 | ||
| 9: My practice is likely to change as a result of this visit | 0.40 | 0.41 | 0.43 | 0.45 | 0.53 | 0.46 | 0.36 | 0.44 | 1 | |
| 10: The key messages were consistent with my practice | 0.65 | 0.63 | 0.61 | 0.58 | 0.58 | 0.59 | 0.61 | 0.66 | 0.38 | 1 |
| Visit 2 ( n =158) | ||||||||||
| 1: The detailer was knowledgeable | 1 | |||||||||
| 2: The detailer was an effective communicator | 0.84 | 1 | ||||||||
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 0.71 | 0.75 | 1 | |||||||
| 4: The printed material was useful | 0.61 | 0.65 | 0.70 | 1 | ||||||
| 5: I would be receptive to future visits | 0.56 | 0.57 | 0.65 | 0.64 | 1 | |||||
| 6: This topic was relevant to my practice | 0.59 | 0.58 | 0.61 | 0.61 | 0.72 | 1 | ||||
| 7: This is an important topic | 0.53 | 0.44 | 0.52 | 0.50 | 0.55 | 0.73 | 1 | |||
| 8: The key messages are feasible to implement in my practice | 0.58 | 0.59 | 0.69 | 0.70 | 0.69 | 0.74 | 0.69 | 1 | ||
| 9: My practice is likely to change as a result of this visit | 0.44 | 0.43 | 0.51 | 0.55 | 0.60 | 0.59 | 0.46 | 0.65 | 1 | |
| 10: The key messages were consistent with my practice | 0.60 | 0.56 | 0.66 | 0.62 | 0.65 | 0.70 | 0.67 | 0.78 | 0.55 | 1 |
ρ, Spearman’s rank correlation coefficient
All reported values were statistically significant (p<0.1)
In both visits, KMO values obtained for the full set of items as well as for each individual item are all well above 0.8, suggesting that the data are adequate to perform factor analysis (Table 5). The principal axis factoring model identified a single factor with an eigenvalue greater than 1.00 that accounted for most of the underlining variance in the data. These findings were consistent in both visits 1 and 2. The factor loadings (FL) observed for all items were high, with item 9 presenting the lowest FL and highest uniqueness (U) in visit 1 (FL=0.54; U=0.71) being replaced in this by item 7 in visit 2 (FL=0.65; U=0.57) (Table 5).
Table 5.
Item factor loading and uniqueness for visits 1 and 2.
| Visit 1 (n=182) | Eigenvalue | Factor loadings | Uniqueness | KMO |
|---|---|---|---|---|
| ‘Satisfaction’ factor | 5.80 | Total=0.92 | ||
| 1: The detailer was knowledgeable | 0.82 | 0.33 | 0.86 | |
| 2: The detailer was an effective communicator | 0.75 | 0.43 | 0.86 | |
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 0.82 | 0.32 | 0.95 | |
| 4: The printed material was useful | 0.81 | 0.34 | 0.95 | |
| 5: I would be receptive to future visits | 0.79 | 0.37 | 0.91 | |
| 6: This topic was relevant to my practice | 0.76 | 0.42 | 0.93 | |
| 7: This is an important topic | 0.77 | 0.40 | 0.90 | |
| 8: The key messages are feasible to implement in my practice | 0.82 | 0.33 | 0.93 | |
| 9: My practice is likely to change as a result of this visit | 0.54 | 0.71 | 0.96 | |
| 10: The key messages were consistent with my practice | 0.68 | 0.53 | 0.96 | |
| Visit 2 ( n =158) | ||||
| ‘Satisfaction’ factor | 5.69 | Total=0.91 | ||
| 1: The detailer was knowledgeable | 0.73 | 0.47 | 0.85 | |
| 2: The detailer was an effective communicator | 0.70 | 0.51 | 0.83 | |
| 3: Academic detailing is an effective way to stay updated on important topic(s) | 0.79 | 0.37 | 0.94 | |
| 4: The printed material was useful | 0.77 | 0.41 | 0.95 | |
| 5: I would be receptive to future visits | 0.77 | 0.40 | 0.94 | |
| 6: This topic was relevant to my practice | 0.79 | 0.37 | 0.93 | |
| 7: This is an important topic | 0.65 | 0.57 | 0.90 | |
| 8: The key messages are feasible to implement in my practice | 0.86 | 0.26 | 0.90 | |
| 9: My practice is likely to change as a result of this visit | 0.67 | 0.56 | 0.95 | |
| 10: The key messages were consistent with my practice | 0.78 | 0.39 | 0.91 |
KMO, Kaiser–Meyer–Olkin measure of sampling adequacy test.
An attempt to explore construct validity was made using the available data. Convergent validity was tested by correlating the scores resulting from the sum of the full set of the PSAD measure candidate items with the DAVE. The results obtained showed that these measures are weakly correlated (r=0.2115, p<0.005 for visit 1; r=0.3110, p<0.000 for visit 2).
The criterion-based validity of this measure was not supported by our findings. Based on the data available, we failed to reject the null hypothesis that satisfaction scores in the baseline visit were not associated with a higher likelihood of agreeing to participate in the follow-up visit (43.41 for those that accepted visit 2 versus 46.1 for those that rejected visit 2; p=0.037).
Considering the results obtained in the item–item correlation matrix and the factor analysis, we concluded that the items have a sufficiently high correlation to justify the estimate of a single latent trait score. This is relevant as, under the item response theory, each item of the test measures the underlying latent trait; thus, most of the models applied assume the unidimensionality of the group of items being tested. Figure 2 shows the amount of information plotted against ability (item information function), for each of the ten candidate items evaluated in this analysis, for visits 1 and 2. Due to the categorical nature of the data, the item information functions are neither unimodal nor symmetric. This happens because each category contributes its own amount of information, which may peak over a different ability range. The results show that Item 9 (likelihood to change) yields much less information than the other candidate items. The graphic representation of the item information function also shows that all the curves begin to drop after the mean (ability=0), indicating the presence of a ceiling effect. Figure 3 shows the test information function for visits 1 and 2. No significant differences were found when comparing the test information functions estimated with and without the contribution of Item 9. The results were consistent between visits 1 and 2.
Figure 2.
Item information function for visits 1 and 2.
Figure 3.
Test information function graphs with and without item 9 (visit 2).
Discussion
This paper reports the development of a new measure to assess prescriber’s satisfaction with AD and educational outreach interventions – the PSAD. A series of carefully conceived steps were undertaken to ensure that the items included in this instrument cover a substantial range of constructs relevant for the assessment of satisfaction with an AD intervention, whilst being comprehensive enough to apply to a broad range of themes and settings. This systematic process included literature reviews, content-expert review and psychometric testing in over 180 PCPs. This process narrowed down an initial list of 10 candidate items to a 9-item unidimensional measure. In the end, we obtained a short measure, amenable to be applied in only a couple of minutes, that seemed to be well received and well understood by the target population as it had a very low rate of missing data.
The literature review stage identified several publications in the field of AD and educational outreach that collected and discuss data on prescriber’s satisfaction with interventions. However, a validated measure was not used in any of the identified literature. Rather, single items or non-psychometric lists of questions were applied.11–13 We believe that the development of a measure with sound psychometric properties will allow for a standardized and systematic collection of data on prescriber satisfaction, thus contributing information about the process of implementation that can help to monitor and improve AD programmes. Items related to a detailer’s effectiveness of communication and knowledge may be useful in monitoring team performance and helping to inform training and messaging decisions. Likewise, the item related to the usefulness of printed materials may support decisions regarding the need to refine the content and format of materials used during the visit.
Classic test theory methods were applied to assess the measure’s internal consistency reliability, criterion validity and convergent validity and to explore its factor structure. The results obtained in this first application of the measure support its internal consistency reliability, but future research is needed to further establish other forms of reliability (e.g. inter-rater and test-retest reliability). As noted in the results section, the slight increase in the Cronbach’s α, when excluding item 9, suggests it might be separately reported. Our findings did not support this measure’s criterion-based validity. However, it seems relevant to highlight that the choice of the criterion was made in an opportunistic fashion, contingent on the data that was available to us at this stage. Future research applying the PSAD measure should revisit this issue. Regarding the attempt to establish the PSAD measure’s convergent validity, it is important to note that there is no gold standard to measure the prescribers’ satisfaction currently available in the literature; therefore, we are limited in our ability to interpret the weak correlation coefficient obtained in our analysis. Evidence obtained in the EFA strongly suggests that the PSAD measure is a unidimensional measure of satisfaction, which is compatible with the factor structure hypothesized a priori and supports the construct validity.
Item response theory methods were applied to examine the amount of information yielded by each of the candidate items. Once again, the item about ‘likelihood to change’ (item 9) displayed a weak performance when compared with the remaining item pool. The distribution of the responses to this instrument as well as the graphical representation of the item and test information functions suggest that there is an apparent ceiling effect in this measure. Literature in the field of satisfaction measurement proposes that items measuring basic features of the intervention will differentiate amongst individuals at the lower end of the satisfaction continuum, and items that tap specific features or services that exceed expectations will do the same at the upper end.27–29
Whilst the item related to the prescribers’ likelihood to change their prescribing behaviour because of the AD visit (item 9) covers an important desirable outcome of AD interventions, our results suggest that it is not a good fit in a satisfaction measure. The version of the PSAD developed in this study is composed of nine items covering the constructs knowledge, effectiveness of communication, effectiveness, usefulness, willingness to repeat experience, acceptability, acceptability, feasibility and consistency.
Finally, there are several limitations to this study that we aim to address in future endeavours. The present study was not strictly designed for measure validation but was rather conducted opportunistically, leveraging an ongoing CDC-funded AD intervention. As a result, some of the information relevant to test the construct validity of the newly developed instrument was not collected. Furthermore, this study may not be powered to test hypothesized relationships (i.e. criterion-based and convergent validity hypothesis testing); therefore, those results should be interpreted carefully. Future research should assess its generalizability as an evaluative tool for AD and the extent to which the measure correlates with changes in prescribing and patient outcomes.
Conclusion
The PSAD measure is a new 9-item instrument intended to assess healthcare professional’s satisfaction with AD. Considering the broad nature of its items, this measure has the potential of being applied in educational outreach programmes beyond the scope of opioid prescribing. The results obtained in this first field test support the reliability and validity of this new instrument, and it is hoped that the final version of this measure will generalize to broad use as an evaluative tool for AD.
Acknowledgements
Academic Detailers from University of Illinois at Chicago College of Pharmacy: Victoria Kulbokas, Esther Lee, Shannon Menard, Ammarah Nadeem, Dayna Redini, Aleksandra Ruseva, Sarette T Tilton, Nevena Varagic. AMITA Health: Darin Jordan, MD; Ankur Dave, MD; Reinhold Llerena, MD. Illinois Prescription Monitoring Program: Craig Berberet; Stan Murzynski; Ed Dowllar; Sarah Pointer, PharmD; Andrew Hollo. Expert panel: David Liebowitz, MD; Scott Glaser, MD; Chris Herndon, PharmD; and Julie Adkins DNP.
Footnotes
Contributions: ALM is the lead author. All authors took part in writing the manuscript and approval of the final draft. All named authors meet the International Committee of Medical Journal Editors (ICMJE) criteria for authorship for this article, take responsibility for the integrity of the work as a whole and have given their approval for this version to be published.
Disclosure and potential conflicts of interest: Dr Fischer has consulted for Alosha Health. The remaining authors declare that they have no conflicts of interest relevant to this manuscript. The views expressed in this manuscript are those of the authors and do not necessarily represent the views of those acknowledged or funding sources. MAF is a clinical consultant for Alosa Health, a health education non-profit that provides academic detailing services. The International Committee of Medical Journal Editors (ICMJE) Potential Conflicts of Interests form for the authors is available for download at: https://www.drugsincontext.com/wp-content/uploads/2021/12/dic.2021-9-7-COI.pdf
Funding declaration: All phases of this study were supported by the Centers for Disease Control and Prevention: Prevention for States (PfS) via the Illinois Department of Human Services and Illinois Prescription Monitoring Program provided support for the academic detailing component of this study (Grant#: 49I1099M40).
Correct attribution: Copyright © 2022 Monteiro AL, Smart M, Saffore CD, Lee TA, Tilton ST, Fischer MA, Pickard AS. https://doi.org/10.7573/dic.2021-9-7. Published by Drugs in Context under Creative Commons License Deed CC BY NC ND 4.0.
Article URL: https://www.drugsincontext.com/development-of-a-measure-of-prescriber-satisfaction-with-academic-detailing-the-psad
Provenance: Invited; externally peer reviewed.
Drugs in Context is published by BioExcel Publishing Ltd. Registered office: Plaza Building, Lee High Road, London, England, SE13 5PT.
BioExcel Publishing Limited is registered in England Number 10038393. VAT GB 252 7720 07.
For all manuscript and submissions enquiries, contact the Editorial office editorial@drugsincontext.com
For all permissions, rights and reprints, contact David Hughes david.hughes@bioexcelpublishing.com
References
- 1.Yeh JS, Van Hoof TJ, Fischer MA. Key features of academic detailing: development of an expert consensus using the Delphi method. Am Health Drug Benefits. 2016;9(1):42–50. [PMC free article] [PubMed] [Google Scholar]
- 2.Soumerai SB, Avorn J. Principles of educational outreach (‘academic detailing’) to improve clinical decision making. JAMA. 1990;263(4):549–556. [PubMed] [Google Scholar]
- 3.O’Brien MA, Rogers S, Jamtvedt G, et al. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007;4:CD000409. doi: 10.1002/14651858.CD000409.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Elnaem MH, Nik Mohamed MH, Huri HZ. Pharmacist-led academic detailing improves statin therapy prescribing for Malaysian patients with type 2 diabetes: Quasi-experimental design. PLoS ONE. 2019;14(9):e0220458. doi: 10.1371/journal.pone.0220458. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ragan AP, Aikens GB, Bounthavong M, Brittain K, Mirk A. Academic detailing to reduce sedative-hypnotic prescribing in older veterans. J Pharm Pract. 2021;34(2):287–294. doi: 10.1177/0897190019870949. [DOI] [PubMed] [Google Scholar]
- 6.Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157(4):408–416. [PMC free article] [PubMed] [Google Scholar]
- 7.Larson MJ, Browne C, Nikitin RV, et al. Physicians report adopting safer opioid prescribing behaviors after academic detailing intervention. Subst Abus. 2018;39(2):218–224. doi: 10.1080/08897077.2018.1449175. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. doi: 10.1007/s10488-010-0319-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Schillinger D. An introduction to effectiveness, dissemination and implementation research From the Series: UCSF Clinical and Translational Science Institute (CTSI) Resource Manuals and Guides to Community-Engaged Research. In: Fleisher P, editor. Clinical Translational Science Institute Community Engagement Program. University of California; San Francisco: 2010. [Accessed January 5, 2022]. p. 2. https://accelerate.ucsf.edu/files/CE/edi_introguide.pdf . [Google Scholar]
- 10.Lorenzi NM, Unertl KM. Effective strategies for implementation and evaluation innovation in public e-health innovations. Stud Health Technol Inform. 2012;172:45–53. [PubMed] [Google Scholar]
- 11.Costa JO, Almeida-Brasil CC, Godman B, et al. Implementation of clinical guidelines in Brazil: should academic detailing be used? J Pharm Health Services Res. 2016;7(2):105–115. doi: 10.1111/jphs.12133. [DOI] [Google Scholar]
- 12.Anthierens S, Verhoeven V, Schmitz O, Coenen S. Academic detailers’ and general practitioners’ views and experiences of their academic detailing visits to improve the quality of analgesic use: process evaluation alongside a pragmatic cluster randomized controlled trial. BMC Health Serv Res. 2017;17(1):841. doi: 10.1186/s12913-017-2797-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.May F, Simpson D, Hart L, Rowett D, Perrier D. Experience with academic detailing services for quality improvement in primary care practice. Qual Saf Health Care. 2009;18(3):225–231. doi: 10.1136/qshc.2006.021956. [DOI] [PubMed] [Google Scholar]
- 14.Saffore CD, Tilton ST, Crawford SY, et al. Identification of barriers to safe opioid prescribing in primary care: a qualitative analysis of field notes collected through academic detailing. Br J Gen Pract. 2020;70(697):e589–e597. doi: 10.3399/bjgp20X711737. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Dowell D, Haegerich TM, Chou R. CDC guideline for prescribing opioids for chronic pain – United States, 2016. JAMA. 2016;315(15):1624–1645. doi: 10.1001/jama.2016.1464. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Streiner DL, Norman GR, Cairney J. Health Measurement Scales: A Practical Guide to their Development and Use. Oxford University Press; USA: 2015. [Google Scholar]
- 17.Patrick DL, Erickson P. Health Status and Health Policy: Quality of Life in Health Care Evaluation and Resource Allocation. Oxford University Press; 1993. [Google Scholar]
- 18.Rust J, Golombok S. Modern Psychometrics: The Science of Psychological Assessment. Routledge; 2014. [Google Scholar]
- 19.Williams B, Brown T, Onsman A. Exploratory factor analysis: a five-step guide for novices. Australas J Paramedicine. 2010;8(3):990399. [Google Scholar]
- 20.Yuan KH, Jiang G, Cheng Y. More efficient parameter estimates for factor analysis of ordinal variables by ridge generalized least squares. Br J Math Stat Psychol. 2017;70(3):525–564. doi: 10.1111/bmsp.12098. [DOI] [PubMed] [Google Scholar]
- 21.Abdi H. Factor Rotations in Factor Analyses Encyclopedia for Research Methods for the Social Sciences. Sage; 2003. pp. 792–795. [Google Scholar]
- 22.Sellbom M, Tellegen A. Factor analysis in psychological assessment research: common pitfalls and recommendations. Psychol Assess. 2019;31(12):1428–1441. doi: 10.1037/pas0000623. [DOI] [PubMed] [Google Scholar]
- 23.DeVellis RF. Scale Development: Theory and Applications. Vol. 26. Sage Publications; 2016. [Google Scholar]
- 24.Smart MH, Monteiro AL, Saffore CD, et al. Development of an instrument to assess the perceived effectiveness of academic detailing. J Contin Educ Health Prof. 2020;40(4):235–241. doi: 10.1097/CEH.0000000000000305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Schober P, Boer C, Schwarte LA. Correlation coefficients: appropriate use and interpretation. Anesth Analg. 2018;126(5):1763–1768. doi: 10.1213/ANE.0000000000002864. [DOI] [PubMed] [Google Scholar]
- 26.Hambleton RK, Swaminathan H, Rogers HJ. Fundamentals of Item Response Theory. Sage; 1991. [Google Scholar]
- 27.Moret L, Nguyen JM, Pillet N, Falissard B, Lombrail P, Gasquet I. Improvement of psychometric properties of a scale measuring inpatient satisfaction with care: a better response rate and a reduction of the ceiling effect. BMC Health Serv Res. 2007;7:197. doi: 10.1186/1472-6963-7-197. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Howorka K, Pumprla J, Schlusche C, Wagner-Nosiska D, Schabmann A, Bradley C. Dealing with ceiling baseline treatment satisfaction level in patients with diabetes under flexible, functional insulin treatment: assessment of improvements in treatment satisfaction with a new insulin analogue. Qual Life Res. 2000;9(8):915–930. doi: 10.1023/a:1008921419108. [DOI] [PubMed] [Google Scholar]
- 29.Dunsch F, Evans DK, Macis M, Wang Q. Bias in patient satisfaction surveys: a threat to measuring healthcare quality. BMJ Glob Health. 2018;3(2):e000694. doi: 10.1136/bmjgh-2017-000694. [DOI] [PMC free article] [PubMed] [Google Scholar]