Abstract
The main aim is to examine whether patients’ viewing time on information about colorectal cancer (CRC) screening before a primary care physician (PCP) visit is associated with discussion of screening options during the visit. We analyzed data from a multi-center randomized controlled trial of a tailored interactive multimedia computer program (IMCP) to activate patients to undergo CRC screening, deployed in primary care offices immediately before a visit. We employed usage time information stored in the IMCP to examine the association of patient time spent using the program with patient-reported discussion of screening during the visit, adjusting for previous CRC screening recommendation and reading speed.
On average, patients spent 33 minutes on the program. In adjusted analyses, 30 minutes spent using the program was associated with a 41% increase in the odds of the patient having a discussion with their PCP (1.04, 1.59, 95% CI). In a separate analysis of the tailoring modules; the modules encouraging adherence to the tailored screening recommendation and discussion with the patient’s PCP yielded significant results. Other predictors of screening discussion included better self-reported physical health and increased patient activation. Time spent on the program predicted greater patient-physician discussion of screening during a linked visit.
Usage time information gathered automatically by IMCPs offers promise for objectively assessing patient engagement around a topic and predicting likelihood of discussion between patients and their clinician.
Introduction
Colorectal cancer (CRC) screening reduces death from CRC and can potentially decrease the incidence of disease through detection of pre-malignant polyps [1]. However, CRC screening is underutilized in the United States, indicating the need for interventions that can increase CRC screening [2, 3].
Patient-primary care provider (PCP) discussion and recommendation of screening are particularly strong predictors of eventual CRC screening [4–6]. Thus, one approach to improving screening is to foster patient and PCP CRC screening discussion during visits.
Evidence indicates that patient-focused tailored interactive multimedia computer programs (IMCPs) encourage the discussion of CRC screening during office visits [7–11]. To our knowledge, no studies have measured the direct effect of time on task devoted to an IMCP on patient-PCP discussion. The goal of our study was to estimate the effect of time on task using a tailored intervention to predict the likelihood of CRC screening discussion in the PCP visit.
We conducted analyses using data from a randomized controlled trial of a tailored IMCP, aimed at influencing CRC screening-related outcomes in primary care [12]. The IMCP was based on the theoretical underpinnings of the Health Belief Model (HBM) and the Theory of Reasoned Action (TRA). In tandem the HBM and TRA postulate perceived susceptibility and the perceived benefit of screening in general will be associated with greater motivation to learn more about screening options [9], and thus greater time on task. Morever, time on task may be a useful implicit marker for patient interest in CRC screening.
In the current analyses, we examined whether total time spent viewing the tailored CRC screening program predicted subsequent patient-reported discussion with their PCP regarding CRC screening.
Methods
Setting and participants
Participants aged 50–75 years were recruited from five primary health care sites throughout the United States: Rochester, NY (3 health centers), Bronx, NY (1 health center), Sacramento, CA (10 offices), San Antonio, TX (2 health centers and 2 offices) and Colorado (1 site near Denver, 7 in a system 200 miles southwest of Denver). Participants were deemed eligible if they were not up to date for CRC screening based on U.S. Preventive Services Task Force [13] and American Cancer Society recommendations: no fecal occult blood testing within the prior year and no flexible sigmoidoscopy within 5 years and no colonoscopy within 10 years [14] Determination for eligibility was based on review of electronic medical records or paper medical charts. Additional eligibility criteria, assessed via telephone, included the ability to read and speak either English or Spanish, and willingness to use a touchscreen computer.
Procedures and content
The procedures have been described in detail [7]. Participants were asked to arrive 60 minutes prior to their scheduled routine medical appointment (i.e. these visits were not specifically dedicated to screening or preventive care). A research assistant (RA) greeted potential participants in the waiting room at their doctor’s office to obtain informed consent and provided them with access to the study touchscreen computer. The research assistant logged onto the computer program using a unique identification code and provided the patient with a brief overview on how to navigate the computer program using the touchscreen. Next, participants spent time completing a baseline electronic questionnaire, which included questions about socio-psychological factors that have been shown in prior research to predict CRC screening (e.g., screening preferences, self-efficacy, readiness and perceived barriers) [15]. Next, for patients randomly assigned to the experimental arm, the computer program offered CRC screening information that was tailored to their responses to the socio-psychological questionnaire items [12]. By contrast, for patients randomly assigned to the control arm, the program presented non-tailored CRC screening information developed by the National Cancer Institute. After completing the pre-visit, participants proceeded with their scheduled doctor appointments and returned to the RA in the waiting room. Using the same unique patient identification number employed pre-visit, the RA then logged them into the post-visit questionnaire with items assessing post-visit status of the socio-psychological factors and whether or not they had discussed CRC screening during the visit. Of the 595 participants randomly assigned to the intervention [8]. About 407 participants had complete data on all covariates and were included in our final model. Institutional Review Board approval was obtained from all study performance sites.
Measures
Time
While the RA was available to provide assistance if needed; the participant controlled the computer program. When participants were ready to advance to the next screen, they touched the ‘next’ icon at the bottom of the screen. The time of day (recorded in seconds) when the patient advanced each of the screens were recorded automatically in a program back-end database. Duration of total time was calculated based on the difference (seconds) in the program section entry and exit timestamp points. In addition, to assessing total time spent viewing the program, we assessed the duration of time the patient spent on the personally tailored modules that specifically addressed CRC screening. Time spent on personally tailored modules was based on the time spent on program sections that attempted to (1) bolster CRC screening knowledge, (2) bolster knowledge regarding potential risks to screening and inconveniences of FOBT or colonoscopy, (3) provide the patient with a personally tailored CRC screening recommendation, (4) encourage the patient to adhere to the screening recommendation, and (5) encourage the patient to engage in a discussion with their PCP regarding their tailored screening recommendation [12].
The amount of time the patient spent answering questions in the demographic section (such as race, age, ethnicity, and income) of the program was subtracted from the total time spent on the program. The difference was then included in the final model to account for differences in reading speed.
Demographics
Socio-demographic characteristics included age (years), sex (male; female), software use language (Spanish or English), race/ethnicity (Hispanic; Black; White; Other race), education level (less than high school; high school graduate; some college; or college graduate or more) and income (<$10 000, $10 000 to $14 999; $15 000 to $24 999; $25 000 to $50 000; >$50 000).
Health
Health measures included the SF-12 Mental and Physical Component Summary scores with a possible range 0–100 and a higher score is equated to a better mental or physical health (Cronbach’s alpha = 0.91) [16] We measured health literacy using responses to the question [17], ‘How often do you need to have someone help you when you read instructions, pamphlets or other written material from your doctor or pharmacy?’ Answers ranged from 1 = never to 5 = always.
Patient attitudinal characteristics included screening confidence (undergo screening versus not), prior screening recommendation from their provider, and the patient’s level of activation, assessed using the 13-item Patient Activation Measure(PAM) (Cronbach’s alpha= 0.90) [18]. Five Factor Model personality factors (agreeableness, conscientiousness, extraversion, neuroticism, and openness) were measured using the Big Five Inventory [19, 20](factor scores ranges 1–5, higher scores = higher standing on the factor) (Cronbach’s alpha = 0.90).
We assessed CRC discussion based on the participant’s self-report immediately after their visit. Specifically, the post-visit survey asked participants if ‘During the visit you just had with your provider, did you talk about colon cancer screening with your provider?’ (1 = yes, 0= no).
Analyses
We used a logistic regression of CRC discussion on the time variables, controlling for confounding factors and significant predictors. The confounding factors were time spent on the demographic section and previous CRC screening recommendation. The other predictors were identified using a stepwise elimination procedure: which included all variables listed in Table I with a univariate P values < 0.40. Categorical variables were compared using Chi-square test and continuous variables by Student’s t test. Variables with a P values <0.05 were included in the final model. To aid in the interpretation of any changes in probability of patient-provider discussion, we calculated the marginal effects using the estimates from the final multivariable model.
Table I.
CRC screening discussion |
P values | ||
---|---|---|---|
Yes (n = 345) | No (n = 240) | ||
Total time (minutes) (mean) (std) | 34 (21) | (33) (23) | 0.58 |
Demographic section time (min) (mean) | 2 | 2 | 0.06 |
Age, (mean) (years) | 57 | 58 | 0.14 |
Confidence to screen (range 1–5) (mean) | 4 | 4 | 0.01 |
Prior CRC screening recommendation | 49% | 40% | 0.03 |
Female | 63% | 69% | 0.13 |
Black | 27% | 22% | 0.17 |
White | 24% | 17% | 0.05 |
Other | 4% | 5% | 0.89 |
No. years with PCP (mean) | 3 | 3 | 0.36 |
No health insurance coverage | 17% | 24% | 0.05 |
Hispanic | 45% | 57% | 0.01 |
Low health literacy | 12% | 19% | 0.04 |
Education | 0.00 | ||
Less than HS | 15% | 22% | |
HS graduate/GED | 25% | 21% | |
Some College | 25% | 16% | |
College graduate or < | 20% | 16% | |
Personality Factor Scores (range 1-5) (mean) | |||
Extraversion | 3.3 | 3.2 | 0.56 |
Agreeableness | 3.9 | 3.9 | 0.07 |
Conscientiousness | 3.7 | 3.8 | 0.63 |
Neuroticism | 2.8 | 2.7 | 0.06 |
Openness | 3.6 | 3.5 | 0.47 |
Patient Activation Score (pre-visit) (range 1–100) (mean) | 56.68 | 54.71 | 0.25 |
Never use computers | 32% | 42% | 0.02 |
Prefer not to screen | 7% | 9% | 0.33 |
SF-12 physical | 42.90 | 41.10 | 0.11 |
SF-12 mental | 44.60 | 46.80 | 0.05 |
Participant enrollment site | 0.00 | ||
Bronx, NY | 19% | 32% | |
Denver, CO | 15% | 17% | |
San Antonio, TX | 14% | 20% | |
Rochester, NY | 24% | 17% | |
Sacramento, CA | 28% | 15% |
P values are for comparison between groups (chi-squared test for categorical variables, t test for continuous variables.
Mean time spent on program overall was 33 min (std 22 min).
Similar to the preceding analysis, we assessed whether time spent viewing the personally tailored modules was related to discussion of CRC screening using logistic regression. The analyses were adjusted for time spent on each module, time spent on the demographic section, mental and physical component score, patient activation, health literacy, and prior CRC screening recommendation.
Post model specification testing denoted our models were properly fit. Data were analyzed using STATA version 12.1 (Copyright © 1985-2011 by StataCorp LP., College Station, TX, USA).
Results
Fifty-nine percent of the participants discussed CRC screening options during their visit with their provider. Sociodemographic and other characteristics of the participants are presented in Table I. The mean total time spent on the program was 34 minutes among those who discussed CRC screening with their PCP and 33 minutes among those who did not. Compared with participants who did not discuss CRC screening during the visit, those who did were more likely to have a prior screening recommendation from their PCP, had higher patient activation, and better self-reported health.
In adjusted analyses, 30 minutes spent using the program was associated with a 41% increase in the odds of the patient having a discussion with their PCP (1.04, 1.59, 95% CI). Other predictors of CRC screening discussion included better physical health and greater patient activation. An increase in better health was associated with a 2% increased odds of screening discussion (1.004, 1.04 95% CI). Whereas, an increase in patient activation was related to a 2% increase in the odds of discussion (1.003, 1.04, 95% CI). Participants who indicated their PCP had previously recommended they get CRC screening had 61% greater odds of having a discussion during their visit compared to those who had not received a previous recommendation (95% CI 1.04, 2.49) (Table II).
Table II.
Odds Ratio | P > z | [95% Conf. | Interval] | |
---|---|---|---|---|
Total time (30 min) | 1.41 | 0.02 | 1.06 | 1.89 |
Demographic section time (min) | 1.00 | 0.16 | 1.00 | 1.00 |
Patient activation score | 1.02 | 0.02 | 1.00 | 1.04 |
Low health literacy | 0.76 | 0.00 | 0.64 | 0.91 |
Prior CRC screening recommendation | 1.61 | 0.03 | 1.04 | 2.49 |
SF-12 physical | 1.02 | 0.02 | 1.00 | 1.04 |
SF-12 mental | 0.97 | 0.01 | 0.95 | 0.99 |
Figure 1 shows the predicted probability of CRC screening discussion associated with total time spent on the program. Those who used the program for at least the mean time (34 minutes) were at 7% increased probability of discussion compared to the baseline probability of 0.50. The difference in probability between using the program twenty minutes greater than the mean total time (0.62) and twenty minutes less than the mean total time (0.53) spent on the program results in a 0.09 increase in the predicted probability of patient-provider CRC screening discussion.
Regarding the analyses for specific tailored program modules, a one minute increase in the time spent on the module that encouraged discussion of the personally tailored screening recommendation during the office visit was associated with an 206% increase in the adjusted odds of discussion (1.42, 6.56, 95% CI). An additional minute of time spent on the module that encouraged adherence to the personally tailored screening recommendation was associated with a 16% increase in the odds of discussion (1.02, 1.32, 95% CI). For the remaining specific tailored modules, viewing time was not significantly associated with CRC screening discussion (Table III).
Table III.
Tailored module | Odds ratio | 95% CI | |
---|---|---|---|
M1. Attempt to bolster CRC Screening Knowledge | 1.07 | 0.97 | 1.18 |
M2. Attempt to bolster CRC Screening Risks | 1.05 | 0.98 | 1.11 |
M3. Personally tailored CRC Screening recommendation | 1.40 | 0.90 | 2.17 |
M4. Encourage adherence to personally tailored screening recommendation | 1.16 | 1.02 | 1.32 |
M5. Encourage discussion of personally tailored screening recommendation with PCP | 3.06 | 1.42 | 6.56 |
Adjusted using all covariates in Table II except total time.
M1–M5 are the five personally tailored modules of the program.
Discussion
In our study, participants who spent more time using the program were more likely to report they had discussed CRC screening with their physician [15, 21–24]. Moreover, time using modules 4 (encourage adherence to the personally tailored screening option) and module 5 (encouraging subsequent discussion of the recommendation with PCP) of the tailored information were associated with a greater likelihood of patient-provider discussion. These findings were statistically significant and provide evidence for the hypothesis that time spent viewing the program including specific modules predicts subsequent CRC screening discussion. To our knowledge, the relationship between the time a patient spends on a pre-office visit tailored intervention and its immediate association with patient-PCP communication during the visit regarding the health issue addressed in the tailored intervention has not been previously examined.
Our finding that most participants who reported discussing CRC screening with their provider spent more time on some of the tailored modules of the program, suggests that tailoring helped facilitate patient-provider communication. Possibly, the tailored information was most salient and influential in prompting participants to engage their providers in this discussion. However, we also found that prior recommendation to be screened was associated with screening discussion. Therefore, we cannot exclude the possibility that participants with greater predisposition to CRC screening spent more time viewing tailored information. In other words, we do not know whether time spent on tailored screens affected participants’ level of interest or simply represents an implicit marker of pre-disposed interest. Future research is needed to confirm the causal nature of this association. We found that the marginal probability of participants engaging in CRC screening discussion with their provider was improved with increased use time. Therefore, it may be reasonable that programs such as these can be utilized while participants are waiting to be seen. Especially, in the case of open-access clinics where patient wait times are much less predictable.
The success of computerized tailored interventions may depend in large partly on the users’ level of interest in the focus health topic(s) [25]. However, the optimal approach to measuring patient interest in tailored materials addressing CRC screening and other health topics is unclear. Questionnaires asking about interest in a health topic may be useful, but a drawback is that respondents may exaggerate their level of interest, due to social desirability bias [26]. Alternatively, information regarding the time users spend viewing different sections of a tailored IMCP, captured automatically by the programs, may offer an implicit, objective measure of patient interest in the focus health topic(s), will less likely to be affected by social desirability bias [27–29].
A strength of our study is that, to our knowledge, it is the first study to examine the relationship between time on task for a tailored intervention and patient–physician discussion. Our findings will help future studies with the development of interventions to make better use of tailored information based on patient-driven interest (using time on task as a proxy). In addition, our study includes a sample drawn from large metropolitan areas in three of the four major United States geographic regions.
Limitations
One limitation to our study is that the CRC discussion variable was based on patient self-report, without independent verification. However, patient report of counseling has shown moderate sensitivity and high specificity compared to direct observation [30, 31]. We also cannot exclude the fact that CRC screening may not have been clinically appropriate to discuss at the time of the visit for some of the participants. Moreover, while we were able to gauge patient time spent on the computer program we were not be able to discern if they were actually actively engaged with the program content as they were using it. Competing priorities by the patient or the clinician may have crowded out time for discussion [32, 33]. Unobserved indicators such as reason for the PCP visit were not captured in our study. Additionally, we were unable to differentiate the effect that the order and content of the information may have had. The sequencing of the information during the program may have had an effect on our outcomes of interest. This may be reflected in the fact that of the five tailored modules, only the last two modules in the tailored portion of the program were found to be statistically significant. Future studies will need to be done in order to understand possible information placement effects of the intervention.
Conclusion
Our findings support the claim that more time spent on tailored, interactive computer programs designed to promote CRC screening leads to a greater likelihood of patient-physician discussion. However, additional research to address limitations and substantiate this claim is warranted.
Practice implications
Usage time information gathered automatically by IMCPs offers promise for objectively assessing patient engagement around a topic and predicting likelihood of discussion between patients and their clinician.
Acknowledgements
We are grateful to the following individuals, who facilitated recruitment and participation of patients in the study: Christina Slee, MPH, Dionne Evans-Dean, MHA, Dustin Gottfeld, BS, Lizette Macias, BS, Lori Reid, RN, and Linda Marks, MPA (Sacramento); Leticia E. Serrano, AAS (Rochester); Sandra Monroy, MA (New York City) and Brandon Tutt, MA (Colorado). We also wish to thank Robert Burnett, MA, and Simon Dvorak, BA for their programming contributions to the tailored software program. Finally, we are indebted to all of the primary care offices and patients who participated.
Funding
Funding for this research was provided by National Cancer Institute Grant # 1R01CA131386-01A1 (Jerant). The funder had no role in the study design; collection, analysis and interpretation of data; writing of the report; nor in the decision to submit the paper for publication.
Conflict of interest statement
None declared.
References
- 1.Screening for colorectal cancer: recommendation and rationale. Annals of Internal Medicine 2002; 137: 129–31. [DOI] [PubMed] [Google Scholar]
- 2.Cokkinides VE, Chao A, Smith RA. et al. Correlates of underutilization of colorectal cancer screening among U.S. adults, age 50 years and older. Prev Med 2003; 36: 85–91. [DOI] [PubMed] [Google Scholar]
- 3.Levin B, Smith RA, Feldman GE. et al. Promoting early detection tests for colorectal carcinoma and adenomatous polyps: a framework for action: the strategic plan of the National Colorectal Cancer Roundtable. Cancer. 2002; 95: 1618–28. [DOI] [PubMed] [Google Scholar]
- 4.Lafata JE, Cooper G, Divine G. et al. Patient-physician colorectal cancer screening discussion content and patients' use of colorectal cancer screening. Patient Educ Counsel 2014; 94: 76–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Laiyemo AO, Adebogun AO, Doubeni CA. et al. Influence of provider discussion and specific recommendation on colorectal cancer screening uptake among U.S. adults. Prev Med 2014; 67: 1–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Vernon SW, Bartholomew LK, McQueen A. et al. A randomized controlled trial of a tailored interactive computer-delivered intervention to promote colorectal cancer screening: sometimes more is just the same. Ann Behav Med 2011; 41: 284–99. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Jerant A, Kravitz RL, Fiscella K. et al. Effects of tailored knowledge enhancement on colorectal cancer screening preference across ethnic and language groups. Patient Educ Counsel 2013; 90: 103–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Jerant A, Kravitz RL, Rooney M. et al. Effects of a tailored interactive multimedia computer program on determinants of colorectal cancer screening: a randomized controlled pilot study in physician offices. Patient Educ Counsel 2007; 66: 67–74. [DOI] [PubMed] [Google Scholar]
- 9.Jerant A, Sohler N, Fiscella K. et al. Tailored interactive multimedia computer programs to reduce health disparities: opportunities and challenges. Patient Educ Counsel 2011; 85: 323–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Shah R, Franks P, Jerant A. et al. The effect of targeted and tailored patient depression engagement interventions on patient-physician discussion of suicidal thoughts: a randomized control trial. J Gen Intern Med 2014; 29: 1148–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Bass SB, Gordon TF, Ruzek SB. et al. Developing a Computer Touch-Screen Interactive Colorectal Screening Decision Aid for a Low-Literacy African American Population Lessons Learned. Health Promotion Practice 2013; 14: 589–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jerant A, Kravitz RL, Sohler N. et al. Sociopsychological tailoring to address colorectal cancer screening disparities: a randomized controlled trial. Ann Family Med 2014; 12: 204–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lieberman DA, Rex DK, Winawer SJ. et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer. Gastroenterology 2012; 143: 844–57. [DOI] [PubMed] [Google Scholar]
- 14.American Cancer Society guidelines. American Cancer Society guidelines for the early detection of cancer. 2013.
- 15.Lee W, Nelson R, Mailey B. et al. Socioeconomic factors impact colon cancer outcomes in diverse patient populations. J Gastrointest Surg 2012; 16: 692–704. [DOI] [PubMed] [Google Scholar]
- 16.Ware J,, Jr., Kosinski M, Keller SD. A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Med Care 1996; 34: 220–33. [DOI] [PubMed] [Google Scholar]
- 17.Morris NS, MacLean CD, Chew LD. et al. BMC Family Practice 2006; 7: 21.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hibbard JH, Mahoney ER, Stockard J. et al. Development and testing of a short form of the patient activation measure. Health Serv Res 2005; 40: 1918–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Benet-Martinez V, John OP. Los Cinco Grandes across cultures and ethnic groups: multitrait multimethod analyses of the Big Five in Spanish and English. J Personal Social Psychol 1998; 75: 729–50. [DOI] [PubMed] [Google Scholar]
- 20.John O, Donahue E, Kentle R. The big five inventory: versions 4a and 54, institute of personality and social research. University of California, Berkeley, CA: 1991. [Google Scholar]
- 21.Lumpkins C, Cupertino P, Young K, et al. Racial/ethnic variations in colorectal cancer screening self-efficacy, fatalism and risk perception in a safety-net clinic population: implications for tailored interventions. J Commun Med Health Educ 2013; 3: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Rawl SM, Skinner CS, Perkins SM. et al. Computer-delivered tailored intervention improves colon cancer screening knowledge and health beliefs of African-Americans. Health Educ Res 2012; 27: 868–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.VanEenwyk J, Campo JS, Ossiander EM. Socioeconomic and demographic disparities in treatment for carcinomas of the colon and rectum. Cancer 2002; 95: 39–46. [DOI] [PubMed] [Google Scholar]
- 24.Zimmerman RK, Tabbarah M, Trauth J, et al. Predictors of lower endoscopy use among patients at three inner-city neighborhood health centers. J Urban Health 2006; 83: 221–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Fleisher L, Kandadai V, Keenan E. et al. Build it, and will they come? unexpected findings from a study on a web-based intervention to improve colorectal cancer screening. J Health Commun 2012; 17: 41–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Fisher RJ. Social desirability bias and the validity of indirect questioning. J Consumer Res 1993; 303–15. [Google Scholar]
- 27.Dou Z, Song R, Yuan X. et al. Are click-through data adequate for learning web search rankings? Proceedings of the 17th ACM conference on Information and knowledge management: ACM. 2008; 73–82.
- 28.Fox SK, Mydland K, Dumais S. et al. Evaluating Implicit Measures to Improve Web Search. ACM Trans Inform Syst 2005; 23: 147–68. [Google Scholar]
- 29.Joachims T, Granka L, Pan B. et al. Accurately interpreting clickthrough data as implicit feedback. Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval: ACM; 2005. p. 154–61.
- 30.Ferrante JM, Ohman-Strickland P, Hahn KA. et al. Self-report versus medical records for assessing cancer-preventive services delivery. Cancer. Epidemiol Biomarkers Prev 2008; 17: 2987–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Stange KC, Zyzanski SJ, Smith TF. et al. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care 1998; 36: 851–67. [DOI] [PubMed] [Google Scholar]
- 32.Nutting PA, Rost K, Smith J. et al. Competing demands from physical problems: effect on initiating and completing depression care over 6 months. Arch Fam Med 2000; 9: 1059.. [DOI] [PubMed] [Google Scholar]
- 33.Nutting PA, Baier M, Werner JJ. et al. Competing demands in the office visit: what influences mammography recommendations? J Am Board Family Practice 2001; 14: 352–61 [PubMed] [Google Scholar]