Callers' Ability to Understand Advice Received from a Telephone Health-Line Service: Comparison of Self-Reported and Registered Data

Abstract

Objective

To validate users' perception of nurses' recommendations to look for another health resource among clients seeking teleadvice. To analyze the effects of different users' and call characteristics on the incorrectness of the self-report.

Data Sources/Study Setting

This study is a secondary analysis of data obtained from 4,696 randomly selected participants in a survey conducted in 1997 among users of Info-Santé CLSC, a no-charge telenursing health-line service (THLS) available all over the province of Québec.

Study Design/Data Collection

Self-reported advice from follow-up survey phone interviews, conducted within 48 to 120 hours after the participant's call, were compared to the data consigned by the nurse in the computerized call record. Covariables concerned characteristics of callers, context of the calls, and satisfaction about the nurses' intervention. Association between these variables and inaccurate reports was identified using multinomial logistic regression analyses.

Principal Findings

Advice to consult were recorded by the nurse in 42 percent of cases, whereas 39 percent of callers stated they had received one. Overall disagreement between the two sources is 27 percent (12 percent by false positive and 15 percent by false negative) and kappa is 0.45. Characteristics such as living alone (adjusted OR=2.5), calls relating to psychological problems (OR=2.8), perceived seriousness (OR=∼2.6), as well as others, were associated with inaccurate reports.

Conclusions

Telephone health-line providers should be aware that many callers appear to interpret advice to seek additional health care differently than intended. Our findings suggest the need for continuing quality control interventions to reduce miscommunication, insure better understanding of advice by callers, and contribute to more effective service.

The use of telephone technology to deliver 24-hour, out-of-hours or after discharge health care advice is growing in many countries. Such services usually offer advice and information on which health care provider to contact and the urgency with which to contact them, or on how the caller can care for the problem themselves. Up to now, the existing services seemed to go on the assumption that consultation skills used in the traditional face-to-face client–provider encounters are immediately transferable to the telephone (Crouch and Dale 1998), and that the caller will necessarily hear, understand, and agree with the advice offered by the telephone adviser (all of which affects compliance with the advice given) (Munro et al. 2001).

Some publications have addressed outcome issues in telephone advice-line services and performed follow-up of calls. However, only sparse and incomplete literature was found that documented or pointed out the miscommunication that may occur between provider and client via telephone, and the ability of individuals to understand the teleadvice they have received. For example, Evens et al. in 1985 evaluated physician-based after-hours calls and reported that there was a lack of agreement between the viewpoint of physician and caller on several aspects of the contact, such as the primary reason for calling and the necessity of the encounter. Nearly all of the patients interviewed said that they were able to understand the physician's instructions completely; unfortunately the authors had not validated this concern. Dale et al. (1997), in a study of compliance with teleadvice given by accident and emergency nurses suggested that there are some discrepancies between the advice patients recalled receiving, the advice that was recorded by the nurse as having been given, and what patients said they actually did subsequent to the call. Nevertheless, on the basis of the overall percentages provided by the authors it is not easy to make conclusions about the disagreement between the advice documented by the nurse and that reported by the callers; a cross-tabulation of values would have been more informative. More recently, Payne, Shipman, and Dale (2001) stressed that many patients were anxious about their ability to describe symptoms over the telephone, or understand and follow the teleadvice that they received from a general practitioner out-of-hours co-op. Their findings and conclusions are provisional because they are based on only 47 telephone consultation follow-ups. Finally, Munro et al. (2001), the researchers who were commissioned to undertake the evaluation of the U.K. nurse-led telephone helpline, have linked the data from 155 call transcripts with that from an equivalent number of postal survey questionnaires, and realized a qualitative analysis of reasons for lack of compliance. The authors have described some evidence of poor communication between nurse and caller, of ambiguity in the advice given by the nurse and of mishearing or misinterpretation of what she said. However, no quantitative appreciation of the differences between the patients' and nurses' perspective is available.

As many countries move to health systems in which it becomes more common for patient–provider encounters to occur via telephone, there is a need for investigation about such interactions to make sure that the telenursing health-line service (THLS) really enhances users' self-care abilities and reduces unnecessary clinic and emergency room visits. A larger survey published elsewhere was conducted to assess the perceived accessibility, quality, and outcomes of the Québec THLS. This Province-wide Evaluation of Info-Santé CLSC Units has collected data about nurse's recommendations to consult from the computerized client call-record and from a follow-up phone interview (Hagan, Morin, and Lépine 1999; Hagan, Morin, and Lépine 2000). The data, which are based on a large random sample, provided us with an opportunity to examine the telephone advice-line users' perception of whether or not a formal recommendation had been made to seek another consultation, and to compare users' perceptions to what the nurse documented. Also, circumstances of the call and users' characteristics associated with miscommunication are assessed.

Description of the Service

Since 1995, Info-Santé CLSC provides the population in all health regions with a no-charge 24-hour, seven days a week, telephone nursing response. During weekday working hours, the THLS is delivered through local call centers, and outside these hours (evenings, nights, weekends, and statutory holidays) calls are automatically redirected to a regional call center. At the time of the survey from which this secondary analysis is derived, approximately 7.3 million persons had access, of whom 75 percent were aware of the service and 28 percent had already used it at least once. All advice nurses were registered, well experienced, and had received telephone-consultation skills training, together with decision support software to assist process managing, clinical assessment, and advice giving (Ministère de la Santé et des Services sociaux [MSSS] 1999; Ordre des Infirmières et des Infirmiers du Québec 1998).

All nurses taking and processing calls are required to record the personal and clinical user's concern and the intervention directly into the client's call record. The intervention may consist of: (1) providing information about health, social, and community services; (2) giving advice about actions that can be performed at home to solve or stabilize the problem (self-care recommendations); and/or (3) referring callers to the most appropriate resources when needed (recommendations to consult). Nurses consign information about references to other resources in cases when they advise to consult immediately or within a specific time delay. However, nurses commonly suggest that the caller may consult the health care resource if the problem aggravates or does not improve, without giving any explicit level of urgency or time. This cannot be considered as a formal imperative recommendation and, consequently, is not recorded (Hagan, Morin, and Lépine 1999).

Methods

Data Collection and Variables

Survey contents, recruitment methods, and informed consent procedures were in accordance with the provincial Act Respecting Access to Documents Held by Public Bodies and the Protection of Personal Information. A more complete description of study methods may be found elsewhere (Hagan, Morin, and Lépine 1999; Hagan, Morin, and Lépine 2000).

Two sources of data were used. First, the following coded information was drawn from the computerized call records: relationship of callers to users (the users are those about whom the calls were made: callers themselves or someone else); user's age, sex, and health region; time of call; type of call center (i.e., local or regional); and nature of the problem and the recommendations. The second source is a computer-assisted telephone survey follow-up carried out by trained interviewers with people who had contacted the service within a period of 48–120 hours after their call. At least 10 attempts at different times of day were made to reach enrollees. Among the variables collected by the pretested close-ended questionnaire were the characteristics of callers (sex, age, education level, living and employment situations, family income, language), number of calls to Info-Santé CLSC in the 12 months prior to the survey, previous recourse to another resource, perceived urgency, anxiety, and seriousness of the problem at the time of the call, satisfaction with different aspects of the service, and perceived advantages of using THLS rather than another resource. To ascertain the self-reported information about the nurses' recommendation, callers were asked: “Did the nurse suggest that you consult another resource?” If the answer was “Yes,” the respondent was asked: “When or on what conditions did the nurse advise you to consult?” The following choice of response was proposed to them: “Consult immediately,”“Consult within a short time/a few hours,”“Consult only if the situation worsened or did not improve.” The first two recommendations were collapsed into a single category and compared to what was consigned on the call records.

Sampling Procedures

The population under study is made up of all calls received between October 6 and December 15, 1997, to Info-Santé CLSC units over the Québec health regions. Eligible callers were aged 14 years and older, were able to speak and understand French or English, had not called about STD or conjugal violence problems, and had verbally agreed to be recalled for a possible telephone survey.

A random sample of 59 call centers was selected according to the type of service they operated, whether regional or local, and to the health region they covered, from which a sample of calls was randomly drawn out in relation to the age of the callers and the proportional distribution of calls in each call-center. Thus, the current study was based on a total of 4,696 completed interviews with consenting callers. The response rate is estimated at 86.9 percent (the product of the rates of verbal consent and of response to the interview). The data was weighted using a probability calculation, taking into account both the sampling probability of the calls by age and the sampling probability of the call centers.

Statistical Methods

Callers' failing to correctly report that they were or were not advised to get care elsewhere were assessed by the numbers of false negative and false positive self-reports (SR) using the computerized records (CR) as the gold standard. Numbers of false negatives (FN) and positives (FP) were divided by the total number of individuals in the sample, in which false negative disagreement rate equal b/N, false positive disagreement rate equal c/N, and similarly total agreement rate equal (a+d)/N (see Table 1 for the signification of the letters). Cohen's kappa statistic, which adjusts for agreement caused by chance, was also calculated.

Table 1.

Comparison of Nurses' Recommendations from Survey Phone Interviews and Computerized Call Records, for All Study Callers

	Computerized Record (CR)
Self Report (SR)	Yes	No	Total
Yes	1,223 (a)*	543 (b)	1,766 (a+b)
No	659 (c)	2,094 (d)	2,753 (c+d)
Total	1,882 (a+c)	2,637 (b+d)	4,519 (N)

^*The values a, b, c, and d denote the observed frequencies for each possible combination of CR and SR, corresponding respectively to TP, FP, FN, and TN.

The associations between characteristics and likelihood of inaccurate SR were identified through multinomial logistic regression since the categorical dependent variable takes three possible values: (1) total agreement rate; (2) false positive disagreement rate; and (3) false negative disagreement rate, as defined above. A model was fit to include all independent variables with a p<0.25 in univariate analysis. Independent variables with the largest p-value were next removed one at a time and the model was recomputed at each step. This process was repeated until the statistical significance of the effect of any variable in the presence of others did not exceed 0.01. Model selection and testing (p-values) were based on the likelihood ratio test, and Wald-type confidence intervals (CI) were constructed for the odds ratios (OR). Change in the value of -2 log-likelihood if an effect is omitted from the model was computed for all covariables and compared to the appropriate chi-square distribution to determine the significance level of its global effect on both types of disagreement simultaneously. Two sets of coefficients were generated, one for the comparison of each of the two disagreement groups to the perfect agreement group as the reference category.

All statistical analyses were performed with the Statistical Package for Social Sciences (SPSS 2001) and Statistical Analysis System (SAS 1996) software using weighted data. The significance level was fixed to 0.01 in order to account for the underestimation of the variance arising from the cluster sampling design. This approximate strategy, judged conservative in our case, avoids declaring significant results by error too often. Therefore, only tests that are significant at this level are discussed and 99 percent CI were computed (Fleiss 1981).

Results

Study Participants

The users were mainly women (85 percent), of a mean age of 36 years, who usually spoke French at home (93 percent). Forty-six percent had completed at least 12 years of education, and 52 percent held a full-time or part-time job. A little over half the calls (54 percent) were processed on a centralized service base, 73 percent on weekdays. Fifty percent of calls occurred during the day, 39 percent in the evening, and 11 percent at night. The number of calls made by the same individual in the 12 months prior to the survey was high, with large variations (average, 5.8; standard deviation, 8.7; median, 3; mode, 1).

Fifty-three percent of callers contacted the THLS for someone else. On the whole, 31 percent concerned children under five years of age, and 48 percent concerned middle-aged adults. Most calls were related to physical health problems (94 percent), and the most often recommended professionals were physicians (86 percent). More than 80 percent of people had a highly favorable opinion on the clarity of language used to advise them and on the ease of understanding the advice provided.

Comparison of Self-reports and Computerized Records

Table 1 compares nurses' recommendations to consult another resource according to the computerized record with self-reported callers' perceptions. On the whole, such recommendations were recorded in 42 percent (99 percent CI: 40–44 percent) of cases, whereas 39 percent (99 percent CI: 37–41 percent) of callers stated they had received one. Even if the prevalence estimated from CR is oddly comparable to that from SR, disagreement was found in 27 percent (99 percent CI: 25–28 percent) of cases when both sources are compared for each call. Kappa coefficient is 0.45 (99 percent CI: 0.41–0.49), which represents a moderate agreement (Landis and Koch 1977).

This discrepancy involved a substantial rate of callers' perceiving that a recommendation to seek additional care had been given when no such documentation existed (FP: 12 percent; 99 percent CI: 11–13 percent). After adjustment, factors associated with falsely perceiving that a recommendation to seek care had been made included sex of caller, living arrangement, caller's utilization of the THLS in the 12 months prior to survey, time of call, nature of the problem, perceived degree of anxiety and seriousness of the problem, and perceived helpfulness of the nurses' intervention (Table 2). For example, people who called for psychosocial problems were 2.8 times more likely than those that called for physical problems to report that they were told to consult, while no recommendation to this effect had been given. However, since the odds ratio is less than one and significantly different from zero (the CI does not include number one), we know that callers being not very anxious compared to those not anxious at all (OR=0.58) are less likely to make this type of error. This discrepancy also involved a substantial rate of callers' perceiving that a recommendation to seek additional care had not been given despite the existence of such documentation (FN: 15 percent; 99 percent CI: 13–16 percent). Factors associated with falsely perceiving that a recommendation for seeking care had not been made included caller's age and nature of the problem. Notably, callers who have contacted the service for psychosocial problems were more likely to make an erroneous report in one direction or another (Table 2).

Table 2.

Factors Associated to the Disagreement between Nurses' Recommendations from Survey Phone Interviews and Computerized Call Records, by Selected Characteristics

	Likelihood of Inaccurate Report
Variable	False Positive (OR)†	99 % CI*	False Negative (OR)‡	99 % CI*
Callers' sex
Males (n=679)	1.87	1.35–2.59	1.03	0.74–1.44
Females (n=3,840)	ref.	—	ref.	—
Callers' age group (years)
14 to 17 (n=59)	1.74	0.58–5.25	1.58	0.54–4.65
18 to 24 (n=749)	0.68	0.44–1.07	1.08	0.73–1.60
25 to 34 (n=1,850)	1.00	0.72–1.40	1.07	0.78–1.47
35 to 44 (n=1,023)	ref.	—	ref.	—
45 to 54 (n=301)	0.86	0.49–1.51	1.01	0.60–1.71
55 to 64 (n=241)	1.20	0.65–2.23	2.12	1.26–3.40
65 and older (n=281)	0.90	0.49–1.68	1.28	0.74–2.24
Living arrangement
Alone (n=415)	2.46	1.58–3.81	1.42	0.91–2.20
Without partner, with children (n=443)	1.33	0.87–2.04	1.36	0.93–1.98
With partner, without children (n=726)	1.51	1.01–2.27	1.22	0.85–1.75
With partner, with children (n=2,620)	ref.	—	ref.	—
Other (n=305)	1.07	0.59–1.94	0.92	0.54–1.57
Time of call handling
08:01 A.M. to 05:00 P.M. (n=2,239)	ref.	—	ref.	—
05:01 P.M. to midnight (n=1,789)	1.16	0.88–1.53	0.85	0.66–1.09
12:01 A.M. to 08:00 A.M. (n=491)	1.46	0.99–2.17	1.37	0.95–1.96
Number of calls in 12 months prior to interview
1 to 2 (n=1,746)	ref.	—	ref.	—
3 to 4 (n=1,097)	1.30	0.92–1.83	1.17	0.87–1.58
5 to 6 (n=627)	1.43	0.94–2.17	0.87	0.58–1.29
7 or more (n=1,017)	1.82	1.27–2.61	1.12	0.80–1.56
Callers' state at the time of call
Perceived anxiety
Very anxious (n=791)	1.34	0.79–2.27	0.93	0.58–1.49
Fairly anxious (n=1,772)	1.09	0.69–1.74	0.93	0.63–1.39
Not very anxious (n=1,328)	0.58	0.36–0.95	0.90	0.62–1.32
Not anxious at all (n=605)	ref.	—	ref.	—
Perceived seriousness
Very serious (n=271)	2.47	1.08–5.64	0.66	0.34–1.27
Fairly serious (n=2,066)	2.68	1.36–5.03	0.75	0.49–1.14
Not very serious (n=1,707)	2.11	1.08–4.11	0.73	0.49–1.09
Not serious at all (n=420)	ref.	—	ref.	—
Nature of the problem
Physical health (n=4,328)	ref.	—	ref.	—
Psychosocial problem (n=174)	2.77	1.58–4.85	1.93	1.14–3.27
Perceptions of service provided
Helpfulness in choosing best solutions
Very helpful (n=3,057)	ref.	—	ref.	—
Fairly helpful (n=1,199)	1.43	1.08–1.89	0.99	0.76–1.29
Not very helpful and not at all (n=230)	1.50	0.87–2.58	1.18	0.71–1.98

^*Confidence interval.

OR from logistic regression, controlling for all significant variables. Likelihood of inaccurate report using the agreement group as reference:

^†CR no, SR yes (false positive);

^‡CR yes, SR no (false negative).

However, variables such as users' age, type of caller, callers' employment and income situations, type of call center, perceived degree of urgency, and usefulness of the intervention in finding solutions were identified as significant in the univariate analyses but were not after adjustment for other variables. On the other hand, variables such as education, language, period of the week, prior consultation, recommended self-care actions, suitability of solutions offered, acquisition of knowledge, and transport savings were neither significant in the univariate analyses nor in the multivariate analyses.

Disagreement, Type, and Condition of Recourse Recommended

The proportion of disagreement has been examined with regard to the condition attached by the nurse to the recommendation, as perceived by the users. Among the 659 false negative reports, 375 callers (57 percent) said the nurse advised them to consult another resource only if the current situation worsened or did not improve (a conditional type of recommendation that is not recorded), whereas the other 284 (43 percent) claimed that they had not received any recommendation to consult. On the other hand, among the 543 false positive reports, 151 callers (28 percent) said they were told to consult another health care professional resource immediately, and the other 392 (72 percent) said they were told to consult within a few hours.

Discussion

Even if the great majority of callers felt that the language used to advise them was clear and the advice they received was easy to understand, our findings show that there is a significant discrepancy between telephone health-line callers' perceptions of having been advised to seek additional health care and the health-line nurses' documentation of whether such advice was given. Thus, we confirm what is expressed by some other authors about problems of miscommunication in health care advice delivery over the telephone. The information we provide is based on a more explicit quantitative design meant to shed light on discordance between what was recorded by the nurse and what was reported by the caller, and on a much larger and more representative sample than has been previously available. So, as far as we know, we present for the first time the type and the magnitude of the discrepancies as well as the particular callers' characteristics and the context of the calls in which they occurred. This has important implications for any intervention that might be implemented.

With such a short time interval (48 to 120 hours) after the event, discrepancies are more probably the result of misinterpretation or reinterpretation of the information than of memory bias. In fact, the discrepant reporting may be explained in a few other ways. First, some users may have misinterpreted the condition specified by the nurse as she gave her recommendation: either she in fact gave a conditional recommendation (consult if things do not improve or get worse) and they understood and reported this as an imperative recommendation to consult within a short time/a few hours, or, she formally advised them to consult and they understood and reported this as a conditional recommendation. A more restrictive definition of “falses” taking into account the perceived timescale condition attached to the recommendation shows that the more disturbing FN are callers who believed they received no recommendation at all and the more disturbing FP are those who said that they were explicitly advised to consult immediately. A second possible explanation for the lack of agreement is that some callers were unsatisfied with the nurse's advice, disagree with it, and chose not to comply. They may then, unconsciously or not, have wanted to justify their subsequent behavior and thus reported that it was what the nurse advised them to do. A third possibility is in the case of people who were advised to consult as well as take some self-care actions: if these self-care actions were sufficient to solve the problem (which was frequently the case according to the interviews), users may then have reported the recommendation to consult as a conditional rather then as a formal one.

The implication of these observed discrepancies may be more worrisome in certain situations than in others. This could be the case, for example, when the problem is identified by the nurse as requiring immediate professional attention, and when delay in receiving appropriate care could lead to a deteriorated situation. Another example would be when a perceived advice to consult resulted in unnecessary worry for the caller as well as useless expense and transportation in order to reach a doctor when not required. Obviously, it seems pertinent to recommend nurses to make sure that the information provided to the caller is focused on the essential points to avoid ambiguity; to be very explicit concerning the advice, and particularly about the urgency and timescale condition of recourse they recommend; and to verify that the callers' understanding about the information/advice is adequate at the end of the phone intervention. The significant multivariate factors associated with the lack of accuracy of report provide useful leads about special attention to be given to particular groups of callers.

A certain number of practical inferences can be drawn from this. First, the factors that affect self-reporting suggest that the discrepancies are from multifarious sources. Sociodemographic characteristics of callers (sex, age, living arrangement) can explain their propensity to consult (or not) other resources, independently of the advice they received. Also, it is a question of the clients' previous use of the THLS, the circumstances and nature of the problem (time of call, calls that concerned a psychological problem), the emotional situation of the callers when they contact the THLS (anxiety, seriousness), as well as satisfaction about the nurses' intervention (helpfulness). Worry sometimes hampers the caller's ability to offer complete information about the situation and to understand the health information/advice (Wilkins 1993). Also, callers who are more worried would have more propensity to go out for help, and would tend to misinterpret the nurse's advice in the direction of their intended behavior. Nevertheless, the presence of a little bit of anxiety appears instead as a predictive factor to correctly report the events. In fact, a degree of anxiety may have the most potent effect on causing a client to initiate a contact with the THLS, and can motivate the caller to try to resolve their problem, and therefore, to be more inclined to pay attention or ask questions to better understand the advice given (Evens et al. 1985).

Second, most of the factors can be identified in the course of the intervention, which means that the nurse might be able to assist callers to understand the health information and recommended actions. Third, as could be expected, certain variables were associated with one type of disagreement but not with the other. Fourth, the nurses need to be aware that the effects of the factors are additive. In other words, any individual who accumulates more and more factors is more and more likely to inaccurately understand the advice received. Fifth, it is worrisome that the caller groups who are among the most frequent users of the THLS over the last 12 months are also those most subject to misreport the advice received.

A methodological issue is that any validation of measure implies that an independent gold standard is available for comparison. In the current study, there was not any direct assessment about the accuracy of the nurses' data entry about the encounter. We presumed that the nurse reports were a more accurate source than the users' perception and considered the computerized record as the best single approach. Several factors support this: all of the nurses worked with the same software, they received an initial training, and were well experienced with the computerized tools; data collection on the call records has not developed as a procedure specific to the needs of the study, but represent a routine practice for the nurses; data were entered at time of event, and patients were likely more stressed than nurses. Consequently, it is unlikely that anything but small data recording errors would have occurred. Lacking an objective record of the content of calls (such as tape recordings) and clinical outcome data, it is impossible to know beyond doubt the extent to which the observed differences reflect communication skills, inaccuracies of users' hearing, understanding and reporting of advice received, or poor record keeping by the nurse (Dale et al. 1997).

In conclusion, the benefits expected from a THLS on individuals' and families' empowering attitudes and behaviors are largely dependent on the callers' ability to clearly understand the advice they receive. Nurses involved need to be aware that their advice, as clinically sound as it may be, is not always interpreted accurately by a significant proportion of callers, and especially so among some subgroups. Further studies, notably using call tapes, are needed to investigate these aspects and others, such as conditional instruction to seek care, self-care advice, and time spent with clients. In this last concern, Mayo (1998) identified connecting relationships and time spent with patients on the phone as important considerations for nurse advice. It would be interesting to know if call centers providing longer duration calls are associated with reduced miscommunication, and vice versa. Nevertheless, our chief finding suggests a need for improving the effectiveness of the telephone intervention program. We should mention that the results of the current study do not invalidate previously published estimations of avoided health care utilization due to Info-Santé CLSC since these estimations were not based on perceived recommendations but on users' intentions and declared behavior (Ministère de la Santé et des Services sociaux 1999).