The introduction of integrated out‐of‐hours arrangements in England: a discrete choice experiment of public preferences for alternative models of care

Abstract

Objective  To establish which generic attributes of general practice out‐of‐hours health services are important to the public.

Methods  A discrete choice experiment postal survey conducted in three English general practitioner (GP) co‐operatives. A total of 871 individuals aged 20–70 years registered with a GP. Outcomes were preferences for, and trade‐offs between: time to making initial contact, time waiting for advice/treatment, informed of expected waiting time, type of contact, professional providing advice, chance contact relieves anxiety, and utility estimates for valuing current models of care.

Results  Response rate was 37%. Respondents valued out‐of‐hours contact for services for reducing anxiety but this was not the only attribute of importance. They had preferences for the way in which services were organized and valued information about expected waiting time, supporting findings from elsewhere. Participants were most willing to make trade‐offs between waiting time and professional person. Of the predicted utility for three models of care utility was higher for fully integrated call management.

Conclusions  Greater utility might be achieved if existing services are re‐configured more in line with the government's fully integrated call management model. Because the attributes were described in generic terms, the findings can be applied more generally to the plethora of models that exist (and many that might exist in the future). The approach used is important for achieving greater public involvement in how health services develop. Few experiments have elicited public preferences for health services in the UK to date. This study showed valid preferences were expressed but there were problems obtaining representative views from the public.

Introduction

United Kingdom government policy aims to get the public more involved in decision making about the way health services are to develop. ¹ However, eliciting this information in a form that decision makers can use presents a challenge. ² It can be difficult both to find representative samples and to elicit valid information from individuals who may not have relevant experience as patients. ³ Thus, not surprisingly, we understand very little about what members of the public actually want from their health services and, in turn, the social desirability (value) of health services.

The government is committed to raising standards for patients accessing out‐of‐hours primary care. ⁴ A number of new gateways into out‐of‐hours NHS services have emerged in recent times, ¹ making the system more complex. Current policy is for the implementation of the fully ‘integrated’ call management model. ⁵ This new configuration of out‐of‐hours care aims to provide a single point of access via NHS Direct, thus ‘getting the patient to the right service at the right time’. ⁴ Early evidence shows integration has not been uniformly adopted and that variants of this basic model exist. ⁶ However, from the individual's perspective there are a number of generic characteristics (attributes) of alternative out‐of‐hours models that may be of value. ⁷ , ⁸ , ⁹

To make an informed choice about these options, the individual needs to be able to weigh up the differences in these characteristics. For example, which is more preferable; a telephone consultation which can be accessed quickly, but after which the caller remains quite anxious, or a face‐to‐face contact which takes longer to access but is more reassuring? The discrete choice experiment technique is an economic method of assessing stated preferences. It is an attribute‐based measure of preference intensity (utility) that has been validly applied to health care. ¹⁰ Validity is judged with respect to theoretical validity, face validity and consistency with axioms of economic consumer theory (for further details of evidence see Ryan and Gerard ¹⁰ ).

A discrete choice experiment quantifies preferences by analysing responses individuals provide in surveys about how they would behave in hypothetical (yet realistic) situations. Individuals choose their preferred option from a series of choices in which each alternative is described by a unique combination of attribute levels. A probabilistic discrete choice model is used to analyse the data. The approach has desirable properties. Being attribute‐based means it is especially useful in contexts where benefits of health services are multi‐dimensional and it is important to have benefit values which are decomposed into a set of attributes. This includes contexts where benefits are believed to be not solely about improvements in health but where other aspects of care and service delivery may also be relevant. ¹¹ The data from a discrete choice experiment provide policy makers with detailed information about preferences for multiple alternatives (either currently available or that might exist).

Furthermore, the approach is based on a realistic context. It explicitly incorporates the notion of sacrifice by recognizing that any decision must involve choice and all choices involve sacrifice of something of value. As such one important way it measures intensity of preferences is by reporting the marginal rates of substitution between attributes (i.e. the rate at which individuals are willing to give up levels of one attribute for improvements in other attributes) and this can be used to inform priority setting. ¹⁰

We used a discrete choice experiment to establish which generic attributes of general practice out‐of‐hours health services were important to members of the public. Few experiments have been used to elicit public preferences for health services in the UK. ¹⁰ Two of them have been applied to out‐of‐hours health services and a third study examined patient preferences for out‐of‐hours health services. ⁷ , ⁸ , ⁹ The first found the single most important attribute was ‘whether the doctor seemed to listen’; to the extent that some respondents would not ‘trade’ this attribute to obtain more of another. ⁷ The authors concluded that improvements in doctor–patient communication might be most important when deciding how best to upgrade services. The second study also confirmed the importance of the attribute ‘doctor's manner’ in the care patients wanted to receive but here subjects were more prepared to trade this for more of other attributes such as reductions in waiting times and where the patient was seen. ⁸ One unexpected finding was subjects’ relative dislike for telephone consultations. This is pertinent given the current trend to extend reliance on telephone‐led out‐of‐hours services. As stated earlier, the third study concerned patients rather than community preferences but provided important information about attributes of out‐of‐hours health services. It showed ‘being consulted by a doctor’ was the most important attribute. Respondents were prepared to wait an extra 2 h 20 min to be consulted by a doctor. This was followed by ‘being consulted by a nurse’, than ‘being kept informed about waiting time’ and ‘quality of the consultation’. ⁹ The results had been used by policy makers to directly inform the development of a local service framework for out‐of‐hours care.

It should be noted that the first two of these studies were undertaken prior to the introduction of new initiatives integral to current models of care (particularly NHS Direct and NHS Walk‐in Centres). As such neither study is now likely to represent the full range of choices relevant to today's more complex out‐of‐hours environment.

Although out‐of‐hours contact is most likely to be made in the belief that it will improve health‐related quality of life or save or extend life all the previous studies presented choices as trade‐offs between health service attributes only. In part this may be explained by the fact that out‐of‐hours contacts reflect such a wide range of care it is challenging to find a suitable health or ‘health‐related’ variable which adequately covers the multiplicity of conditions or reasons for contact. Thus no attempt was made to trade between health and non‐health characteristics. Our study is novel in the sense that we used the notion of relief of anxiety to reflect the health‐related benefit obtained from initial out‐of‐hours contact and then tested whether this was traded for improvements in other aspects of service delivery by members of public. By focusing in this study on initial out‐of‐hours contact the full episode of care and resultant recovery/improved health could not be accounted for. Such a perspective made it difficult to find a suitable ‘health‐related’ variable which would adequately cover the multiplicity of conditions and reasons for an initial contact. We selected relief of anxiety on the basis that worry motivates contact and alleviating anxiety is a major source of benefit. ¹²

Methods

Pilot study

The content of the discrete choice questionnaire was based on six key generic attributes from policy documents, literature ⁷ , ⁸ , ⁹ , ¹³ , ¹⁴ , ¹⁵ and discussion with general practitioners (GPs). Four related to aspects of service delivery (time to making initial contact, time waiting for advice or treatment, type of contact, and profession of the person providing initial advice), one to information (whether informed of expected waiting time) and another related to health outcome (chance out‐of‐hours contact relives anxiety). The questionnaire was piloted to learn more about: the realism of the levels selected to represent the ‘health‐related’ attribute; how to administer the survey in busy general practices to members of the public; obtain an estimate of response rate to inform main survey sample size; and check completeness of data.

Eighty individuals registered in two general practices aged between 20 and 70 years were randomly selected. The invitation letter was written on each practice's letter‐headed paper, signed by the GPs and enclosed with the postal questionnaire. Data Protection guidelines meant that personal details of registered patients could not be released for administration of the survey by the research team off site. It was necessary to explore the best way the team could help the busy general practices administer the survey.

A response rate of 28% was obtained using one reminder letter. We learnt that choice questions could be completed satisfactorily and that the two levels assigned to the attribute ‘chance anxiety is relieved by the out‐of‐hours contact’ were considered realistic. We also learnt that busy GP practices were unable to administer the survey without heavy reliance on help from a researcher and that practices were unwilling for further reminders to be sent out. On the basis of this experience we decided to proceed with a postal format for the main survey but doubled the invitation rate to ensure there would be sufficient responses. By collecting socio‐demographic information we planned to assess the nature of any response bias.

Main study

The study's attributes and levels are outlined in Table 1. Altogether this created 256 (2⁴ × 4²) combinations of attributes to describe possible out‐of‐hours models. Clearly this number was too great to evaluate and experimental design theory was used to sample from the total set of possible pairs to reduce the number of options presented to 16 such that ‘main effects’ could be estimated independently of one another [i.e. no multi colinearity between attribute levels ¹⁶ , ¹⁷ (further details on request from K.G.)]. Two versions of the questionnaire were made so that an individual was only asked to complete eight choice pairs but the experimental design could be accommodated. Each choice was between two options labelled as Option A and Option B (for example see Table 2). The respondent was asked to assume that they faced an urgent, but not life threatening, condition requiring medical advice or treatment during a time that their GP surgery was closed. This followed the experience of a previous study. ⁹ Theoretical validity of responses was checked by considering the relationships between the choice and particular attributes. Consistency of responses was checked using a specially constructed choice containing two options, one of which was unequivocally superior to the other. Individuals were randomly allocated to one of the two questionnaire versions.

Table 1.

Attributes and levels

Attribute	Attribute full description	Level 1	Level 2	Level 3	Level 4
Time to making initial contact	Time it takes you (others) to make initial contact with the OOH service (e.g. your call answered or details taken down)	1 min	5 min	10 min	15 min
Time waiting for advice or treatment	Time you (others) expect to have to wait between contacting the service and before you see or speak to a doctor or specialist nurse	5 min	20 min	1 h	5 h
Informed of expected waiting time (code for each level)	Whether you (others) are informed of the time you (others) can expect to wait to see or speak to the doctor or nurse	No, not informed (No = 0)	Yes, informed (Yes = 0)
Type of contact (code for each level)	Type of contact you (others) most likely to have, either see who advises or treats you in person or speak to them by telephone	By telephone (No = 0)	In person (Yes = 0)
Professional person (code for each level)	Professional person providing initial advice or treatment about your problem	Specially trained nurse (No = 0)	Doctor (Yes = 0)
Chance OOH contact relieves anxiety	Likelihood anxiety about medical problem is relieved following OOH contact	50%	90%

OOH, out‐of‐hours.

Table 2.

Example of a choice

Attribute of out‐of‐hour service	Option A	Option B
Time to making initial contact is	1 min	5 min
Time waiting for advice or treatment is	20 min	5 min
Informed of expected waiting time	No	Yes
Type of contact is	By telephone	In person
Professional is	Doctor	Specially trained nurse
Chance contact relieves anxiety is	90%	50%
If I had to choose, my preference is (place a ‘×’ in one box)	□ Option A	□ Option B

Study sample

We were interested in individuals’ responses as members of the general public. Our study sample was obtained from the practice lists for seven general practices on the basis that over 90% of the population is registered with a general practice. The seven practices belonged to three general practice co‐operatives across England that were selected in accordance with the aims of the wider study. ⁶ Individuals were randomly selected from current practice lists after excluding patients who were: temporary residents; <20 years of age (to corresponds with Census data); older than 70 years (as advised by some GPs who felt these patients could be made anxious/worried by being asked to participate in such a survey); and following the option given to GPs participating in the study to judge the suitability of patients invited (no GP took up this option). As some of our sample would have recent contact with out‐of‐hours services, this was recorded in the questionnaire but the sample frame did not rely on patients’ having their own recent experiences as patients.

A sample of 600 was required based on: previous research which recommends that samples are between 30 and 100 for each proposed subgroup ¹⁸ (we had three subgroups – co‐operative model, questionnaire version and socio‐demographic variables – and opted for 100 per subgroup), and regression analysis requiring a minimum sample size greater than the number of independent variables. As we anticipated 25% response rate we mailed 2400 questionnaires during March and May 2004.

Ethics approval

Approval for the study was given by the Trent Multi‐Centre Research Ethics Committee.

Consent, data protection and administration

Consent was implied by the return of the questionnaire (the invitation included an information sheet and left the subject sufficient time to decide whether or not to participate). We did not require personal medical information of any kind. Data Protection rules meant that the research team supported general practice staff on site with the administration of the survey using pseudo‐anonymized lists of study subjects to send a reminder letter 3 weeks later.

Analysis

Model estimation

The data were modelled using a ‘random utility maximization’ framework ¹⁵ , ¹⁶ and stata 8.0 software. ¹⁹ Researchers predicted the probability respondents chosen from a choice set but with some error. A logit regression main effects linear additive model was thus fitted ¹⁶ and was specified as:

where V = α ₀ + β ₁ X ₁ + β ₂ X ₂ + β ₃ X ₃ +β ₄ X ₄ + β ₅ X ₅ + β ₆ X ₆, and where U is ‘latent’ utility, V is the deterministic part of utility that researchers can observe and ɛ reflects unobservable factors known only to respondents. V is a linear function of the attribute levels characterizing out‐of‐hours contact (X ₁ to X ₆) where the coefficients β ₁ to β ₆ are estimated in the model and α ₀ is a constant term which picks up any unobservable influences affecting individual's choices. A condition of this model is that the error term is assumed to be Gumbel distributed. ¹⁵

Multinomial logit regression is appropriate because the dependent variable is binary (taking the value 1 if the option is chosen and 0 if it is not). The design of the experiment meant that testing interactions between attributes was not allowed but interactions between attributes and sample characteristics were investigated. Independence of irrelevant alternatives was assumed in the model specification.

Marginal rates of substitution were calculated by taking the ratio of the coefficient estimate for an attribute and the negative coefficient estimate for time waiting for advice or treatment. ¹⁷

Predicting utility

We used the estimated model to predict utility for three out‐of‐hours models currently provided in England, these are examples of current practice: ‘fully integrated’ call management systems, ‘partially integrated’ and ‘not integrated’ ones. The fully integrated model embodies government standards for integrated call management. ⁵ This means that patients calling their GP out‐of‐hours would be automatically diverted to NHS Direct for initial assessment over the telephone, making access to an initial clinical assessment faster and simpler for patients who ordinarily on contacting their surgery would hear an answer phone message requiring them to make a second call to an out‐of‐hours service. NHS Direct would transfer calls from patients with immediate, life‐threatening problems to the 999 ambulance service, but the majority of patients would have their needs met through telephone information or advice from an NHS Direct nurse adviser.

The ‘partially integrated’ model reflects a different GP co‐operatives and NHS Direct partnership. In our example of this, there was integrated call management provided only to a proportion of out‐of‐hours’ hours. Non‐integrated models, however, tend to be models of GP co‐operatives working independently of NHS Direct. Indeed some may employ their own practice nurses to triage calls. The three co‐operatives in our study represented examples of these three broad types of arrangements. We collected data on how they typically operated with respect to the attributes of interest in the study making it possible to predict utilities for these particular combinations of attribute levels. This was done by summing the product of each coefficient estimate for each attribute with the relevant attribute level observed for that co‐operative. Table 6 describes key differences and similarities across these models as they pertain to the discrete choice experiment.

Table 6.

Predicted utility for models of general practice out‐of‐hours health services

Care model	Attribute level (X i)
	Fully integrated	Partially integrated	Non‐integrated
Attribute
Time to making initial contact (min)	1.5	5	1.5
Time waiting for advice or treatment (min)	30	45	40
Informed of expected waiting time	Yes (1)	No (0)	No (0)
% type of contact, in person	35	56	40
% professional person, GP	65	84	80
% chance contact relieves anxiety	50	50	50
Predicted utility	0.694	0.057	0.102

Note: Utility is estimated by equation U = α ₀ + β ₁ X ₁ + β ₂ X ₂ + β ₃ X ₃ + β ₄ X ₄ + β ₅ X ₅ + β ₆ X ₆ using results in Table 3 and levels of X _i given above.

Results

Of the 2408 questionnaires sent 66 were returned undelivered. Of those remaining 871 (37%) returned completed questionnaires after a single reminder. This is in keeping with other postal community‐based discrete choice experiment surveys of health care. For example, a response rate of 25% was obtained in a study by van der Pol and Cairns. ²⁰ Other out‐of‐hours postal studies achieved response rates of 68% (due to a greater number of reminders) ⁷ and 65% (due to a two‐staged sampling frame where agreement to participate in the discrete choice experiment was conditional on a prior survey). ⁸ , Table 3 shows the characteristics of the participants and compares them with 2001 Census data. Compared with the English population our sample comprised a higher proportion of females, people in their mid and late years and people who were more highly educated, but a lower proportion employed than would be expected.

Table 3.

Characteristics of community sample (n = 871)

Characteristic	Sample number (%)	% England average, Census 2001 21
Gender (male)	301 (35)*	49
Age distribution (years)
20–39	143 (16)†	42
40–64	459 (52)†	44
65–70	269 (31)†	12
Ethnicity (non‐white)	59 (7)	9
Education (degree or higher)	359 (41)*	20
Economic activity employed	454 (52)*	61

*Sample and England population t‐test statistic statistically significant at the 1% level.

^†Sample and England population χ ² statistic statistically significant at the 1% level.

Discrete choice experiment results

Only a small number of responses (6%) were shown to be inconsistent (they were included in the analysis) and the model was theoretically valid. Table 4 shows the best fitting results of the choice experiment for all responses. The coefficients for the attributes were all statistically significantly different from zero and correctly signed. Negative values for time to making initial contact and time waiting for advice or treatment indicate that the longer the time the less likely the respondent will prefer that option and conversely for those attributes that are positively signed. Thus, given the coding used, respondents would prefer out‐of‐hours contact to have: a higher chance of relieving anxiety, contact in person and with a doctor, and shorter contact and waiting time. Respondents also preferred to be kept informed of expected waiting time. The model shows that being seen by a doctor (β = 0.443) is the most important attribute. However, all else equal and assuming a linear utility function, a 65‐min reduction in time spent waiting to be seen or spoken to would be more beneficial [β = −0.007, total effect =65 × (−0.007)].

Table 4.

Results of logit regression analysis

Variable	Coefficient, β I (95% CI)	SE	P value
Time to making initial contact (β 1)	−0.022 (−0.030 to −0.014)	0.004	<0.001
Time waiting for advice or treatment (β 2)	−0.007 (−0.008 to −0.006)	0.0002	<0.001
Informed of expected waiting time (β 3)	0.234 (0.175 to 0.293)	0.030	<0.001
Type of contact (β 4)	0.235 (0.176 to 0.293)	0.029	<0.001
Professional person (β 5)	0.443 (0.384 to 0.502)	0.030	<0.001
Chance contact relieves anxiety (β 6)	0.010 (0.009 to 0.012)	0.0007	<0.001
Constant (α 0)	−0.071 (0.012 to 0.130)	0.030	0.018
No. of observations	6643		<0.001*
LR χ 2 statistic	2349
% correct predictions	74.6%

*Chi‐square test. CI, confidence interval; SE, standard error; LR, loglikelihood ratio.

There were no statistically significant interactions between attributes and sample characteristics, including whether the individual had previous use of out‐of‐hours services (details from K.G.).

Using discrete choice experiment results

Table 5 shows the marginal rates of substitution (trade‐off) between amount of time waiting for advice or treatment and the other attributes. In order of relative importance these rates show respondents were willing to wait 58 min longer for advice or treatment to be seen by a doctor than a nurse; 30 min longer to be kept informed about expected waiting time and to be seen in person rather than spoken to by telephone. Respondents were only willing to wait a few minutes longer to foreshorten initial contact time by a minute and relieve anxiety by a percentage point.

Table 5.

Marginal rates of substitution between time waiting for advice/treatment and other attributes (β _i/β ₂)

Attribute of out‐of‐hour service	Time willing to wait in minutes (95% CI)	Single‐level improvement
Time to making initial contact	2.9 (1.8–3.8)	Per minute less time to making initial contact
Informed of expected waiting time	30.8 (21.7–40.7)	From absent to present
Type of contact	30.9 (22.3–40.9)	From ‘spoken to by telephone’ to ‘seen in person’
Professional person	58.1 (47.1–69.4)	From ‘specially trained nurse’ to ‘doctor’
Chance contact relieves anxiety	1.5 (1.2–1.7)	Per percentage reduction in anxiety

Note: all values were obtained using bootstrapping simulation of 1000 repetitions and rounded up.

Table 6 shows for the three care models considered the attribute levels imputed into the logit regression and predicted utility. All else being equal the fully integrated model yielded highest utility (0.694), next was the non‐integrated model (0.102) and next the partially integrated model (0.057). This suggests that utility gains of 0.637 and 0.592 could be obtained if an out‐of‐hours service was re‐configured from a partial to fully integrated model or from no integration to fully integrated. Interestingly, as all expected utility scores are positive, these models appear more desirable than waiting for the GP surgery to re‐open.

Discussion

Of those who responded to our survey we showed that out‐of‐hours contact for general practice services was valued for reducing anxiety but this was not the only attribute of importance. Respondents also had preferences for the way in which services were organized and valued information about expected waiting time, supporting findings from elsewhere. ⁷ , ⁸ , ⁹ , ²² Indeed the more traditional attributes of out‐of‐hours services (being seen by a doctor and in person) were ranked the first and third most important (according to size of coefficient estimate). This may appear a somewhat negative finding for current policy given both its direction and a commitment to developing services in line with public values. However, using the results to estimate the value of what could be achieved with different combinations of attribute levels we showed that our less integrated models had lower utility estimates and thus greater potential to increase utility by moving towards models of greater integration.

It has been shown for the first time in a discrete choice experiment of out‐of‐hours services that anxiety reduction is a plausible intermediate health characteristic of out‐of‐hours initial contacts and that individuals are prepared to trade‐off some of this to gain more in terms of non‐health attributes. One interpretation is that respondents view anxiety and anxiety reduction as key elements in community choices for generic aspects of out‐of‐hours services. But this attribute needs further development on a number of fronts. For example, we know that individuals’ understanding of concepts such as risk and probability are problematic but, as yet, we are unsure of the most valid way of presenting such attributes. ¹⁰ In this particular context further research would also be beneficial in terms of exploring whether interactions between this attribute and others were important, for example with being seen in person and seen by a doctor.

Even if it were not feasible or desirable to implement the kind of change required to bring about greater access to be seen by the doctor or in person, the study made clear that the value of being kept informed of expected waiting was an important generic feature of out‐of‐hours services. This is an area of service delivery that is particularly amenable to change whatever the constraints are locally.

It is important for policy makers not to overlook the value of the information gleaned from marginal rates of substitution. From these it is possible to look at the way current services are provided and to understand the level of compensation required to improve certain aspects of service in line with public preferences. As discussed, one area might be to introduce a mechanism for keeping patients informed of expected waiting time. In compensation, however, average waiting time could rise by up to 30 min while maintaining the same level of benefit, any more and utility deteriorates.

A limitation of this study was its lack of representativeness. That was despite the fact that the sample frame was chosen because it could be assumed that over 90% of individuals would be registered with a GP and that piloting indicated respondents could undertake the exercise asked of them. We felt this was largely outside our control due to Data Protection guidelines. As patients’ names and addresses could not be released it meant we were reliant on the survey being administered through busy general practices. Whilst we did all that was possible within the ethical, time and cost constraints of the study, we were, nonetheless, restricted. We are aware that others have managed better response rates by different sampling frames and more intensive follow‐up regimes but nevertheless it is important that policy makers be aware of some of the problems of obtaining views that are genuinely representative.

Since carrying out the study, the policy landscape has changed. Primary Care Trusts (PCTs) across England and Wales are now responsible for commissioning out‐of‐hours health services and setting quality standards for these services. The results of this study could be used to inform a debate between PCTs and local providers about alternative models (which may or may not currently exist) as well as the identification of quality of care standards for service contracts.

Conflicts of interest

None.