21st century medical education: critical decision-making guidance through smartphone/tablet applications—the Lothian pilot

Abstract

Introduction

In starting a new clinical placement, doctors in training must become aware of and apply standard operating procedures, as well as learn guidelines, simultaneously adjusting to new patient presentations, environments and personnel. This transition is thought to correlate with increased risk to patient safety, notably during the annual UK changeover. Mobile technologies are increasingly commonplace throughout the National Health Service. Clinicians at all levels are employing medical technology and applications (apps) with minimal local guidance. We set out to test the feasibility and utility of offering medical apps to out-of-hours (OOH) practitioners as an aid to clinical decision-making at point of patient contact. The theorised benefits were threefold: clinical education—real time support for clinical decision-making as one component of deliberate practice to build expert performance; decreased administrative burden–updating and accessing current guidelines; and service development—readily accessible feedback from users.

Method

We provided 32 devices in our emergency departments and OOH environments. The devices were preloaded with apps approved by our medical education department and clinical service leads to be used in support of care delivery.

Results

We surveyed 123 clinical staff prior to the pilot discovering that 65% had used mobile apps to aid their decision-making. During our project, we saw the number of clinical users expand with our data series, suggesting the apps most useful to care delivery for this group of service providers.

Future developments

There was huge enthusiasm for the project and we hope to maintain a clinician-led environment.

Introduction

Internationally, there is frequent turnover of junior doctors in acute environments. While rotation offers training benefits to doctors, the process creates inherent obstacles. In August 2014, the UK Foundation programme office reported that there were 7259 F1 places nationwide.¹ Every 4 months, this group of junior doctors embark on different rotations in new departments. Over a 12-month period in the UK, there are ∼21 777 incidences where a doctor in their first year of postgraduate training starts a new job.¹ In starting a new clinical placement or beginning a medical career, doctors in training must learn the content of departmental guidelines, understand and apply standard operating procedures while simultaneously adjusting to new patient presentations, clinical environment and personnel. This transition is thought to correlate with an increased risk to patient safety, more widely known as the ‘August effect’.²

The traditional way to circumnavigate these transitional hurdles is often with written or local intranet-based resources highlighted or made available at corporate or unit induction. Boulos et al³ have demonstrated that access to professional library services—knowledge into action—has a positive effect on patient care and subsequent length of hospital stay. We are also aware of the challenges of keeping written clinical guidance and hospital protocols up to date and accessible for clinical teams.

The comparative efficacy and impact of clinician authored paper-based protocols against (non-regulated) mobile applications (apps) is difficult to gauge, especially as one study demonstrated that <35% of dermatological-based medical apps had any medical expert involvement.⁴ However, we already know that mobile technologies are increasingly commonplace in medical education and clinical environments^5–8 with practitioners at all levels using apps with minimal or no local guidance or validation.⁷ Hardyman et al⁹ noted that clinical students opting to have support devices used them to augment their knowledge, achieve diagnoses and guide subsequent treatment. Other work has demonstrated that students use technology when they see a clear application and when information is readily accessible.^{8 10}

We therefore propose a further smartphone/mobile device-based intervention to test the feasibility and utility of offering standardised and quality controlled medical apps to out-of-hours practitioners (medical and nursing) as an aid to clinical decision-making at the point of patient contact.

Aim

Our aim was to develop a further solution to the modern difficulty of junior doctor transition, while merging this with the exponential increase in tablet-based availability and technology. We tested the feasibility of offering bespoke medical apps to out-of-hours medical and nursing practitioners as an aid to clinical decision-making at the point of patient contact. The devices were not intended as a replacement for appropriate supervision but as additional assistance in providing bedside evidence-based medicine by facilitating easier access to existing guidelines and protocols.

It was hoped that the benefits would be threefold:

1. Clinical education—real time support for clinical decision-making as one component of deliberate practice to build expert performance;

2. Decreased administrative burden—ease of updating and accessing current guidelines;

3. Service Development—readily accessible feedback from users on utility of guidance.

By developing ‘in-house’ apps aligned to existing protocols, we planned to support access, promote awareness and improve compliance, the goal being implementation of best practice standardised interventions within our hospitals.

Method

Prepilot

We conducted a prepilot survey of 123 current clinical staff within our acute departments. The staff included advanced nurse practitioners, consultants (attending physicians), foundation doctors (interns) and middle grade doctors (residents). We aimed to gauge awareness of protocols (existence, location), current mobile device use, barriers to protocol application and perceptions of mobile technologies. We used this information to blueprint implementation of the pilot. The questionnaire was designed to gauge current smartphone usage patterns among staff groups while also gathering information in mobile apps already in use. An example of the questionnaire is shown in online supplementary appendix 1. The form was designed using predetermined response categories to allow collection of a breadth of data.

Pilot

The pilot was run over 2 months. The project was advertised throughout the organisation via email, and with posters in settings where the devices would be used. The high-intensity environments chosen were emergency medicine departments and ‘hospital at night’ (out-of-hours cover). These were selected due to the diverse nature of pathology seen and the high turnover of staff within these areas to test the utility and acceptability criteria. Participation in the pilot was entirely voluntary; the devices were supplied within departments and doctors chose if and when to use them. We visited each department and gave a short induction on use of the devices prior to the pilot. This was kept brief, as one of the outcomes in which we were interested was the intuitiveness of the interface and content.

We provided a total of 32 devices (30 iPod touches and 2 iPad minis) in our emergency departments and out-of-hours environments. Alongside the devices, three secure lockers for storage and charging were supplied. The nature/size of the devices was chosen as a result of comments from the prepilot surveys and also logistical considerations of the areas they were to be used in. The larger iPad mini devices were placed in a department where there was a central room used as a hub, while the smaller iPod touch devices were intended to be more portable in environments where the practitioners did not have a central base to work from.

Each device was preloaded with our internally developed apps and hospital approved guidelines, including a collection of ‘smart’ PDFs that had already been created by our emergency departments. The application suite included a mobile version of an internal Medical Emergencies Handbook widely used within the organisation detailing best practice assessment and internal protocols. It was necessary to set the devices up to work offline only, due to a lack of secure WiFi within the test hospital settings.

Additionally, a number of pre-existing externally developed apps approved by our medical education department and clinical service were chosen; in selecting these apps, we amassed input from participants during the prepilot survey. Alongside the apps and protocols, the devices were loaded with a contact book containing internal extension numbers and pager numbers (figure 1).

Figure 1.

Resources loaded on to mobile devices. *SIGN, Scottish Intercollegiate Guidelines Network **NEWS, National Early Warning Score.

Data collection during the pilot

In collecting the data, we started with a broad concept and thus aimed to draw out patterns in usage while gaining feedback on the interface and thus opted to collect qualitative data. We collected information during the pilot from participants on device usability, interface and suggested improvements. This occurred in the clinical environment and was done in an open format with no predetermined response categories.

Alongside this, we retrospectively collected feedback by numerous additional means; questionnaires were distributed to participants as both paper and electronic formats, an email was sent to colleagues who had used the devices and focus groups were arranged to discuss the project. The email feedback and focus groups were designed to provide an open forum while the questionnaires focused again on theoretical principles and drawing out illustrative data from a small group of users.

Although the devices were set up to work offline, users were aware that we were able to harvest quantitative analytics detailing real-time usage data (eg, duration of use of each application, number of interactions, etc). In order to collect qualitative data, the devices were loaded with the iSurvey¹¹ tool. We created an evaluation template, in conjunction with National Health Service Education Scotland, which was imported into the format of the survey app. As well as collecting key anonymous demographics, it was designed to question clinicians on their use of the apps and the apps’ usefulness. Users were encouraged to complete the iSurvey; however, it was not made compulsory for usage. The iSurvey tool collected data using a Likert style scoring system before expanding this to allow further subjective opinion as desired.

Ethical concerns

Recent Medicines and Healthcare Products Regulatory Agency (MHRA) guidance¹² has served to clarify decision support making software on mobile devices. The use of software designed to inform healthcare professionals, subsequently enabling them to make clinical decisions is not considered a medical device.¹²

No patient data were kept on or transferred to the devices and the data contained on the devices were already available within the clinical context. To this end, we set out initially to test the intervention for utility and usability rather than content or delivery of care; as a result and with guidance, ethics approval was not required and the study was conducted adhering to the Declaration of Helsinki principles.

Given the confidentiality considerations, the camera facility on the device was disabled and in addition we provided protective housings that covered the camera lens on the devices. Furthermore this allowed identification of the devices as being work related rather than personal phones.

Results

Our prepilot survey was distributed to 123 potential users; the response rate was 41% (50/123).In total 93% of responding staff had medical apps installed on their own mobile devices and 65% used these mobile apps at some point to aid their decision-making process in clinical care. We examined pre-existing barriers to Information Technology (IT)-based protocol adherence in order to set out the pilot in the most helpful manner to attain as high a participation rate as possible. The most prominent barriers identified to access and use of current protocols were IT/terminal availability in the clinical setting (61%), current IT/website layout (50%) and excess time to access required guidance (32%).

Pilot iSurvey data

The demographics of those who responded to iSurvey (n=53) were 53% women and 47% men. Figure 2 shows the age demographic of participants taper with 70% of respondents via this mechanism being under 30.

Figure 2.

Age variance in respondents through iSurvey.

We hypothesised that the major users would be foundation doctors and had initially designed the devices and apps with this in mind. The iSurvey data reflected this with 53% of users being foundation doctors, 68% of them being Foundation Year (FY) 2s and 32% FY1s. Of the remaining users (47%), 60% were trainee grades specialist Trainee (ST) 1–3+, 30% were nurse practitioners and the remaining 10% were either consultants or staff grades.

The devices were left in departments throughout the day. In total 89% (47/53) of the iSurveys were filled out between 20:00 and 09:00 with a peak use between 00:00 and 09:00 reflecting active use of the devices. The iSurvey was created with around 70 specific questions targeted at individual components of the app package. The questions were user navigable and varied to reflect the components of the app package used by the user. With touch screen methodology this took on average 3 minutes to fill in for each participant.

In total 74% of iSurvey respondents used the Medical Emergency Handbook app with 87% finding it more user friendly than the paper-based equivalent. About 85% considered that the instant access had a positive effect on the timeliness of a clinical outcome for their patient, with a further 82% believing that it gave them extra confidence in the clinical decision they made. In total 93% said that it left them less confused or uncertain than before accessing it (7% had no opinion).

When we analysed the usage of our bespoke apps, 33% of iSurvey responders engaged with the apps with the results depicted in figure 3. The ‘Hip Fracture’ app reflected National Institute for Health and Care Excellence (NICE) recommendations for optimum management of patients with a neck of femur fracture. The ‘Suture’ app provided an interactive schematic of considerations when closing facial wounds. The ‘Head Injury’ app provided an interactive navigable platform aimed at demonstrating both Scottish Intercollegiate Guidance Network (SIGN) and NICE head injury assessment recommendations. The ‘Pulmonary Embolism’ app provided pathways for risk stratification, treatment and management of patients with a confirmed PE.

Figure 3.

In-pilot bespoke application usage.

One additional observation was that 20% of users (despite corporate and unit induction) were not aware of existing non-electronic versions of some protocols that they were using in an electronic format. From this, we can theorise additional benefits in providing junior staff with a collated library of e-guidance.

Pilot analytics results

Objective statistical measurement of the ‘traffic’ or usage data of apps within the devices showed that 50% of users accessed the locally authored acute Medical Emergencies Handbook, while 55% of people also accessed the bespoke apps providing guidance on management of varying presentations ranging from facial wounds to hip fractures. The analytics data enabled breakdowns of the duration and volume of screen views of specific components within the apps. The highest ranking data are summarised in figure 4. Additionally, this information shows a prospective benefit in tailoring a curriculum for induction and clinical training around the information actually needed on the frontline rather than learner or trainer perceived subject areas.

Figure 4.

Application sections with high viewing figures.

Pilot qualitative data

In addition to the qualitative data collected via the iSurvey application, further qualitative data were collected via three different means: face-to-face discussion, email correspondence and feedback sheets following direct device encounters. Across the course of the trial, we received 60 individual feedback statements. The use of qualitative data collection allowed us to navigate beyond numerical data describing quantifiable phenomena and appreciate the user experience and perception of the interface.

Two clinicians separately interrogated the statements and the main themes were extracted before being compared. For data analysis, the subjective feedback was then categorised into: device functionality, application specific, creative suggestions, protocol adherence and miscellaneous.

Device functionality

A consistent theme in our feedback was benefit of portability, convenience and ease of access provided by the portable technology with 60% of the comments highlighting this. Protocol adherence featured strongly among 22% of respondents with comments such as ‘I never realised there was a protocol for this’ demonstrating an added benefit of the devices. This was extended with a further 12% commenting on the educational benefits of the tools for continued professional development. The devices were seen to be positive in making desktop computers available for other tasks such as results checking and test requests (14% of respondents).

The overarching theme among the negative perception focused on poor connectivity of the devices at this stage of our pilot. About 55% felt this was hampered by a lack of universal WiFi access (an issue which subsequently has been resolved partially as a result of our project).

In total 35% of participants felt that bringing their own devices would be their preferred solution. Finally, 10% of participants who engaged with feedback felt that the training on the devices had been limited.

Application specific

In total 33% of all the comments were specific for the apps involved in our pilot. About 35% of these comments suggested additional apps to add to the devices. In total 45% offered positive comments on existing apps. The most useful apps identified were the British National Formulary (BNF) and the Medical Emergencies Handbook. One of the encouraging aspects of this feedback was that one doctor changed their future practice as a result of the device interaction stating they “have since downloaded the BNF…and Acute Medical Emergencies Handbook onto my own iPhone and use this on the wards”.

The importance of intuitiveness in user interfaces was highlighted through our feedback with 15% of users describing the Lothian Joint formulary as being challenging to navigate.

Creative suggestions

The level of staff engagement in terms of suggestions for ongoing development was fantastically encouraging. A creative theme which continued on from the functionality category was for an expansion of device connectivity (52%) and 48% suggesting alternative apps that they had themselves used and felt could benefit their colleagues and patients, highlighting the educational benefits we had hoped the devices would promote.

Further device expansion to include interactive ordering of investigations and result sign off was the most ambitious theme among our feedback being mentioned by 20% of respondents “I see the future of this evolving into a way of communicating/auditing/receiving results/signing off results/checking SOP's/giving clinical information which is up to date at the point of access/education/simulation. The uses are multiple and the rewards potentially great”.

Protocol adherence

In total 15% of responses highlighted the positive influence the devices had on influencing protocol adherence through ease of access with comments such as “It is often difficult in any organisation to achieve commonality of ways of working…these devices clearly help us to achieve this and we are at the moment just scratching the surface of the utility”.

Discussion

This pilot demonstrates that the provision of tablet-based medical apps was welcomed as a useful adjunct in the clinical environment. Other than the passcode to the devices and an outline of what we were looking at, there was no other formal induction to the user interface of the devices. The knowledge, ability and enthusiasm to interact with technology is already present within our clinical workforce.¹³ This was highlighted by the prepilot questionnaire and the amount of engagement throughout our pilot by varying ages, grades and professions. The use of the mobile devices to provide feedback has been shown to be a reasonable alternative to paper format both in experienced and naïve mobile technology users.¹⁴ The age demographic disparity in the iSurvey data is thought to reflect a training issue on how to feedback through the devices rather than an issue with technology usage.

The pilot demonstrated benefits that we had not anticipated by highlighting variation in protocol awareness for subgroups of staff. The ability to gather data on usage within the apps could inform clinical managers of protocols with low usage volumes and potentially serve to identify outdated or conflicting guidance.

The emergency department chosen is one of the busiest in Scotland¹⁵ and the idea of theft played heavily on our minds when starting the pilot. With no WiFi access, there was little way to track the devices in or out of the department other than through paper-based sign in sheets. This was a concern that did lead to some hesitancy in the departments to engage initially (this was reflected in poor usage data in the early phases of the pilot). However, we ended the pilot with the same number of devices we started it with. We did take the precaution of encasing the devices in such a way as to mask the fact that they were ‘Apple’ products, to lessen any opportunistic theft attempts and also so that patients realised that they were clinical devices and not phones. The cameras were also disabled in order to prevent any breaches of confidentiality.

Our goal throughout was to provide aids to clinical decision-making among our junior workforce. The overall impression was that this intervention invigorated people about making the right decision with the appropriate protocol as the devices made this easier to do. Since the pilot, there has been a frenzy of protocol and guideline suggestions for the next phase. The use of technology in this manner increases the channels by which staff access guidance and subsequently improves the possibility of adherence by augmenting accessibility.

Conclusion

The pervasive growth of technology in all areas of life including clinical medicine cannot be ignored.^{3 5 16} Investing in front-line staff by supplying modern technology in this manner to help support their decision-making can only contribute to patient safety and create a more efficient and effective care environment. We acknowledge a converse body of opinion that the place of protocols and perhaps the role of technology equates to a dumbing down of medicine and autonomous decision-making that we would disagree with. Acute hospital environments are challenging places in which to work, especially for junior clinicians in training. Variation in clinical practice and decision-making based on the clinician alone is widely documented. The addition of an electronic aide-mémoire could be the difference between making a good decision or a flawed one.

System-wide financial investment in mobile devices would be significant; however, our prepilot survey indicates the existing prevalence of personal smartphones or devices that could easily run medical apps. This raises the possibility of using ‘bring your own device’ (BYOD) approaches that others have advocated.¹⁷ Although this approach has many merits, this must be balanced against the potential loss of corporate regulation and quality assurance, ceding control of what is being used and accessed. This prompts the discussion and consideration in further work of a corporate ‘app-store’ populated with known, impactful and quality assured apps to support high volume, high acuity and protocol-rich environments.

People want to make the right choices, 21st century clinician-led technological advancements can facilitate this. We welcome others’ work in this area.

“If you have knowledge let others light their candles with it”—Winston Churchill