What is the impact of Apps in medical education? A study of CAPSULE, a case-based learning App

Abstract

Introduction

Mobile applications (Apps) are popular in medical education; yet, the actual benefits for students are yet to be formally researched. Clinical And Professional Studies Unique Learning Environment (CAPSULE) is an App created by Brighton and Sussex Medical School. The App provides 650 cases offered to students in their final two years of the undergraduate programme. The App performed consistently well in student feedback, and therefore, a study into the educational benefits of the App was constructed.

Methods

A cross-sectional study was performed following two years of use by students to investigate the relationship between App usage and decile ranking.

Results

The study found that the students who completed more cases tended to score higher per case (p value=0.0037). The study also found a trend between having higher case scores and being part of a stronger decile (p value=0.019).

Conclusions

Greater App usage was linked with performing better in the App itself and this was further associated with being in a stronger decile rank. From a user perspective, the data generated from the App could help with identifying students who are underperforming or help students to recognise areas on which they need to focus.

INTRODUCTION

The advancement of technology over the past decade has introduced new devices into the e-learning field. ¹ Smartphones and tablet computers continue to transform the landscape of both healthcare and e-learning, allowing users to learn and work ‘on-the-go’ or in a way that is more convenient to them. ^{2 3} Medical students have reported using mobile applications (Apps) in all aspects of learning about patient care, including history taking, examination, investigations, prescribing and clinical management. ⁴ In addition, medical students have been shown to use Apps for exam preparation and to make constructive use of downtime between clinics or lectures. ^{2 4–6} Although largely perceived positively by medical professionals, Apps have not yet become part of the foundations of medical education. ⁷

The most commonly reported disadvantages of mobile learning are inconsistent availability of internet in hospitals, ^{6 8 9} limited phone storage capacities, ¹⁰ short mobile battery life, ^{8 10} small screen ^{4 9} and cost of a subscription. ^{5 8} There is also the matter of etiquette and when is appropriate to use a mobile phone in the clinical environment. ¹¹ Finally, whether this form of learning makes any difference to the knowledge levels needed to succeed at medical school is yet to be conclusively demonstrated, although there have been many individual studies. ^{12 13} Despite these caveats, the educational healthcare App market continues to grow rapidly. ¹⁴ Governing bodies such as the General Medical Council (GMC) in the United Kingdom have acknowledged the importance of the use of new technologies to deliver teaching as early as 2003 in ‘Tomorrow’s Doctors’ and again in the more contemporary ‘Outcomes for Graduates 2018’. ^{15 16}

Clinical and professional studies unique learning environment: CAPSULE

Brighton and Sussex Medical School (BSMS) created a bespoke digital learning resource known as the Clinical And Professional Studies Unique Learning Environment (CAPSULE). In 2017, the App was offered to students in the final two years (fourth and fifth) of the undergraduate medical degree. The resource was designed to support students on clinical placements to help them to acquire, rehearse and consolidate knowledge on patient history taking, examination, investigations and disease management. The App and website (which contained the same content) house around 650 case-based clinical scenarios, with approximately 3500 questions across all medical, surgical and other specialities taught during those senior two years. The cases contain over 500 images and videos. The question formats include multiple-choice questions, both true or false and single best answer, answer matching, ranking and numeric style questions, although the majority of questions are in the single best answer format. Also, students can work through ethical, medico-legal and professionalism-related cases. There is also a separate therapeutics section to help students understand relevant medications. The App allows you to access the cases individually or generate ‘mini-quizzes’ of a random or selected collection of cases, which is one way it can be incorporated into formal teaching. For example, following a lecture on respiratory pathology select cases can be suggested to reiterate or add further clinical relevance to the lecture. Students can select cases on which they have previously performed poorly on and are also able to view the average performance for that case.

CAPSULE has a dedicated editorial board to oversee and review the content and ensure it remains accurate and aligned with the BSMS curriculum, GMC learning outcomes and the Medical Licencing Assessment. ^{16 17} The cases were written by clinical faculty and moderated by subject leads at BSMS before being mapped to the course curriculum. Each question contains feedback, available to view at the end of the case, along with a score for answers given and a mean score for that case by the cohort. The feedback includes an explanation or reasoning behind the answers given. The App was designed with many of the pitfalls of current Apps in mind and was therefore written at an appropriate difficulty level for medical students, provided to students free of charge, available offline and accessible on multiple platforms with only modest demands on phone memory. During the yearly course evaluation by students, the App constantly scored consistently well with very positive written feedback.

The study aimed to evaluate the potential relationship between CAPSULE usage and summative performance in the form of decile ranking.

METHODS

Following ethical approval by the University of Sussex Research Governance and Ethics Committee (Ethical approval number—ER/BSMS9E4C/1), a cross-sectional study was performed on the 2019 graduating cohort at BSMS. These students had access to CAPSULE for their final two years at medical school. A short lecture on CAPSULE was conducted at the start of their fourth year to introduce the App to the students. The lecture explained how and why CAPSULE was created and encouraged students to make use of the resource. Students were informed that completing cases on CAPSULE was not a compulsory part of the course nor was it incorporated into any formal assessment. Following the final exams, student consent was obtained to access both the student CAPSULE meta-data as well as student decile rankings throughout their time at BSMS. Each student’s meta-data and decile ranking was paired, before being anonymised prior to analysis. Statistical analysis was conducted in Microsoft Excel.

At BSMS, each medical student is placed into 1 of 10 deciles, with decile 1 containing the students with the highest summative scores (stronger decile) and decile 10 containing the students with the lowest summative scores (weaker decile). For this cohort of students, decile ranking was a combination of summative assessments from multiple years, with a weighting towards knowledge tests over practical examinations or coursework and with latter years being weighted more heavily than former years. The students’ decile data included their decile at the start of their third year before any access to CAPSULE, as well as their final decile ranking. With this data, a change in decile ranking could be calculated.

The inclusion criterion for the study was any graduating final year student who had access to CAPSULE for their final two years. The number of cases completed in CAPSULE by any student was not an inclusion or exclusion criterion and so even students who had completed zero cases in CAPSULE were included in the study, to ensure that all students could be represented no matter how they chose to use CAPSULE, to ensure no selection bias to any findings. Any student who was not part of the main final year cohort (such as those who had to re-sit one of their two final years) was excluded from the study as their access to CAPSULE would not be consistent with the rest of the group.

The CAPSULE meta-data included the number of cases completed at the end of each year, the overall number of cases completed and the combined mean average score for all completed cases.

RESULTS

Participation

A total of 75 final year medical students consented to have their results analysed. Five students were excluded from analysis having had to re-sit one or more of their final two clinical years. This meant that 70 students entered the final analysis, approximately 60% of the total graduating cohort.

Decile ranking analysis

All decile ranks were represented (figure 1). The mean decile rank (addition of all decile ranks of students divided by the number of students) for the group was 5.1 at the start of their third year and 5.0 for their final ranking, demonstrating fairly even distribution of students willing to partake in the research. The majority of students (46/70) did not change ranking at this time. The remaining students changed by one place, except for one student who dropped two decile ranks.

Figure 1.

Distribution of students and their decile ranks.

Case analysis

All students analysed completed at least one case. The mean number of cases completed was 235, with a range of 1–985 (repeated cases were counted separately). The mean score for all completed cases per student was 71.93% (range 55–82%). The mean score for each case was compared to the number of cases completed (figure 2). Logistic regression analysis was used to investigate associations and p values of <0.05 were considered significant. This analysis demonstrated that as the number of completed cases increased, so too did the average score for each case with a p value of 0.0037, which implies a statistically significant correlation between completing more cases and an improved total mean score.

Figure 2.

Total cases completed and student average score.

The mean score of cases completed per student was compared to their final decile ranking (figure 3). Logistic regression analysis was again used to analyse results and investigate any association. Analysis of the data showed students with a higher total mean score for cases belonged to stronger deciles, with a p value 0.019, which again demonstrates a statistically significant correlation between students who scored a higher mean for cases with a stronger decile ranking (deciles 1, 2 and 3).

Figure 3.

Student average score and decile rank.

DISCUSSION

There were a number of significant findings in this study. First, students who completed more cases in CAPSULE tended to score higher marks for the cases they completed and thus had a higher overall mean score on completed cases (figure 2). Analysing the data, it suggests that for every 100 cases completed, the mean score of the student increased by almost 1%. This correlation may be due to the ability to reattempt cases having learnt from the feedback provided, enabling improvement on previous performance. It is likely that students who fully embraced this learning resource would also have incorporated it into their exam revision schedule, when their factual knowledge was at its greatest, thereby contributing to improved performance. However, how or when CAPSULE was used was not included in the data collection; therefore, this question cannot be addressed from this study. There were still many individual outliers to this finding, with some students scoring extremely well having answered relatively few questions, and some students scoring poorly despite answering a large number of questions. Nevertheless, the majority of students had a higher mean case score when they completed more cases.

Second, the students who scored higher marks on their CAPSULE cases tended to be part of a stronger decile and students who scored poorly on cases were part of a weaker decile (figure 3). This finding could be used in many ways, especially for those who scored poorly. For example, to give extra support to students who self-identify as scoring poorly in the App in a specific subject or speciality.

Together, these findings would suggest that students who completed more cases scored higher marks overall and students with a better score in the App were part of a stronger decile. Further investigation is required before causality can be considered. Despite statistical significance, this does not ultimately answer the question about whether CAPSULE has indeed improved exam performance. How to determine which new teaching intervention has positively impacted students is a problem far beyond the scope of this study. However, these findings, along with its high usage, are suggestive that CAPSULE is of benefit to our students. If CAPSULE was shown to be used in preference to other educational resources, then there would be a potential to add to the number of cases, and thus support students in a way they find enjoyable.

With the mean number of cases completed per student being 235, this is clearly a resource into which students are investing a substantial quantity of their time. A complementary study into how much time students spent on this App in comparison to all other educational resources would provide information into how much value students placed on using CAPSULE and might give guidance into how much time should be invested by universities into App integration.

Finally, very few students at BSMS changed decile ranking over the observed period. Future studies could select a more sensitive outcome measure to assess the impact of the App or resource they are assessing on student learning.

Limitations

This was a small study and only looked at one App; therefore, these results are not generalisable to other Apps. During this study, there was no enquiry into other resources used by students. There is a possibility of several confounding variables, including collateral use of other Apps. Furthermore, the whole target cohort did not consent to have their information taken and there was no examination into why certain students did not consent to have their data analysed. As previously discussed, how or when the student chose to use CAPSULE would have greatly added to how the data could be interpreted. Also, no assessment of tacit knowledge was made prior to use of the App, nor was there any tracking of how the mean scores of each student changed as they completed more cases. The only information included in this study was the number of cases attempted and the mean result from these.

CONCLUSIONS

CAPSULE has been demonstrated to be a useful and versatile educational tool that is popular among students. The App provides rich data about its users and allows universities to identify poorer performing students or allows students to self-identify areas on which to focus. This study shows the types of data that can be generated when universities own their Apps. Future studies could use the App to research tacit knowledge or allow for comparison of different student groups as data is easily comparable between different cohorts, even between universities. Following this research, BSMS will continue to develop and integrate CAPSULE, a popular App with potentially promising educational benefits.

What is already known on this subject.

• Mobile learning is becoming increasingly popular among students.

• There are a wide variety of Apps available, all designed to offer education in the majority of medical specialities.

• For most Apps, there is no governance or standard setting and little is known about their educational impact.

What this study adds.

• Determining the overall educational impact of a single intervention is challenging.

• For this App, increasing use improves in-App scores.

• Students who scored higher in the App tended to be part of a higher decile rank, suggesting that well-designed and curriculum-linked Apps could be course-integrated in a way that was beneficial to students.