A ‘SMART’ way to determine treatment goals in pharmacotherapy education

Abstract

Aim

Determining treatment goals is an important part of the treatment decision‐making process, but medical students are not trained in a structural way on how to define these goals. ‘SMART’ criteria are widely used in non‐medical professions for determining goals and may improve treatment goal setting. The aim of this study was to assess the effect of implementation of SMART criteria on medical students' ability to set treatment goals and to analyze the effects on treatment choice and monitoring.

Methods

We performed a prospective, randomized controlled minimal intervention study with one control and two intervention groups (WHO group and SMART group). Second year medical students had to complete a WHO six step treatment plan for four written case reports of patients with asthma. The treatment plans were assessed using a standard scoring sheet developed by a Delphi procedure among respiratory physicians from all eight university medical centres in the Netherlands.

Results

A total of 251 second year medical students participated. The SMART group had significantly higher scores for setting treatment goals than the WHO and control groups (68.5 % vs. 29.6 % and 30.8 %, respectively, both P < 0.001). The SMART group also had significantly better scores for treatment monitoring than the WHO and control groups (34.2 % vs. 19.3 % and 24.6 %, respectively, both P < 0.001). There were no between group differences in treatment choice. Regardless of the study group, better setting of treatment goals was associated with better treatment monitoring, an association not reported earlier.

Conclusion

SMART criteria improve the setting of treatment goals and treatment monitoring.

What is Already Known about this Subject

• Determining treatment goals is important in therapeutic decision‐making. However medical students are not trained in a structural way on how to define these goals.

• ‘SMART’ criteria are widely used in non‐medical professions to optimize goal setting but it is not known whether they are effective in pharmacotherapy education and rational prescribing.

What this Study Adds

• SMART criteria improve the setting of treatment goals and treatment monitoring in medical students.

• Improved goal setting is associated with improved treatment monitoring.

• Providing students with simple but specific instructions on how to set treatment goals as a structural part of the pharmacotherapy curriculum is necessary to improve rational prescribing.

Introduction

New prescribers require training to ensure that they act safely and rationally. The WHO six step plan was developed to improve rational and safe prescribing by junior doctors 1. It is a normative model for therapeutic reasoning and starts by establishing the indication for treatment and ends with monitoring treatment. It has been shown to improve rational prescribing by medical students and junior doctors 2, 3, 4, 5, 6, 7. However, there have been few studies of the second step, determining treatment goals, even though medical students find this difficult in practice. Setting treatment goals is an essential step of the WHO Guide to Good Prescribing 1, and improved goal setting might contribute to the prevention of avoidable medication errors and unnecessary high medication costs, and might improve the follow‐up of patients. There is little or no information or a generally applicable guideline about how to set treatment goals, even though doctors do this daily, consciously or unconsciously, when treating patients. While a number of studies describe treatment goals for specific diseases 8, 9, 10 even the WHO Guide to Good Prescribing provides little support, stating only the importance of setting treatment goals without providing practical tools on how to do so. Moreover, goal setting has received relatively little attention in the medical literature 11 in contrast to research in other disciplines, such as psychology, organizational behaviour, management and education 11, 12, 13.

A well‐known and frequently used method to optimize the process of goal setting in general is ‘SMART’ 14. According to SMART, goals have to be Specific, Measurable, Acceptable, Realistic and Time‐bound, but other interpretations of the letters can also be found. SMART criteria have been used effectively in rehabilitation medicine 15, 16 but it is not known whether SMART criteria are effective in terms of setting treatment goals and rational prescribing. Therefore, the aim of this minimal intervention study was to assess the effect of implementation of SMART criteria on the setting of treatment goals by medical students and to analyze how these criteria influence the choice of treatment and treatment monitoring.

Methods

Study design

We performed a prospective, randomized controlled intervention study with one control group and two intervention groups (WHO group and SMART group).

Study population

All second year undergraduate medical students from the VU University Medical Centre in Amsterdam, the Netherlands, were included when they attended a mandatory pharmacotherapy training programme. These students had not previously received pharmacotherapy training and were therefore relatively homogeneous in their knowledge. The students were randomly divided into three study groups for their first pharmacotherapy training session, given on 3 consecutive days depending on their education schedule. The first group, the control group, which attended training on the first day, received no specific further instructions about setting treatment goals. The second group, the WHO group, received general instructions obtained from the WHO Guide to Good Prescribing on day 2, 1 day after the control group. This one page information leaflet indicated only the relevance of setting treatment goals to rational prescribing without providing practical tools. The third group, the SMART group, received a one page information leaflet with specific instructions regarding the use of SMART criteria to set treatment goals on day 3. These instructions consisted of an explanation of the acronym SMART and some examples on how to use these criteria to set treatment goals. It took students 1–2 min to read the English WHO and SMART instructions, but they were free to use them during the rest of the training.

Materials

In the training session, students had to set treatment goal(s), choose appropriate medications and determine what measures to monitor for four different cases of patients with bronchial asthma. These patient cases, developed by a respiratory physician, consisted of common patient information (e.g. name, birth date, gender etc.), a summary of the medical history, allergies and intoxications, comedication and/or comorbidities, a description of the current indication (containing anamnesis, physical examination and, if necessary, results of peak flow analysis for instance) and the final diagnosis. All cases were of comparable difficulty and suitable for the level of experience of our study population and could be solved with the help of generally accepted clinical guidelines.

Besides the patient case and instructions on treatment goals (WHO and SMART groups), all students received a form, based on the six step from the WHO Guide to Good Prescribing, on which to record their treatment plan. The students were allowed to use a pharmacotherapeutic reference book to look up specific drug information.

Scoring template by Delphi procedure

A standard scoring sheet for the four patient cases was developed by using a Delphi procedure among pulmonary physicians from all eight university medical centres in the Netherlands. This Delphi procedure consisted of three consecutive rounds to establish the best treatment plan for the patient cases. In the first round, 25 pulmonary physicians gave their own treatment plan. In the second round, all summarized items from round 1 were scored by 22 participants, with scores ranging from 1 (total disagreement) to 5 (total agreement). In the third round, 18 remaining members of the Delphi panel reviewed their previous scores, in relation to the average scores of all participants and were given the opportunity to decide whether or not to adjust their scores. Then all items with an average score of >3.5 were included in the scoring template. This model thus consisted of the best items for treatment goal, treatment choice and treatment monitoring according to the nationwide Delphi expert panel.

Scoring and statistical analysis

The treatment plans were scored blind to study group. The number of items matching the scoring template were counted and expressed as a percentage of the maximum attainable score. Scores for treatment goal, choice and monitoring were transferred to a database. Data management and statistical analyses were performed in SPSS (IBM SPSS Statistics, v20). A one way anova with least significant difference (LSD) was used to test for differences in scores among the control, WHO and SMART groups. Spearman's correlation analysis was used to test for associations between treatment goals, treatment choice and treatment monitoring. For statistical analysis no distinction was made between the four different patient cases. Significance level was set at 5 %.

This study fell outside the scope of the Dutch Law on Medical Research (WMO) and when the study started the Dutch Ethical Review Board of Medical Education, which could provide study approval, was not yet operational. All participants were included during their first training session in a mandatory and regular pharmacotherapy training programme in the second year of the medical curriculum.

Results

In total, 251 second year medical students were included over the 3 consecutive days. The control, WHO and SMART groups consisted of 100, 69 and 82 students, respectively. The mean scores for setting treatment goals, treatment choice and treatment monitoring of the three groups are given in Table 1. The SMART group had significantly higher scores for setting treatment goals than the WHO group (68.5 % vs. 29.6 %, P < 0.001) and the control group (68.5 % vs. 30.8 %, P < 0.001). Treatment monitoring was also significantly better in the SMART group than in the WHO group (34.2 % vs. 19.3 %, P < 0.001) and the control group (34.2 % vs. 24.6 %; P = 0.004). There were no significant differences in the scores for treatment choice between the three groups (52.2 %, 53.0 % and 49.3 %, respectively, P = 0.463). The relationships identified remained significant even if a more conservative approach than Fisher's LSD were used (Games–Howell, multicomparison).

Table 1.

Average scores for treatment goal, treatment choice and monitoring of the three study groups (Control, WHO, SMART)

	Control (n = 100) Mean (%) (95 % CI)	WHO (n = 69)	SMART (n = 82)
		Mean (%) (95 % CI)	Mean (%) (95 % CI)
Treatment goal	30.8 (28.4, 33.2)	29.6 (26.8, 32.4)	68.5 (63.9, 73.2) #
Treatment choice	49.3 (45.1, 53.4)	53.0 (48.2, 57.9)	52.2 (47.4, 57.1)
Monitoring	24.6 (20.6, 28.6)	19.3 (14.7, 23.9)	34.2 (28.5, 39.9) *

Significant differences (calculated with anova LSD analysis) are bold and highlighted with

^# (better therapeutic goals for the SMART group, compared with the WHO and control group, both P < 0.001) and

^* (better treatment monitoring for the SMART group, compared with the WHO and control group, P < 0.001 and P = 0.004)

Table 2 shows the associations between the scores for setting treatment goals, treatment choice and treatment monitoring. There was a significant association between the scores for setting treatment goals and the scores for treatment monitoring (P = 0.001), but not between the scores for setting treatment goals and the scores for treatment choice (P = 0.069). Subgroup analysis showed a similar pattern for each group which means that these associations were not significantly different between the three groups.

Table 2.

Correlation (Spearman's) between treatment goal, treatment choice and treatment monitoring, independent of study group

	Treatment goal
	Correlation coefficient	P value
Treatment choice	0.115	0.069
Monitoring	0.212	0.001 #

Significant correlations are bold and highlighted with #

Discussion

This study shows that use of the SMART criteria by medical students improved their ability to set treatment goals and monitor treatment. Moreover, regardless of whether WHO, SMART or no criteria were used, improved goal setting was found to be associated with better treatment monitoring, an association not reported earlier. This is consistent with the claim of the WHO Guide to Good Prescribing that setting treatment goals is an essential step in the six step process of rational prescribing 1.

Previous studies involving students in the preclinical and clinical phases of the medical curriculum have shown that students have relatively low scores for step 6 (treatment monitoring) compared with the other five steps of the WHO guide 6, 7, 17, 18 and this is usually perceived as a difficult step to perform by medical students. Thus it would appear that the currently used WHO six step method does not help students sufficiently to acquire this skill. For this reason, it is interesting that use of the SMART criteria by students prescribing for the first time not only improved their setting of treatment goals (both quality and number of goals) but also treatment monitoring (not the primary target for the intervention in this study). This might be because with the SMART criteria treatment goals are explicitly stated, which might make it easier for medical students to determine what measures should be used to monitor treatment. As far as we know, this is the first study to investigate the effect of use of the SMART criteria by medical students on setting goals for treatment, making treatment choices and monitoring treatment, which makes it difficult to compare our results with previous studies. However, research on goal setting in other fields, such as psychology and management, suggests that setting explicit and specific goals is fundamental to goal achievement 11, 12, 13 and has a positive effect on behaviour and work productivity 12, 13. In the medical literature, pharmacy students have shown to be capable of writing SMART learning objectives after following a continuing professional development training programme 19 and in allied health clinicians a 50 min education programme also improved SMART goal‐writing skills 20. In view of the generalizability of our results, we have therefore no reason not to assume that encountering SMART criteria in another discipline or later phase of the study medicine might also improve (treatment) goal setting.

The association between goal setting and treatment monitoring was similar in all three groups and thus it appears that every improvement in treatment goal setting improves treatment monitoring. However, use of the SMART criteria did not improve treatment choice and setting treatment goals was not significantly associated with treatment choice. In contrast with our findings, it is generally assumed that the quality of treatment goals is related to therapy choice. A possible explanation is that in treatment guidelines, for example, those for bronchial asthma used in our study, therapy choices are explicitly ordered, taking into account relevant treatment goals for a specific condition, so that defining the correct indication leads automatically to the recommended therapy choice. This is in line with the fact that although goals are a critical component of making choices, people are often not aware of the goals they have when making a specific choice 21. The WHO Guide to Good Prescribing claims that treatment goals determine the choice of P(ersonal) drug (i.e. the most effective, safe, suitable and cheap treatment, which is comparable with a guideline) for a condition rather than for a specific patient 1. The latter is affected indirectly. For example in the case of choosing a P drug for angina pectoris which can be prevented and treated 1, if the treatment goal is to stop an attack as soon as it starts, effective drugs might be sublingual short acting nitrates or nitroglycerin. If the treatment goal is prevention, effective drugs are β‐adrenoceptor blockers, long acting nitrates and calcium channel blockers. In this example the treatment goal is decisive for choosing the list of effective treatment options from the P drug or guideline.

If treatment guidelines are not available, then treatment goals are important when it comes to choosing the appropriate drug out of all the drugs available. Specifying treatment goals structures thinking 1, forcing a person to concentrate on the real problem, thereby limiting the number of treatment possibilities and making the final choice much easier 1. In this way, only information relevant to the goal is used to determine the drug choice. Information is processed in a manner that facilitates the evaluation of options relative to active goals 21. Moreover, setting treatment goals might be especially relevant to drug choices when there are potentially multiple treatment goals that are not necessarily linked to each other, for example, when pain and infection in a patient both need to be treated.

The SMART criteria are a widely and effectively used goal setting technique that is easy to use (it takes maximally 2 min to read the instructions), even for first time users. This makes these criteria a highly appropriate educational intervention for second year medical students without prior training or experience in prescribing. Providing students with simple but specific instructions on how to set treatment goals as a structural part of the pharmacotherapy curriculum might be a promising and easily implementable improvement, but this should be evaluated in future studies.

We had a relatively large study population of 251 second year medical students, randomly divided into three study groups. This student population, with no specific prior pharmacotherapy training, was relatively homogeneous in terms of knowledge and experience and therefore ideal for assessing the effect of an experimental educational intervention. However, it was not possible to include a quantitative comparison of the students' academic performance which is a limitation of the study. While we do not know whether the SMART criteria would be equally effective in more trained prescribers, such as interns and junior doctors, there is probably room for improvement in the goal setting skills of these more experienced groups, although this should be addressed in future studies.

Bronchial asthma was suitable as a ‘target’ disease because students had already had lectures on the pathology and ethology of the disorder. Moreover, the availability of well‐defined treatment guidelines for asthma, without compromising the opportunity for personal interpretation of treatment goals, treatment choice and treatment monitoring, makes it a very good subject for a first time training session. Whether the SMART criteria are equally effective when used for other diseases remains to be established.

Taking the strengths and limitations of our study into consideration, we can conclude that the SMART criteria improve the setting of treatment goals and the monitoring of treatment and that improved goal setting is associated with improved treatment monitoring. The SMART criteria might be an effective way to improve the setting of treatment goals. The WHO six step method of the Guide to Good Prescribing is used both in the Netherlands 22 and worldwide by many young, beginning prescribers and has proven effective in pharmacotherapy training 23. To improve further the prescribing skills of future prescribers, and more specifically their setting of treatment goals and treatment monitoring, the next revision of the Guide should not only stress the importance of setting treatment goals but also provide practical tools, such as the SMART criteria, to structure and improve the process of goal setting by inexperienced prescribers.

Competing Interests

All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare no support from any organization for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.

The authors thank the participating respiratory physicians of the University Medical Centres in the Netherlands for their contribution to the Delphi procedure.

Supporting information

Appendix S1 Specify your therapeutic goal: SMART

Supporting info item

Tichelaar, J. , Uil den, S. H. , Antonini, N. F. , van Agtmael, M. A. , de Vries, T. P. G. M. , and Richir, M. C. (2016) A ‘SMART’ way to determine treatment goals in pharmacotherapy education. Br J Clin Pharmacol, 82: 280–284. doi: 10.1111/bcp.12919.