Abstract
Challenges in assessment implementation include assessment blueprinting, accuracy and timeliness of result processing, item analysis and personalised feedback. These challenges were addressed by an online assessment system (OAS) that translates the educational framework of outcome-based education (OBE) into an integrated platform applicable across academic programmes. OBE principles of transparency, measurability, relevance, and individualisation was feasibly addressed using technology. The system development required continuous stakeholder engagement and feedback. Its implementation involved change management at institutional level with shifts in procedures, responsibilities, staff competencies and resource allocation. This article describes approaches taken and the lessons learned in developing and implementing OAS to enhance assessment practice for medical science programmes.
Keywords: Online assessment, Development, Implementation
Introduction
The values of assessment in education include providing feedback to students on learning progress, informing curriculum managers of students’ achievement of intended learning outcomes, making decisions about readiness of students to progress, and contributing to quality assurance (QA) and, ultimately, accreditation of academic programmes [1]. Evidence related to the design and quality of assessments is reviewed during programme accreditation for the purpose of assuring all stakeholders that graduate outcomes have been achieved and that new graduates are safe to enter the workforce [2]. This evidence includes the range of assessment methods, assessment blueprints, evidences of assessment validity and reliability, as well as standardisation of marking [3]. The credibility of an assessment system in an institution relies critically on the availability of effective policies, regulations and processes [4]. The existence of a quality assessment system ensures trust and confidence among the stakeholders, particularly in international education partnership programs. An academic institution that emphasises quality education will attract high calibre faculty and students, which in turn contributes to the academic ranking of the university [5]. Nevertheless, there are challenges for efficient assessment practices, especially when assessment related resources are shared across various medical sciences programmes within an institution. Common challenges that may inhibit assessment efficiency and effectiveness include assessment blueprinting, accuracy and timeliness of result processing, post-test item analysis and individualised feedback to students.
The need for an online assessment system arose from assessment challenges faced at our institution, but these appear no different from those frequently reported in literature. A single integrated platform was developed to manage the end-to-end assessment process, facilitate assessment QA and guide enhancement strategies. The design of the system was guided by the principles of outcome-based education, strong constructive alignment between curriculum and assessment using a variety of assessment tools, and provision of timely and individualised feedback to students [6]. Consequently, the system features include item writing for a variety of assessment tools, item review, blueprint-based test construction, administering tests, marking tests, post-examination item analysis, item banking, provision of immediate and individualised feedback to candidates. These features are available for multiple choice questions, as well as written and clinical performance assessments. The objective of this paper is to share and describe lessons learned from the development and implementation of online assessment systems across several programmes related to medical sciences including medicine, pharmacy, biomedical sciences, dentistry, dietetics, and nursing through collaboration between faculty, students, administrative staff, and information technologists. The practical tips would be applicable to implementation of other online systems, as similar issues and challenges can be anticipated in the changes.
Approach
Carry Out a Comprehensive Needs Assessment to Address Challenges in Assessment Management
Assessment management can be complex, particularly when different academic programmes share human and physical resources, yet use different assessment strategies. Different groups face different challenges. For administrative staff, the consequences of high-peak workloads, tight deadlines and complex logistic arrangements especially for skills-based assessments can be stressful. For teaching staff, blueprinting, exam vetting, standard-setting, item selection and review, marking and providing timely and personalised feedback contribute to peaks in workload. For the education office, post-examination analyses are critical to support decision making for assessment outcomes as well as quality improvement, but can be neglected amidst the rush of managing results.
Information technology can address some of these challenges, but there is no ‘magic’ recipe. While the concept of online assessments is not novel, a contextual needs analysis is essential, based on the needs of the different user groups, contributing to formulation of the design brief, priority setting and resource allocation [7]. This stage is very important to get right, as changes may be difficult and expensive after the development has commenced.
Develop Expertise in Assessment Among Faculty and Administrative Staff
An online assessment system does not replace the need for human assessment expertise close to the action. Holmboe et al. have stressed the need for medical education to focus on faculty development around assessment to ensure that decision making on competencies and student progression meets healthcare and population needs [8]. It is important to ensure that the faculty, with both administrative and academic roles have not only expertise in assessment principles [6], but also the ability to contextualize and implement these concepts in practice. For medical sciences programmes, often, there is a need to assess comprehensively both cognitive and non-cognitive competencies, hence more advanced data analysis and psychometric expertise is highly desirable. Therefore, faculty training in assessment should be a priority that is resourced well. A critical mass of faculty and administrative staff with knowledge of assessment principles play an influential role in promoting the use of online assessment system in the institution, as they can appreciate its values.
Evaluate Readiness of Institutional Resources
There are two broad approaches to implementing an on-line assessment system. The first is to acquire one of the many available online assessment software products that incorporate common assessment practices and features. However, an institution may have unique assessment practices and processes that do not fit into these off-the-shelf products, hence product customisation may be necessary, usually supported by off-site technical support in a different time zone at a price that may limit the ability to employ local, in-house assessment and Information Technology (IT) expertise. Once contracted, changes may be difficult to negotiate and expensive. The second approach is to develop an in-house developed system with its own software and in-house IT support. Institutions should evaluate these options carefully, including the availability of financial, physical and IT resources and implications for ownership, system security, data privacy and the cost of system development, maintenance and future upgrades. System security should be emphasised regardless of the approach, in order to assure all stakeholders of the integrity of the assessment system in the university. This includes dedicated server for the system as well as password protection for test access. The physical layout of facilities, computer hardware and software requirements, internet accessibility and data storage capacity also need consideration. A comprehensive evaluation plan reduces the risk of midway project abandonment, which can be very expensive.
Use a Project Management Approach
A project team that comprises a core team of members from the registrar, teaching and learning, examination and IT offices facilitates a consultative and collaborative approach to system development. The team is responsible for defining the purpose, key system features and requirements based on the stakeholders’ feedback, and communicating with the software vendor/developer to generate the system prototype. A project management approach ensures that the roles of the members, project timeline and milestones, funding and deliverables are transparent to all stakeholders. Given the diverse needs of the potential users of the system, the project team plays a critical role in moderating, developing consensus and aligning requirements with institutional policies for assessments, guided by assessment best practice.
Adopt a Collaborative and Consultative Approach in Defining User Requirements for the System Throughout the Project
The stakeholders of online assessment system include teachers, programme administrators, examination administrative staff, IT support staff and students. Their engagement in the system development is important and consistent with the design-thinking approach which has a rationale based on the concept of collaborative stakeholder engagement [9]. In this model, the stakeholders should be involved in all three phases (i.e. inspiration, ideation and implementation) that form the continuum of innovation and not be limited to identifying needs at the early stage of the project. An open environment of continuous improvement encourages innovation [10].
Decisions on system features that reflect the needs of all stakeholders create a sense of respect and collaboration among the various groups of users. Besides testing the scenarios and verifying whether the features meet the users’ requirements, the stakeholders can provide valuable feedback regarding the user friendliness of the online assessment platform and the assessment environment. This is a critical phase as subsequent system changes after product signing off will not only incur cost, but also possibly result in undesirable system instability as functions are often inter-linked.
Prepare a Risk Management Plan for Online Assessments
According to the ISO 31000:2018 risk management guidelines, risk management refers to ‘coordinated activities to direct and control an organization with regard to risk’ [11]. The ISO 31000:2018 standard further states that ‘risk is usually expressed in terms of risk sources, potential events, their consequences and their likelihood’. Since the online assessment system is an integrated, end-to-end system, risks can occur at any stage, including examination scheduling, item authoring, item review, test construction, testing taking and marking. In particular, the risk of technical failure during online test taking is potentially disastrous as it may result in breaches in security, loss of data and delay in results. The causes can be software or hardware problems due to internal or external factors, so a comprehensive risk management plan is required that takes into account the views of all stakeholders and decision makers at all levels. It aims to help the stakeholders (particularly the faculty and examination administrators) to make informed decision and prioritise their actions among the available options. It should be a transparent document that is aligned with the university assessment policy and ensures a systematic, structured, consistent and timely solution when problems arise.
Evaluation
Feedbacks were obtained from the stakeholders after the first year of implementation as part of project evaluation. These are summarised in Table 1. The faculty appreciate the improvement in the efficiency of the assessment process (authoring and vetting of examination questions and test paper construction), as well as the post-examination item analysis data that are produced promptly by the system at the completion of assessments. Besides, assessment blueprinting is feasibly enhanced, and a useful question bank with student past performance data is now available. Other positive comments include the system security features as well as improved timely personalised feedback to the students on their examination performance. Meanwhile, the students appreciate the detailed feedback on their performance according to learning topics and outcomes as well as other metadata tagged to the questions (e.g. learning outcomes).
Table 1.
Stakeholders’ feedback on usefulness of online assessment system
| Stakeholder group | Feedback |
|---|---|
| Faculty |
• Reports generated post examinations as well as individual feedback report for students are useful. • Psychometric analysis provides useful data on the quality of assessment • Ease of preparation of examinations: lecturers key in questions to OAS directly, module/exam coordinators prepares exam papers in OAS, and hence OAS acts as a question bank • Potential use of OAS to support personalised learning |
| Examination administrative staff |
• Effective tracking of examination question vetting processes • Adherence to timelines for preparation for examination papers • Automated results processing reduces transcription errors • Manageable record keeping • Reduced results processing time |
| Students |
• Easy change of options for multiple choice questions during online test taking compared to paper-based answer scripts • Images in questions are clearer compared to hardcopies • Online test taking is environmentally friendly (as it saves paper) • Convenient access of results from within and outside campus. • Feedback on examination performance by learning topics and outcomes and difficulty level is useful for future improvement. |
From the examination administrative perspectives, the overall assessment process efficiency has improved, and there is evidence of better adherence to timelines for examination papers preparation. Transcription errors in examination results processing have been reduced and results can be released within shorter period of time. Record storage of assessment data becomes more manageable.
Discussion
The development of an online assessment system in our university is a collective effort of multi-level staff from various departments, with the common goal of enhancing the student experience in assessments and improving the operational efficiency in assessment management. The online assessment system has successfully transformed the assessment practices in our various medical sciences programmes, bringing about improvement of operational efficiency, personalised feedback to students and potential enhancement of assessment quality. In addition to our own institutional experience, one co-author was also involved in the development of the IDEAL consortium for online assessment items and expertise for medical schools [12], and the other authors had facilitated workshops related to online assessments at national workshops and international conferences such as Ottawa Conference 2018 and Asia Pacific Medical Education Conference 2018. Through our joint experiences, we found similar challenges for implementation encountered by other international colleagues, suggesting that the lessons learned will be helpful for those who are planning to leverage on technology to improve assessment practices. As assessment in health professions programmes is increasingly being regarded as a ‘system’ rather than a collection of individual assessments [13], it is foreseeable that institutions will gradually move towards the use of online assessment system to manage assessment data from multiple sources according to institutional policy.
Lessons Learned
Maintain Proper Documentation During System Development to Assist Project Continuity
The development phase requires testing, modifying and retesting before the prototype is finalised. Proper documentation is necessary to keep track of all aspects considered during system development and the rationale behind. Besides, it facilitates the process of information transfer when staff transition occurs. Issue logs should be maintained on a common platform and accessible by the project and software vendor teams. This helps to ensure efficiency of the tracking and follow-up actions. A well-documented record of the entire process including feedback from the stakeholders and consensus decision is essential to provide clarity to the stakeholders and safeguard the integrity of the project team.
Communicate the Purpose and Values Clearly to all Stakeholders
A major challenge in the implementation of an online assessment system is achieving buy-in from the faculty and administrative staff, particularly in convincing them of the benefits of the new system. Changes in assessment practice from conventional mode (i.e. paper-based) to online assessments implies a need for new IT skills. Change has been categorised by its rate of occurrence, how it comes about and scale [14]. To facilitate the transition, principles related to the Kotter’s process for successful organisational transformation can be applied [15, 16]. A strong rationale for the change needs continuous re-inforcement, supported with real examples that staff are aware of e.g. lack of question bank in the educational programmes, weaknesses in post-test analyses, lack of personalised feedback on student performance and transcription errors in result processing. This helps the stakeholders to gain clear understanding of the reasons and a sense of urgency for change. Further, progress on each new milestone achieved should be communicated to all stakeholders. For example, when the online assessment was launched with one-best-answer question, faculty received immediate post-assessment feedback on their questions and class performance, while students received individualised feedback on their performance based on learning outcomes and topic. It is important to communicate the benefits to the students prior to the implementation online assessment in order to enhance their acceptance [17]. In particular, students need an assurance that the change in exam mode will not affect test scores and progress decisions.
Identify Change Leaders from Various Groups of Stakeholders
The change from paper-based examination to online assessment practice can be facilitated by recruiting effective change leaders who interact well with individuals in the organisation to work closely with the project team. They will assist identification and analysis of potential sources of resistance and failure, and provide feedback so that issues could be resolved in a timely manner. Change-adept organisations focus on empowering their members and building their capacity through personal professional development [10]. Opportunity must be provided to enable adjustment to the change in a safe environment [18].
The common stages of adapting to change include early denial, dip in productivity, frustration, accepting the new reality, starting to make sense of the new way of working and integrating the change into everyday life [19]. It is important to realise that negative behaviour can be obstructive to the change; therefore, the role of the change leader is also to engage in continuous communication with the users and sharing with them of the positive feedback from those who have successfully adapted to the change.
Provide Ample and Timely Technical Support
Three styles of computing use among teachers have been identified: avoidance, integration and technical specialisation [20]. The ‘avoidance’ group distances themselves from computers and related activities, whereas the ‘integration’ group integrates technology into their teaching activities in order to enhance the learning experience of their students, and the ‘technical specialisation’ group embraces and promotes technology to enhance curriculum delivery. Reasons for teachers’ resistance to computers include time management problems, lack of administrative support and personal or psychological factors [21]. Adequate, hands-on training can assist teachers during the transition to online assessment practice, along with easy to follow, step-by-step user guides and video demonstrations that users can refer when necessary. During the initial phase, it is preferable to dedicate technical personnel to help users to troubleshoot problems when encountered and provide individual guidance when necessary. Subsequently, a community of practice can be established to create an atmosphere of collegiality and collaboration among the different groups of users.
Ensure Continuous Engagement with the IT Team
The sustainability of online assessment system depends heavily on the IT infrastructure at the institution which includes the network, servers and application software. The failure of any of these components will affect the system functionality and is especially highly undesirable during online test taking. The situation can be further complicated by the involvement of different external vendors who provide services for the various IT infrastructure components. An efficient system for vendor management is important to ensure effective communication, collaboration and timely resolution of any issues. Hence, continuous engagement with the IT team is essential for coordination of users and vendors in supporting system operation. Further, the IT team should also be entrusted to review periodically and upgrade IT infrastructure to enhance the user experience.
Challenge the Status Quo of Assessment Practice When Transferring from Paper-Based to Online Assessment
Online assessment should not be seen as ‘cut and paste’ of paper-based assessments that are mainly text-based and constrained in assessment formats. Technology offers innovative ways of conducting assessment online such as the feasibility of incorporating multimedia in questions and conducting clinical examination online through observation by the examiner from a distant location [22]. Although our experience suggested that the ‘cut and paste’ approach could facilitate the stakeholders’ engagement at the early phase of the system development due to their familiarity with the existing assessment procedures, it was realised that the addition of technology in the design process had led inevitably to questions about some of the existing procedures and rules.
Online assessment should be viewed as an opportunity to continuously improve assessment practices. Faculty who author test items are responsible for the tagging of learning outcomes, difficulty level and Bloom’s taxonomy to the test items. This ensures that all assessments are blueprinted and students will get personalised feedback on their performance. Moreover, the post-examination item analysis reports can provide information to the faculty regarding the item quality and learning outcome achievement by the students. Unnecessary assessment procedures that limit the full potential of online assessment system need to be removed. An example is the control of system access based on user roles. We found that the automation of results release led to questions about who can access these features. That is, should both faculty and students be given direct access to the student performance data, and how much detail should be released to them? Other opportunities to revisit the current practice include the levels of item review required, which types of multiple choice questions are best, institutional policy for internet data security and the need for increased learning analytics.
Conclusion
The development and implementation of an online assessment system require expertise, teamwork and continuous stakeholder interactions. The lessons learned are presented based on our experience in developing and implementing an integrated, reliable, secure and valid online assessment system for outcome-based education with a range of assessment tools for health professions education. The ultimate goal is to enhance the student experience in assessments, advancing the quality of assessments as well as improving the operational efficiency in assessment management. Moving forward, the longitudinal student performance data in the system can be further explored for the development of learning analytics. This will offer opportunities for individualised guidance to the students during their learning journey.
Acknowledgments
We would like to acknowledge Chin Sheau Yuen, Business Applications Assistant Manager of the IT Department of International Medical University, for the technical assistance provided during the development of the Online Assessment System at the university.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical Approval
NA, as this is a reflection from the authors in the form of a monograph with lessons learned.
Informed Consent
NA, as no participants were recruited. Stakeholders feedback was extracted from the OAS project evaluation report.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.GMC. Assessment in Undergraduate Medical Education: Advice Supplementary to Tomorrow’s Doctors (2009). General Medical Council. 2011. https://www.gmc-uk.org/-/media/documents/Assessment_in_undergraduate_medical_education___guidance_0815.pdf_56439668.pdf. Accessed 12 Mar 2019.
- 2.Downing SM, Yudkowsky R. Importance of assessment in health professions education. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York: Routledge; 2009. pp. 1–20. [Google Scholar]
- 3.WFME. Standards. World Federation for Medical Education. 2018. http://wfme.org/standards/. Accessed 12 Mar 2019.
- 4.QAA. UK quality code for higher education. Part B: assuring and enhancing academic quality, chapter B6: assessments of students and the recognition of prior learning. The Quality Assurance Agency for Higher Education. 2018. http://www.qaa.ac.uk/docs/qaa/quality-code/chapter-b6_-assessment-of-students-and-the-recognition-of-prior-learning.pdf?sfvrsn=9901f781_8. Accessed 12 Mar 2019.
- 5.Davis M. Can college rankings be believed? J Des Econ Innov. 2016;2(3):215–230. [Google Scholar]
- 6.Shumway JM, Harden RM. AMEE guide no. 25: the assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25(6):569–584. doi: 10.1080/0142159032000151907. [DOI] [PubMed] [Google Scholar]
- 7.Owen JM. Program evaluation: forms and approaches. 3. New York: Guildford Publications; 2006. [Google Scholar]
- 8.Holmboe ES, Ward DS, Riznick RK, Katsufrakis PJ, Leslie KM, Patel VL, Ray DD, Nelson EA. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. 2011;86(4):460–467. doi: 10.1097/ACM.0b013e31820cb2a7. [DOI] [PubMed] [Google Scholar]
- 9.Brown T. Design thinking. Harv Bus Rev. 2008;86(6):84–92. [PubMed] [Google Scholar]
- 10.McKimm J, Swanwick T. Educational leadership. In: Swanwick T, editor. Understanding medical education: evidence, theory and practice. Chichester: John Wiley & Sons; 2014. pp. 473–491. [Google Scholar]
- 11.ISO. ISO 31000:2018 Risk Management Guidelines. ISO. 2018. https://www.iso.org/obp/ui/#iso:std:iso:31000:ed-2:v1:en. Accessed 12 Mar 2019.
- 12.IDEAL Consortium. International databases for enhanced assessments and learning. IDEAL Consortium management company limited. 2016. https://www.idealmed.org/2019/. Accessed 12 Mar 2019.
- 13.Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Hays R, Palacios Mackay MF, Roberts T, Swanson D. 2018 consensus framework for good assessment. Med Teach. 2018;40:1102–1109. doi: 10.1080/0142159X.2018.1500016. [DOI] [PubMed] [Google Scholar]
- 14.Senior B. Organisational change. 2nd ed. London: Prentice Hall.
- 15.Kotter JP. Leading change. Boston: Harvard Business School Press; 1996. [Google Scholar]
- 16.By RT. Organisational change management: a critical review. J Chang Manag. 2005;5(4):369–380. doi: 10.1080/14697010500359250. [DOI] [Google Scholar]
- 17.Boeve AJ, Meijer RR, Albers CJ, Beetsma Y, Bosker RJ. Introducing computer-based testing in high-stakes exams in higher education: results of a field experiment. PLoS One. 2015;10(12):e0143616. doi: 10.1371/journal.pone.0143616. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Moran JW, Brightman BK. Leading organizational change. J Workplace Learn: Empl Couns Today. 2000;12(2):66–74. doi: 10.1108/13665620010316226. [DOI] [Google Scholar]
- 19.McKimm J, O’Sullivan H. Leadership, Management and mentoring: applying theory to practice. In: Cleland J, Durning SJ, editors. Researching medical education. Chichester: John Wiley & Sons; 2015. pp. 269–280. [Google Scholar]
- 20.Evans-Andris M. Barrier to computer integration: micro-interaction among computer co-ordinators and classroom teachers in elementary schools. J Res Comput Educ. 1995;28:29–45. doi: 10.1080/08886504.1995.10782150. [DOI] [Google Scholar]
- 21.Robertson SI, Calder J, Fung P, Jones A, O’Shea T, Lambrechts G. Pupils, teachers and palmtop computers. J Comput Assist Learn. 1996;12:194–204. doi: 10.1111/j.1365-2729.1996.tb00051.x. [DOI] [Google Scholar]
- 22.Walsh K. Point of view: online assessment in medical education – current trends and future directions. Malawi Med J. 2015;27(2):71–72. doi: 10.4314/mmj.v27i2.8. [DOI] [PMC free article] [PubMed] [Google Scholar]