A consensus-based tool for capability benchmarking of emergency medical services in South Africa

Abstract

Introduction

Standards for Emergency Medical Services [EMS] have recently been introduced in South Africa in a movement towards the promotion of quality improvement. While these standards identify a minimum set of criteria for EMS quality they do not differentiate between services just meeting them and those exceeding them. Benchmarking may be a helpful exercise in beginning to address the question of comparative levels of capability in EMS beyond a set of minimum standards. The aim of this study was to develop a consensus-based capability benchmarking tool for EMS organizations within the South African context.

Methods

A total of 12 experts in the field of EMS in South Africa consented to participate in two Delphi Surveys in order to achieve consensus on the core components of an EMS organization as well as relevant level descriptors for those components. The resulting data was used to develop a consensus-based capability benchmarking tool for EMS organizations in South Africa.

Results

A consensus-based capability benchmarking tool was developed that allows organizations to distinguish whether the organization's capability, as a whole, is underdeveloped, developing, or well-developed. This is in addition to identifying how capable they are in all individual components or sub-components.

Conclusion

It is recommended that further research be conducted to assess this tool's implementation within different EMS organizations in South Africa, and that this study is used as a stepping-stone for additional research into meaningful quality improvement in emergency medical services in South Africa.

African relevance.

• •Emergency Medical Services in Africa, and South Africa, face unique challenges and have evolved in unique ways despite often drawing upon models of structure and operation from high income countries.
• •While quality improvement efforts are well known in EMS in high income countries, the unique context of EMS in low-to-middle income countries makes translation of this knowledge difficult.
• •With South Africa as an example, and its lack of an established quality improvement framework, this research suggests a first step towards such a framework by proposing a capability benchmarking tool based on local EMS expertise and consensus.

Background

Emergency medical services [EMS] are complex systems worldwide [1]. Significant variations within EMS can be found among service providers, patient care pathways, and quality care indicators, among others [1]. Due to the significant differences in EMS found in various countries, and sometimes even within the same country, it is challenging to identify generic rules or approaches to EMS planning and operation [1]. Emergency medical services in South Africa is still relatively new, and only became a formally recognised part of health services in the 1970s [2]. Since then, South African EMS has grown and developed rapidly [2,3]. This development was closely linked to the changes that took place within EMS education and training and resulted in a somewhat unique EMS system [3]. In addition to some unique characteristics of EMS in SA, are unique challenges.

One such challenge is that of meaningful improvement, stemming from local, relevant research. There has historically been a lack of standardisation in South African EMS, however in December 2022 the National Department of Health, in consultation with the Office of Health Standards Compliance, published a set of EMS regulations which provide standards for EMS [4]. While publication of these EMS standards is a much-needed movement towards the promotion of quality prehospital emergency care, they provide a minimum set of criteria and do not differentiate between services just meeting these standards and those exceeding them. It is believed that benchmarking may be a helpful exercise in addressing the question of comparative levels of capability in EMS beyond a set of minimum standards. Benchmarking can be used to assess several aspects of any organization, and for this research, the focus was on benchmarking the capability of EMS organizations in South Africa.

While benchmarking has been around for many years, it is a relatively new concept in healthcare, first appearing in the 1990s [5]. Over the years, benchmarking has become more common in healthcare and can be used to assess, compare, and improve just about any function within any organization. Benchmarking can be done internally or externally, focusing on clinical practices or non-clinical processes; its uses and benefits in healthcare are extensive.

In order to understand why capability was the focus of this benchmarking tool, it is necessary to appreciate how the terms ‘capability’, ‘performance’ and ‘quality’ relate to each other. Capability can be described as an organization's ability to carry out a particular task. Performance relates to capability as it refers to the way an organization carries out a certain task. More often than not, poor performance can be linked to incompetence or a lack of capability to carry out the tasks required [6]. Performance is also closely linked to quality, and organizations with good quality measures perform better [7]. From these descriptions, one can conclude that an organization capable of achieving the necessary tasks will have the potential to perform well and be of high quality. Therefore, one way to begin improving the services rendered by an EMS organization would be to identify an organization's capabilities and compare those to that of other similar organizations.

Due to an apparent lack of standardized terminology within EMS in South Africa, and no common understanding of the relative performance and quality of an EMS organization operating in South Africa, one needs to start with the very basics. Therefore, while benchmarking is most commonly used as a measure of performance and quality, this study focused on benchmarking as a measure of an EMS organization's capabilities. If one starts by creating a tool that describes the capabilities of EMS organizations of different maturities, standard and consistent terminology may be attained describing the operations of an EMS organization in South Africa. This is the first step in self-assessment and self-improvement, and all further benchmarking tools will have a common starting point. Thus, the aim of this study was to develop a consensus-based capability benchmarking tool for EMS organizations within the South African context.

Methods

This study took place in South Africa and used a prospective, quantitative research design that made use of the Delphi technique. The Delphi technique was used to transform the sample's opinions into group consensus, and this information was ultimately used to develop the capability benchmarking tool [8,9]. Purposive sampling was used to select the sample – a group of experts in the field on EMS in South Africa. An individual was deemed an expert if they had been involved in South African EMS for 10 years or longer and held a position of management in an EMS organization [regional or higher] or an educational institution [Head of Department or higher in a Department offering pre-hospital emergency care qualifications]. A total of 15 individuals who met these criteria were identified and contacted, and 12 of these individuals provided their informed consent to participate in the study. These 12 participants included three from EMS organizations in the private sector, four from public sector EMS and five from EMS education and training institutions in South Africa. Three were academics, two were Heads of Department, one was a Medical Director at a Provincial EMS, one was a Senior Medical Officer, one was a Clinical Head, one was a Provincial Deputy Director: Health, one was an Acting Principal [of a Provincial College of Emergency Care], one was a Provincial EMS Director and one was a Director in the National Department of Health. Furthermore, of these individuals, five were based in Gauteng, three were based in the Western Cape, two were based in KwaZulu-Natal, one was based in the Free State and one was based in Limpopo. This study was done in three phases.

Phase 1 involved reviewing literature to identify the core and generic components that make up an EMS organization. A broad literature review was performed included peer-reviewed journals and secondarily included grey literature relevant to EMS organizational characteristics and structure. The information obtained was then used to formulate the first Delphi survey, which required participants to rate the importance of these components for benchmarking purposes. For this research [this phase and the following one], consensus was defined as agreement from at least 70 % of participants [9,10].

Phase 2 involved reviewing literature and analysing each of the components that achieved consensus in phase one in order to form level descriptors for these components. Three level descriptors were developed for each component based on information found in the literature which we felt reasonably differentiated the three levels. These level descriptors were ‘basic’ that might be found in an underdeveloped EMS, ‘intermediate’ that might be found in a developing EMS and ‘advanced’ that might be found in a well-developed EMS. These level descriptors were then presented to the participants, who were required to indicate to what extent they agreed or disagreed with the provided statements as descriptors of the EMS organization's level of development.

Phase 3 involved mapping the relationship between the agreed-upon level descriptors from phase 2 and the various capability benchmarking levels. This was the development of the final benchmarking tool. Phase 3 also included adding a scoring system, and a set of instructions for the use of this tool.

Ethical approval for this research was obtained from the University of Johannesburg Faculty of Health Sciences Research Ethics Committee with reference REC-584-2021. All participants provided written informed consent prior to the beginning of data collection.

Results

During phase one, 14 core components of an EMS organization were identified through the literature review. These components were then presented to the experts, and consensus regarding which components were necessary to include in the benchmarking tool was achieved in a single round. Ten consenting experts completed phase 1. The participants felt that all 14 components should be included. Structurally, some closely linked components were merged while sub-components were added to others in order to produce meaningful level descriptors in the following phase. The resulting list of components and their sub-components is shown in Table 1 below.

Table 1.

Components of an EMS organization and their respective sub-components.

EMS Component	Sub-component
Human Resources/Assets	Role specialization of personnel
	Emergency care qualifications of operational personnel
Communication Systems	Consistency of communication systems within the organization
	Use of technology for operational communication
	Communication policies within and outside of the organization
Transportation Systems	Type of response/transport vehicles available
	Number of response/transport vehicles available
	Management of vehicles
	Policies governing transportation systems
Financial Systems	Policies governing financial systems
	Persons responsible for financial management
Information and Documentation Systems	Information and documentation of clinical data
	Information and documentation of other [non-clinical] data
Quality Assurance/Evaluation Systems and Clinical Governance	Quality assurance/evaluation
	Clinical governance
Systems for Integration of Health Services	None identified
Equipment and Supplies	Procurement of equipment and supplies
	Management of equipment and supplies
Legislation	None identified
Medical Direction	None identified
Research	None identified
Education Systems	Existence of educational systems within the organization
	Funding of educational activities
	Employee training and development

The level descriptors for each component, and where applicable, the sub-components were presented to the 10 remaining participants from phase 1 of whom 9 completed phase 2. After each round, adjustments were made to the level descriptors based on the expert-feedback until consensus on the level descriptors was achieved. Three rounds were required to achieve consensus on all level descriptors. The agreed-upon components and their level descriptors from phases 1 and 2 were used to develop the final benchmarking tool. A scoring system was added, along with a set of instructions for its use. A summarised version of the final benchmarking tool is shown below in Table 2, followed by a description of how to use it. In Table 2 the narrative under level descriptor progression uses the embedded labels of LD1 [under-developed], LD2 [developing] and LD3 [well-developed].

Table 2.

Consensus-based capability benchmarking tool for EMS organizations in South Africa.

Component	Sub-component	Level descriptor progression
Human Resources/Assets	Role specialization of personnel	Role specialization ranged from organizations with only operational staff who are responsible for all functions [LD1] through organizations with mainly operational staff and some non-operational staff, and some degree of role specialization [LD2] to organizations with operational and non-operational staff having a high degree of role specialization – operational staff only perform operational functions [LD3].
	Emergency care qualifications of operational personnel	Emergency care qualifications ranged from organizations with only non-NQF-aligned qualifications in a mix of BLS and ILS levels of care and thus no resources for clinical escalation [LD1] through organizations with varying degrees of NQF-aligned and non-NQF aligned qualifications at all levels of care and limited resources for clinical escalation [LD2] to organizations with a majority of NQF- aligned qualifications and sufficient resources for clinical escalation unconstrained by time of day, call volume, or geographic location [LD3].
Communication Systems	Consistency of communication systems within the organization	Consistency of communication systems ranged from organizations that have no standardized communication system, within or outside of the organization [LD1] through organizations that have standardized communication system within the organization but not outside of the organization [LD2] to organizations that have one integrated and standardized communication system that allows for communication both within and outside the organization [LD3].
	Use of technology for operational communication	Use of technology for operational communication ranges from organizations that have no specially designed technology within its communication systems [LD1] through organizations that use specially designed technology for operational communication but that technology is not integrated with any other system [such as Computer Aided Dispatch system] [LD2] to organizations that use specially designed technology for operational communication and that technology is partially or fully integrated with a Computer Aided Dispatch system [LD3].
	Communication policies within and outside of the organization	Communication policies range from organizations that do not have any policies, rules, regulations or standard operating procedures governing communication inside or outside the organization [LD1] through organizations that have some informal rules, regulations and/or operating procedures governing communication within the organization and possibly outside the organization [LD2] to organizations that have formal, written policies, rules, regulations and standard operating procedures relating to all communication within and outside of the organization [LD3].
Transportation Systems	Type of response/transport vehicles available	Type of response/transport vehicles available ranges from organizations that have only one type of response/transport vehicle available within the organization, that is usually not suited to treating high-priority patients, and the organization does not have access to specialized medical or non-medical vehicles [LD1] through organizations that have more than one type of response/transport vehicle available within the organization, and the organization can transport any priority patient. Additionally, this organizational level may have their own specialized medical and/or non-medical vehicles or they may have working relationships with other organizations that can provide such vehicles [LD2] to organizations that have more than one type of response/transport vehicle available within the organization, and the organization can transport any priority patient at any given time. This organizational level also has its own its specialized medical and/or non-medical vehicles [LD3].
	Number of response/transport vehicles available	Number of response/transport vehicles available ranges from organizations where the number of functional vehicles available is solely dependent on the organization's resources and is typically small [LD1] through organizations where the number of functional vehicles available is dependent on the resources of the organization, but an attempt is made to consider the size of the population, the geographic area, or an identified demand/call volume that the organization intends to service [LD2] to organizations where the organization operates with the primary intention of tailoring the number of functional vehicles available to the size of the population, the geographic area, or the identified demand/call volume that the organization intends to service [LD3].
	Management of vehicles	Management of vehicles ranges from organizations which have one person or a small number of people, who likely have no specialized training, and also have other responsibilities managing the organization's vehicles [LD1] through organizations that have one person or a small number of people with some specialized training managing the organization's vehicles [LD2] to organizations that have a professional fleet management division [either their own or outsourced] responsible for managing the organization's vehicles [LD3].
	Policies governing transportation systems	Policies governing transportation systems ranges from organizations that have no formal policies, rules and regulations, or standard operating procedures governing the management and use of the organization's vehicles [LD1] through organizations that have informal, basic policies, rules and regulations, or standard operating procedures governing the management and use of the organization's vehicles [LD2] to organizations that have formal and comprehensive policies, rules and regulations, or standard operating procedures governing the management and use of the organization's vehicles [LD3].
Financial Systems	Policies governing financial systems	Policies governing financial systems ranges from organizations that align with the relevant finance-related legislation, but have no policies or standard operating procedures of their own [LD1] through organizations that align with the relevant finance-related legislation and have some basic policies or standard operating procedures of their own [LD2] to organizations that align with the relevant finance-related legislation and laws, and have comprehensive policies or standard operating procedures of their own [LD3].
	Persons responsible for financial management	Persons responsible for financial management ranges from organizations that have one person who likely has no formal specialized training managing the organization's finances [LD1] through organizations that have at least one person [but possibly more] with formal specialized training managing the organization's finances [LD2] to organizations that have has a team of individuals, with at least one manager who has formal specialized training managing the organization's finances [LD3].
Information and Documentation Systems	Information and documentation of clinical data	Information and documentation of clinical data starts with organizations that have basic information and documentation systems guided by relevant legislation, that is paper-based and whose data is used purely for record-keeping and billing purposes [LD1]. It continues with organizations that have information and documentation systems that incorporate some technology and align with relevant legislation whose data is used for record-keeping purposes, billing purposes and occasionally, or to a limited extent, also for quality-assurance purposes [LD2]. Finally, it ranges to organizations that have comprehensive, advanced information and documentation systems, which make use of current, reliable technology and that are secure and aligned with relevant legislation whose data is used for record-keeping purposes, billing purposes, and quality-assurance and research purposes [LD3].
	Information and documentation of other [non-clinical] data	Information and documentation of other [non-clinical] data ranges from organizations that have basic information and documentation systems that are guided by relevant legislation and are paper-based [LD1] through organizations that have basic information and documentation systems that are aligned with relevant legislation and incorporate some technology [LD2] to organizations that have comprehensive, advanced information and documentation systems that are aligned with relevant legislation, and which make use of current, reliable technology [LD3].
Quality Assurance/Evaluation Systems and Clinical Governance	Quality assurance/evaluation	Quality assurance/evaluation ranges from organizations that have no well-established quality assurance and evaluation systems in place and no regular auditing taking place, either internally or from outside the organization [LD1] through organizations that have basic quality assurance and evaluation systems in place and may have occasional internal auditing taking place [LD2] to organizations that have comprehensive quality assurance and evaluation systems in place that allow for relevant and meaningful quality improvement and that have regular internal and external auditing taking place [LD3].
	Clinical governance	Clinical governance ranges from organizations that have no clear clinical governance framework within the organization, but a supervising medical practitioner [with or without extensive expertise] may be contracted out to assist with medical oversight on an ad hoc basis [LD1] through organizations that have a basic clinical governance framework in place to ensure patient safety usually consisting of only one or two people [with or without extensive expertise] [LD2] to organizations that have a comprehensive clinical governance framework that ensures clinical effectiveness and patient safety and that involves multiple people [each with relevant expertise and experience] [LD3].
Systems for Integration of Health Services		Systems for Integration of Health Services ranges from organizations that have no systems in place to promote integrated healthcare within the community in which it operates [LD1] through organizations that have has some systems in place to promote the integration of healthcare, usually limited to receiving facilities and other EMS services [LD2] to organizations that have comprehensive systems in place that promote good integration with all other necessary role players, including but not limited to receiving facilities, other EMS organizations, fire and rescue services, and social services [LD3].
Equipment and Supplies	Procurement of equipment and supplies	Procurement of equipment and supplies ranges from organizations that do not have systems in place for the procurement of equipment and supplies and that have no formal budget for the procurement of equipment and supplies which is done on an ad hoc basis [LD1] through organizations that have basic systems in place for the procurement of equipment and supplies and a formal budget for large equipment, with supplies being bought on an ad hoc basis [LD2] to organizations that have comprehensive and intricate systems in place for the procurement of equipment and supplies and that have formal budgets for all procurement needs [LD3].
	Management of equipment and supplies	Management of equipment and supplies begins with organizations that do not have systems in place for inventory management of the equipment and supplies, with one person [with or without specific training/ expertise] responsible for managing equipment and supplies and with no centralized control or management [LD1]. It continues with organizations that have basic systems in place for inventory management of the equipment and supplies with at least one person with the necessary training and expertise being responsible for managing equipment and supplies and with central control over some large, expensive items [LD2]. Finally, it ranges to organizations that have comprehensive and intricate systems in place to manage equipment and supply inventories with multiple people with the necessary training and expertise being responsible for managing equipment and supplies and with centralized control for all the equipment [LD3].
Legislation		Legislation ranges from organizations that have one person [with or without expertise/training] ensuring that it adheres to relevant legislation [LD1] through organizations that have [or consult with] at least one person with legal qualifications and expertise ensuring that the organization adheres to relevant legislation [LD2] to organizations that have [or consult with] multiple people with legal qualifications and expertise ensuring that the organization adheres to relevant legislation [LD3].
Medical Direction		Medical Direction ranges from organizations that have one medical practitioner who is contracted to assist in providing medical direction and whose availability may be limited by the time of day [LD1] through organizations that have at least one medical practitioner available to provide medical direction at all times of the day [LD2] to organizations that have a comprehensive system in place with many medical practitioners available to provide medical direction in different areas of expertise at all times of day [LD3].
Research		Research ranges from organizations that have no systems in place for or policies governing the gathering of data or the conducting and publishing of research, and that do not have any collaborative agreements with higher education institutions or other institutions regarding research [LD1] through organizations that have secure and reliable methods of recording and storing data that can be used for research but that are not directly involved in conducting or publishing research. These organizations may have their own policies in place regarding research and may have some collaborative agreements with higher education institutions or other institutions regarding research [LD2]. Finally, it ranges to organizations that have secure and reliable methods of recording and storing data that can be used for research, both clinical and systems-based. These organizations have comprehensive policies in place regarding research and in addition to specific policies, the organization has its own research committee that is involved in and oversees all research-related activities within the organization. These organizations also have collaborative agreements with higher education institutions and other relevant institutions for conducting and publishing research [LD3].
Education Systems	Existence of educational systems within the organization	Existence of educational systems within the organization ranges from organizations that have no formal education systems in place for training and educating its employees [LD1] through organizations that offer only mandatory training and education for its employees [LD2] to organizations that have well-established education systems that allow for both mandatory and voluntary training and education to improve the skills and knowledge of its employees [LD3].
	Funding of educational activities	Funding of educational activities ranges from organizations that offer no funding to allow employees to participate in training and education outside the organization [LD1] to organizations that may fund further training and education to allow employees to participate in training and education outside the organization, and applications for such funding are usually unstructured and ad hoc in nature [LD2] to organizations that have budgets to fund further training and education to allow employees to partake in training and education outside the organization, and there is a structured application and selection process for such funding [LD3].
	Employee training and development	Employee training and development ranges from organizations that are not involved in assisting employees to obtain the necessary CPD points as per the HPCSA [LD1] through organizations that may assist employees in obtaining the necessary CPD points as per the HPCSA by informing them of relevant activities outside the organization [LD2] to organizations that are actively involved in assisting employees to obtain the necessary CPD points as per the HPCSA and that offer their own relevant activities as well as inform employees of relevant activities outside the organization [LD3].

BLS = Basic Life Support, Intermediate Life Support, NQF = South African national Qualifications Framework, CPD = Continuous Professional Development, HPCSA = Health Professions Council of South Africa.

Using the tool

The above tool consists of 24 components and sub-components of an EMS organization, each with its own set of level descriptors. An organization wanting to use this tool would examine each component and sub-component and decide, for each one, which of the three provided level descriptors [LDs] best describes their organization. LDs are scored according to their level [one point for LD1, two for LD2 etc.] Once the most accurate level descriptor has been chosen for all components and sub-components, the scores obtained are added to produce a final score.

Final scores in the lower range of 24–36 suggest an underdeveloped EMS, scores in the middle range of 37–60 suggest a developing EMS and scores in the upper range of 61–72 suggest a well-developed EMS.

It is important to note that the total score is an indicator of the overall level of development of the organization's capabilities, but it is possible, and perhaps likely, that an organization may score higher or lower in certain areas compared to its overall score. For example, an organization may find that its overall score suggests that it is a developing organization, but it may be better developed in certain components or sub-components. These could then be considered areas of excellence for the organization, while the lower-scoring components or sub-components may identify areas for improvement. It is also important to note that this tool is not intended to measure the quality of the organization or the services it renders. It solely suggests an organization's capabilities. Overall, this tool identifies the core components of an EMS organization, provides standardized terminology, and describes what the capabilities of an EMS organization would be, for each component, if the organization is considered underdeveloped, developing, or well-developed. These are simply the necessary first steps in a complex process of eventually being able to meaningfully assess the quality of EMS organizations in the South African context.

Discussion

Key findings

While the results showed that there are considerably differing expert opinions regarding certain aspects of EMS in South Africa, this is not the case with the core components of an EMS organization. It is apparent that the expert participants in this study shared a fairly common understanding of the core components of an EMS organization in South Africa. Experts also seemed to have shared views on several of the individual components, including financial systems, information and documentation systems, quality assurance and evaluation systems and clinical governance, systems for integration of health services, equipment and supplies, medical direction, research, and education systems.

These shared perspectives however were not seen throughout. Experts had quite differing views on human resources, particularly regarding the emergency care qualifications of operational personnel. This is perhaps unsurprising given the historical contention around EMS qualifications in South Africa, particularly following the release of the National Emergency Care Education and Training [NECET] policy in 2017 [11]. There were also contrasting opinions on transport, particularly regarding the number of response and transport vehicles available. These differing views appeared to stem from disagreements regarding how resourcing should be done. In other words, it appears that the experts had different ideas on whether the number of vehicles in an organization should be tailored to population size, geographical location and size, or call volume. One possible explanation for this may be South Africa's two-tiered health care system comprising of the private and public sectors which ultimately have different goals and operations. [12]. Finally, there were opposing views on communication systems, particularly regarding the use of technology for operational communication. This may again speak to differences in the operations of the private vs the public sector, but may also speak to socio-economic challenges in South Africa and subsequent unequal access to information and communication technologies [13,14].

Filling the gaps

Since the birth of EMS in South Africa, specific and relevant research related to the pre-hospital environment has been limited, with most of the current principles and practices derived from in-hospital research [15]. Furthermore, studies directly related to and originating from EMS in South Africa and other low- and middle-income countries are even more limited [16], [17], [18]. While research from high-income countries has been valuable in developing South African EMS, it is vital to focus on growing a body of evidence-based literature that is specific to the country's distinctive needs and abilities [16], [17], [18]. While this tool does not directly measure quality, it does address certain perceived gaps that need to be filled prior to tackling the complex concepts of quality measurement and quality improvement in EMS in South Africa. Aside from the actual tool, some additional outputs from this research include:

• •Presenting a holistic, consensus-based list of the components of an EMS organization
• •Providing standardized terminology that is relevant to the South African EMS environment.
• •Describing what different levels of capability would look like for each component of an EMS organization specific to the South African context.
• •Identifying where further research is needed and how this study can be used as a first building block in the complex process of meaningfully assessing the quality of EMS organizations and the services they render in South Africa

Uses of the benchmarking tool

The intention is that this capability benchmarking tool's primary purpose is self-evaluation and self-improvement, much like the Capability Maturity Model [CMM] used in software development [19]. In other words, it was envisioned that EMS organizations operating in South Africa might use this tool to assess and evaluate their overall level of capability and identify areas of excellence and areas requiring improvement within their organization. Beyond self-evaluation, the process of benchmarking also has a competitive aspect [5]. This tool could therefore also be used by EMS organizations to compare their capabilities to those of other EMS organizations in an effort to become ‘the best’ or better than their competitors. It is important to note that, in its current form, this tool weights all components equally and does not rank components [or subcomponents] in any order of priority. Although there is no weighting of components, we believe that those of a more fundamental importance [for example, transportation systems and its subcomponents] are identified as such through their level descriptors which, although differentiating between levels, do recognise the foundational relevance at all levels. In contrast, an element such as research which may not be considered to be of the same fundamental importance only features in the developing and well-developed levels and mainly in the latter.

Additionally, this tool may also be used by individuals or businesses seeking the services of an EMS organization in South Africa, to assess which organization would best suit the needs of the individual or business. In other words, in the event of a tender, this tool could be used to assess the capabilities of each EMS organization in the running. This may assist in ensuring the tender is awarded to an organization with the required level of capability for the task at hand. For example, a large sporting event, such as the Rugby World Cup, would require the services of an EMS organization. Whether the organisers of this event were local or foreign, they may use such a tool when assessing the capability of different organizations that might have applied for the tender. There is currently no other tool in South Africa that offers a framework for objective evaluation of EMS capability in this way.

Limitations

This study has some limitations that should be mentioned. The first limitation relates to the number of participants that ultimately took part in the study. While 12 participants provided their informed consent, not all 12 participants responded to each round of the Delphi surveys. Ten participants took part in Phase 1, and nine participants took part in Phase 2. We specified in advance a minimum of eight participants based on published Delphi method guidance [8] and slightly exceeded this although we acknowledge that a larger number of participants may have produced additional insights. While the objectives of this research did not include engaging in any type of knowledge synthesis, it could be argued that conducting the literature review that informed phases 1 and 2 in a more rigorous way [such as, for example, in the form of a scoping review] would have reduced possible bias in literature selection.

Further research

Since this research produced a capability benchmarking tool that is relevant to the context of South African EMS, it is recommended that further research be conducted to assess the tool's implementation within different EMS organizations in the country. It is recommended that the tool is applied in all the ways mentioned above to assess its use and effectiveness in different scenarios. Furthermore, it is recommended that this research is used as a stepping-stone for additional research into meaningful quality improvement through the investigation of robust quality indicators relevant to emergency medical care in South Africa.

Conclusion

This study produced a capability benchmarking tool for use in South Africa through a consensus-based approach using a panel of EMS experts. It constitutes a first step in the complex process of evaluating and striving to improve quality in EMS. While the tool does not address quality directly, it offers a framework to evaluate and benchmark EMS capability as a first step in this direction. It may also be a useful frame of reference in quality-related policy development by EMS organizations themselves and by those who may want to evaluate their capabilities more objectively in the future.

Dissemination of results

This research was used to produce a dissertation that was submitted to the University of Johannesburg in fulfilment of the degree: Master of Emergency Medical Care. This research has not been published or presented in any other form.

Authors contribution

Authors contributed as follows to the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content: AVL 70%, CS 30%. All authors approved the version to be published and agreed to be accountable for all aspects of the work.

CRediT authorship contribution statement

Ashleigh Vincent-Lambert: Conceptualization, Funding acquisition, Formal analysis, Data curation, Writing – original draft, Writing – review & editing. Christopher Stein: Conceptualization, Funding acquisition, Formal analysis, Data curation, Writing – original draft, Writing – review & editing.

Declaration of competing interest

CS is Chairperson of the Research Ethics Committee that gave ethical approval for this research. This conflict was declared at the time of protocol review, and he was not involved in the decision to approve this study. The authors declared no further conflict of interest.