A novel measure for characterizing ultrasound device use and wear

Abstract

Point‐of‐care ultrasound (POCUS) equipment management is critical in optimizing daily clinical operations in emergency departments (EDs). Traditional consultative ultrasound laboratories are well practiced at operations management, but this is not the case for POCUS programs, because machine upgrade and replacement metrics have not been developed or tested. We present a data‐driven method for assessment of POCUS equipment maintenance and replacement named the ULTrA (a data‐driven approach to point‐of‐care ultrasound upgrade) score. This novel model of assessing each ultrasound machine by quantitative scoring in each of four mostly objective categories: use (U), likeability (L), trustworthiness (Tr), and age (A). We propose the ULTrA model as a method to identify underperforming devices which could be upgraded or eliminated, and to compare relative performance amongst a group of departmental ultrasound machines. This composite score may be a useful objective tool that could replace individual proxies for clinical effectiveness, such as age, use, or individual provider preference. Additional research in multiple centers would be needed to refine and validate the ULTrA score. Once fully developed, the ULTrA score could be deployed in EDs and other clinical settings where POCUS is used to help streamline resources to maintain a functional and state‐of‐the‐art fleet of ultrasound machines over time.

1. INTRODUCTION

Emergency physicians have incorporated point‐of‐care ultrasound (POCUS) into routine clinical practice for decades. ¹ The ability to perform and interpret ultrasound at the bedside aids in timely diagnosis and treatment, making it an invaluable resource in the ED. ² , ³ , ⁴ Currently, the field of POCUS is undergoing a technological revolution as equipment capabilities rapidly evolve and prices for novel devices simultaneously fall. ⁵ Simultaneously, the number of most frequently used or core ultrasound applications are also expanding and more frequently include advanced functionality such as various Doppler modes, strain assessment, and other imaging adjuncts. To match the increasing demand for POCUS and remain at the cutting‐edge of emergency care, EDs have rapidly expanded their repertoire of ultrasound machines. Remaining technologically current in this era of rapid advancement requires proactively developing a systematic approach for optimization of all aspects of ultrasound operation, including documentation, image storage, quality assurance, machine acquisition, and machine maintenance. The latter processes of machine acquisition and maintenance present a major challenge for EDs trying to remain financially solvent.

Although abundant literature exists surrounding the use of POCUS in the ED, ⁶ formal recommendations for machine upgrade and maintenance have not yet been outlined. Nor has a systematic and objective methodology been described to decide when equipment requires maintenance, upgrade or replacement. Outdated, ineffective, and malfunctioning devices detract from quality patient care, may delay diagnosis and treatment, and create logistical complications that can consume departmental resources.

In the majority of EDs, ultrasound machines are primarily managed by clinical faculties who often have little or no technical expertise. Maintaining a machine requires frequent hardware repairs, software updates, and ultimately, retiring of ineffective or outdated devices to support implementation of new technologies. To meet this goal, ultrasound faculty must repeatedly assess the current status of the department's ultrasound machines and request capital support from the department for their maintenance, upgrades, and replacements. There is currently no structured means of performing this assessment. As a result, ineffective or defective machines may be left in clinical use too long, to the detriment of patient care.

A data‐driven, structured tool for proactively evaluating effectiveness of departmental ultrasound equipment machine performance would help support this need. A means of assessing machines would be beneficial for cost‐effective planning and for ensuring high‐quality patient care. An objective approach would apply to any setting where POCUS is used, such as in internal medicine, family medicine, critical care, and anesthesia, amongst others.

We describe the need for and concept of an objective model for proactive ultrasound replacement. We additionally propose a scoring system which leverages data gathered on ultrasound machines including clinical use rate, clinician preferences, maintenance record, and age. Armed with longitudinal information on machine performance and projected costs, POCUS leadership teams and financial stakeholders can strategically anticipate upgrade and replacement costs in any POCUS setting.

2. OUTCOMES

It is critical that a department's ultrasound machines are always current, functioning, available, and meeting clinicians’ and patients’ needs. An ideal scoring system would provide an objective means of assessing these machines and could provide both absolute and relative information about each individual machine.

The primary benefit of a machine scoring system is that it could provide information about how each machine is performing and is expected to further perform relative to other machines. The relative value of each machine could be assessed and could inform the obsolescence of lower value machines. As decisions about machine purchasing and need for replacement need to be made, a scoring system that identifies which specific machines have the poorest performance relative to the other machines can help in strategic planning. Additionally, a scoring system could have value in providing a way to serially quantify a single machine's performance over time, with falling scores suggesting a decrease in performance and ultimately a need for replacement.

A scoring system could also provide absolute predictive data about ultrasound machines. There are many possible endpoints that could be used but one practical endpoint would be prediction of failure or need for replacement within 6 months. In most situations this timeframe would be long enough to contact machine companies for support and inform departmental purchasing of capital. A 12‐month prediction tool could alternatively be useful.

3. COMPONENTS OF A SCORING SYSTEM

Numerous aspects need to be considered when developing a scoring system for assessment of ultrasound machines.

3.1. Age and life cycles of ultrasound machines

Throughout the life cycle of each POCUS machine, devices become increasingly susceptible to malfunction. Additionally, technological innovations can rapidly develop with cutting edge technology. Eventually, a machine transitions from benefiting to detracting from the patient care an institution can provide. Additionally, older machines are more difficult to repair as their spare parts are less available. The European Society of Radiology states that radiology equipment up to 5 years old is reflective of both the current state of technology and allows for reasonable upgrade measures, citing that at least 60% of equipment should exist within this “new” category. Somewhat older equipment, between 6 and 10 years old, is acceptable if properly maintained, and could comprise up to 30% of a department's machines. Finally, equipment older than 10 years requires replacement, and should encompass only 10% or less of a department's machines. ⁷

The Canadian Association of Emergency Physicians has published guidelines specifically addressing the purchase of POCUS machines in the ED. Their recommendations include necessary machine characteristics and suggested quantity of machines. ⁸ However, their analysis of machine features does not address specialization of certain machines for particular clinical purposes; nor does it account for the inevitable malfunction of machines and subsequent replacement plans.

3.2. Ultrasound preventative maintenance

Preventative maintenance is essential to keep POCUS equipment at peak performance. This requires the development of maintenance protocols but can be challenging as each manufacturer has their unique specifications for preventative maintenance. In fact, the Centers for Medicare and Medicaid Services ⁹ and American College of Radiology ¹⁰ have both suggested that preventative maintenance should be done in accordance with the original equipment manufacturer.

Preventative maintenance is usually secured by obtaining a service agreement with the POCUS equipment manufacturer. Alternatively, in‐house biomedical engineers can assist if they are trained appropriately. Most manufacturers offer a variety of plans with a range of prices depending on factors such as: number of and type of transducers, system type (configuration and software), patient volume, amount of preventative maintenance, and hours of coverage needed. ¹¹ When selecting the right plan for a department, it is important to assess the level of risk by evaluating the potential loss of revenue from downtime as well as the impact on patient safety.

Institutional policies and training on proactive machine care can help reduce the need for maintenance thereby decreasing the cost to their institution. The American College of Emergency Physicians issued a policy statement regarding the importance of ultrasound transducer cleaning and disinfection. ¹² Careful adherence to this policy is both beneficial for patients and for the longevity of machines. Machine care should also include visual inspection of transducers including cables, system diagnostics, system cleaning, system disassembly, and assembly assuring all filters and circuit boards are clean and assuring quality diagnostic acquisition and imaging.

Understanding how much maintenance is required for any particular machine is important. If there are frequent issues with probe malfunction, connectivity with wireless networks, battery life, storage or documentation of findings, or any other issues which may render the machine useless or suboptimal, these need to be taken into consideration. These factors all contribute to the reliability of the machine.

3.3. Ultrasound machine costs

The total cost of an ultrasound machine includes three distinct components: acquisition costs, routine maintenance costs, and repair costs. Acquisition refers to the one‐time fee paid to obtain a machine and its probes. Routine maintenance includes the expected costs required for the upkeep of a machine, such as cleaning components, updating firmware, and paying employees who manage the machines. Finally, repair costs include those fees that arise when a machine breaks unexpectedly and must be repaired.

For the majority of POCUS devices, a manufacturer warranty covers these costs. However, as a machine ages, its maintenance cost is expected to rise, and by the time machine issues are significant, the warranty has often lapsed. Therefore, in determining the cost‐effective lifespan of a machine, it is important to consider when and if the warranty has expired.

3.4. Sonologist preference

There are multiple reasons that a clinician who is performing POCUS may prefer to use one machine over another. These may include image quality, ease of use, advanced ultrasound features, or presence of a particular probe that may not be on all machines (eg, endocavitary or transesophageal). However, often such reasons are unknown and may include familiarity of the machine interface, proximity to the patient at the time the machine was needed, or availability of machine. Some of these are intangible and difficult to quantify. As a surrogate, one could consider that how often a machine is used may reflect its likeability.

4. THE ULTRA SCORE

Accounting for the above considerations, we propose the data‐driven approach to point‐of‐care ultrasound upgrade (ULTrA) score as a novel and data‐driven pseudo‐objective means of assessing ultrasound machines. There are four components of this score: use (U), likeability (L), trustworthiness (Tr), and age (A). Each component is scored from 1 to 5, with higher scores indicating better performance in each category. A machine could therefore achieve a score from 4 to 20.

4.1. Use (U)

The use (U) rate refers to the number of scans conducted during a certain time period. Specifically, it shows how intensively a machine is being used, serving as a proxy for clinical utility and provider machine preference. Machines used more frequently are deemed most useful, and therefore receive higher use scores. Absolute usage will vary significantly by practice setting and volume. As such, relative usage scores (machine usage as compared to other machines in the same department) should be used, with higher scores assigned to more frequently used machines. The use component serves as a surrogate for sonologist preference and captures many of the intangible aspects of why some machines are used more than others. To calculate the score, total scans done over a certain time period are counted, and by dividing this number by the total number of machines, one can calculate the number of scans that would be the “expected use” per machine. Subsequently, by dividing the number of scans done on a machine, the “actual use,” by the “expected use” a use ratio (UR) is calculated. If all machines are used equally, they would each have a UR of 1.0. Numbers higher or lower than this indicate greater or lesser usage respectively. Table 1 summarizes how the UR can be translated into points.

TABLE 1.

ULTrA scoring key

Score	Use (U)	Likeability (L)	Trust (Tr)	Age (A)
5	1.75 ≤ UR	Excellent	No reliability issues	≤3 y
4	1.25 ≤ UR < 1.75	Very good	Minor service issues	3–5 y
3	0.75 ≤ UR < 1.25	Good	Significant service issues	5–7 y
2	0.25 ≤ UR < 0.75	Fair	More time out of commission than in clinical use	7–9 y
1	UR <0.25	Poor	Unreliable (decommission or replace, repair cost >50% of device cost)	>9 y

ULTrA, a data‐driven approach to point‐of‐care ultrasound upgrade; UR, use ratio = (total number of scans on a machine) × (total number of machines)/(total number of scans on all machines).

4.2. Likeability (L)

The likeability (L) score for each ultrasound device was determined by averaging a 5‐point Likert scale rating provided by all ultrasound faculty members regarding overall comfort and familiarity (1‐5). Factors that may be considered when assessing likeability may be availability of probes, presence of advanced features, mobility of machine, or ease of documentation, for example. A grade of 5 is reserved for machines with excellent likeability and one for machines with poor likeability. Ultrasound faculty would likely be the greatest arbiters of likeability, being most familiar with nuances and details of ultrasound machines. One could consider incorporating the thoughts of all faculty, or at least some non‐ultrasound faculty. However, the preferences of all users would be indirectly captured by analyzing use, and so allowing expert users alone to determine likeability is reasonable. Table 1 summarizes how faculty ratings are considered.

4.3. Trustworthiness (Tr)

The machine trustworthiness was assessed by determining the number of incidents that required vendor or hospital biomedical engineering intervention. This was recorded by viewing all down‐time events for an ultrasound machine over the past year. Common examples of these events include software issues, battery failures, broken transducers, wireless connectivity, and other issues preventing capture of quality diagnostic images. Table 1 summarizes how trustworthiness issues are considered.

4.4. Age

Using a modification of current European guideline for the maximum reasonable age of an ultrasound machine (10 years), ⁷ we developed a 1–5 scoring scale: machines that were ≤3 years old received the best score (5), 3–5 years old (4), 5–7 years old (3), 7–9 years (2), and 9 years or older (1). Table 1 summarizes how age is translated into points.

The ULTrA score was not derived using standard decision rule techniques. Rather, it is a proposed model based on expert opinion that attempts to take into consideration objective and pseudo‐objective data. Statistical techniques and multi‐centered data would be needed to determine the relative weight of each variable in a final prediction tool. As such, the ULTrA score may require refinement and cannot currently be recommended for use as a definitive tool.

Our experience is that these categories are important determinants of machine quality. However, we do recognize that there is some overlap in these categories. For example, highly “likeable” machines may be used more often and therefore will also have a higher rate of use. Additionally, newer and younger machines may have fewer technological issues and as such have a higher trustworthiness. However, these correlations may not exist in all cases and as such considering them as potentially independent variables is a reasonable first approach.

4.5. Example—using the ULTrA score

The ULTrA score was developed to evaluate the ranking performance of ultrasound machines at an urban, university‐affiliated tertiary‐care ED with an annual patient volume of ≈120,000. At the time of this analysis, there were nine ultrasound machines in use in the ED. The ED consisted of five patient‐care areas divided based on medical needs and acuity including “Acute,” “Urgent,” “Evaluation,” “Fast Track,” and “Pediatrics.” POCUS machines were located within each patient care area. The patient care level of acuity from highest to lowest is Acute, Urgent, Evaluation, and Fast Track.

We tracked data relating to ultrasound use over a 1‐year study period. We obtained the total number of ultrasound scans for each ultrasound machine by querying Qpath (Telexy Healthcare, Maple Ridge, BC, Canada), our imaging archiving system. An inventory of current ultrasound machines was performed by ultrasound faculty. During this inventory, information collected included the manufacturer, model, location of primary use, and age (Table 2). A consensus group of four ultrasound faculty and three ultrasound fellows rated that the most relevant factors when assessing an ultrasound machine were its clinical use, likability, trustworthiness, and age. For each category, a machine could score a maximum of 5 points (best), and a minimum of 1 point (worst), based on specific criteria included in a grading key (Table 1).

TABLE 2.

Ultrasound machines in order of total ULTrA score

Ultrasound machines			Use (U)					Age (A)
Machine	Location	Model	No. of scans (total = 67,942)	UR	Use score	L score	Tr score	Age (y)	Age score	ULTrA score
1	Acute	A	29,320 (43%)	3.88	5	5	4	2	5	19
4	Eval	A	10,787 (16%)	1.43	4	5	5	4	4	18
8	FT	A	12,420 (18%)	1.65	4	5	5	7	2	16
5	Eval	C	9 (0%)	0	1	4	5	1	5	15
3	Acute	C	1037 (2%)	0.14	1	4	4	5	3	12
2	Acute	B	6060 (9%)	0.80	3	3	3	6	3	12
6	Urgent	B	5888 (9%)	0.78	3	3	3	6	3	12
7	Urgent	C	1936 (3%)	0.26	2	4	4	9	1	11
9	Pedi	C	485 (1%)	0.06	1	4	4	7	2	11

L, likeability; Tr, trustworthiness; ULTrA, a data‐driven approach to point‐of‐care ultrasound upgrade; UR, use ratio.

During the study, 67,942 scans were completed. Use for a single machine ranged from 43% of scans to <1%, with correlating URs from ranging from 0–3.88. Likability ranged from 3–5 (median = 4). Trustworthiness ranged from 3–5 (median = 4). Age ranged from 1–9 years (median = 6). ULTrA scores ranged from 11–19 (median = 12). Regardless of machine make/model, likability, trustworthiness, and age, use was highest for machines primarily located in high‐acuity areas of the adult ED when compared with low‐acuity areas. Data are summarized in Table 2.

5. DISCUSSION AND FUTURE DIRECTIONS

The ULTrA score is proposed as a way to implement and evaluate a potential methodology for prospective machine assessment and management. This novel model that incorporates factors including age, use rate, likability and down‐time period can provide an objective rank list of current ultrasound machines and assist in developing a data‐driven plan for ultrasound machine upgrade. Upkeep and replacement of machines can be quite expensive. A system that can help guide a department in terms of value can facilitate fiscal decisions regarding ultrasound machines. We propose the ULTrA score as a novel method to identify underperforming devices that may need upgrade or replacement.

As with any scoring model, there are certain limitations to the ULTrA score. We did not prospectively study how and if an ULTrA score correlates with machine longevity or need for replacement. We also did not study how our composite score compared with any of its individual components. It is possible that one of the components alone could predict 6‐ or 12‐month failure or be useful in scoring POCUS machines relative to each other. As mentioned above, it is also possible that the selected variables are not entirely independent. For example, machines which were deemed as “trustworthy” and “likable” by both ultrasound faculty and fellows were also more likely to be used for clinical imaging. Therefore, a machine that scored high in one of these categories often scored high in both. In simpler terms, clinicians chose to use those machines that they already found both user‐friendly and trustworthy. Use, for this reason, is as much a reflection of clinical effectiveness as it is a proxy for clinician preference. With this reasoning, a POCUS program could consider using only a machine's use to determine its clinical effectiveness. However, our data indicate that clinical use (on its own) is not an ideal predictor of performance, as poorly performing machines located in high‐acuity clinical settings are still used more frequently, and thus, a machine's poor performance may be obscured by high‐use, even though the machine is not likable, trustworthy, or technologically up to date. Therefore, departments must still consider replacing and upgrading machines that are highly used, as significant issues with usability and reliability may still exist.

Similarly, age alone is a poor predictor of an ultrasound machine's clinical use when compared to other characteristics, such as device type, use rate, likability, and trustworthiness. Therefore, while age is an important consideration when planning for machine upgrade and replacement, this finding holds that it must be evaluated in the context of other qualitative and quantitative features for optimal decisionmaking.

Because the ULTrA score was developed at an academic teaching institution with a high volume of patients, applicability in different settings is unknown and should be explored further. However, we feel that adaptation of this systematic approach is likely generalizable. One limitation by using this example was that we did not categorize the scans by their clinical purpose. Given that some machines are uniquely suited for certain functions (ie, ultrasound‐guided venous access vs. echocardiogram), lack of data categorized by use alone could overlook other important factors for future purchase. Additionally, the smaller, cheaper, and more portable “procedure‐friendly” devices used for procedure purposes are not always archived in Qpath, thus machines used primarily for procedures may artificially lower use scores given our method of evaluation.

We support the idea that additional research is needed to create and validate a prediction tool. A multi‐centered study looking at the suggested factors and validating or refuting them as valuable components would be needed. Generalizability would need to be considered, as every department's ultrasound equipment and needs differ. The ULTrA score could be customized to individual departmental needs, perhaps by adjusting cut‐off parameters of use rate or age. Evaluating this score in departments of different sizes, acuity, location, and patient population would add to its generalizability. Multi‐centered derivation of a score would require consideration of additional predictor variables, prospective gathering of information, selection of optimal outcome variables, weighting of predictor variables, and assessing interrater reliability of pseudo‐objective parameters such as likeability and trustworthiness.

6. CONCLUSION

The proposed ULTrA score is an objective tool for assessing ultrasound machines. By looking at four different criteria a score can be calculated, and the relative status and performance of machines determined. Such a model may inform decisions about ultrasound maintenance and replacement plans, thereby streamlining management of resources. Additional multi‐centered research is required in this field for further derivation of an accurate score. Comparing ULTrA score performance to outcomes such as machine failure or need for replacement over a specified time period would add to the score's predictive abilities and usefulness. If validated, this tool could be used in all POCUS settings and amended to best fit individual departments’ needs. But again, additional research would be needed on this conceptual framework before widespread use could be advocated.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Shokoohi H, Goldsmith A, Negishi K, et al. A novel measure for characterizing ultrasound device use and wear. JACEP Open. 2020;1:865–870. 10.1002/emp2.12051