Audit of the Sydney Local Health District Public Health Unit notification and contact tracing system during the first wave of COVID‐19

Abstract

Objective

To conduct a real‐time audit to assess a Continuous Quality Improvement (CQI) activity to improve the quality of public health data in the Sydney Local Health District (SLHD) Public Health Unit during the first wave of COVID‐19.

Methods

A real‐time audit of the Notifiable Conditions Information Management System was conducted for positive cases of COVID‐19 and their close contacts from SLHD. After recording missing and inaccurate data, the audit team then corrected the data. Multivariable regression models were used to look for associations with workload and time.

Results

A total of 293 cases were audited. Variables measuring completeness were associated with improvement over time (p<0.0001), whereas those measuring accuracy reduced with increased workload (p=0.0003). In addition, the audit team achieved 100% data quality by correcting data.

Conclusion

Utilising a team, separate from operational staff, to conduct a real‐time audit of data quality is an efficient and effective way of improving epidemiological data.

Implications for public health

Implementation of CQI in a public health unit can improve data quality during times of stress. Auditing teams can also act as an intervention in their own right to achieve high‐quality data at minimal cost. Together, this can result in timely and high‐quality public health data.

SARS‐COV‐2, the virus that causes COVID‐19, was first reported in December 2019 in China.1, 2 It has since spread around the world, with the first case reported in Australia on 25 January 2020.³ Those at increased risk of contracting the virus and developing the disease include international travellers, people in contact with COVID‐19 positive cases, older people with comorbidities, people in aged care facilities, and those in correctional and detention facilities.4, 5, 6 In countries where the healthcare system has been inundated with cases, the population mortality rate has been as high as 7.2%.⁷ Australia recorded a mortality rate of 3.2% up to 6 December 2020, with 28,049 cases and 908 deaths.⁸

Due to the potential for the rapid spread of COVID‐19 and high mortality rates in at‐risk groups, public health measures have been implemented around the world to control the transmission of SARS‐COV‐2. These measures include education, social distancing and restrictions, cancellation of mass gatherings and isolation or quarantine of at‐risk or positive individuals.9, 10 These measures are informed by surveillance data, which is the systematic and continuous collection, analysis, interpretation and notification of health‐related data to prevent and control health problems.¹¹ Surveillance data help describe the burden of the disease, monitor trends and provide data for setting priorities, designing, executing and evaluating programs and policies.12, 13, 14, 15, 16 It is crucial that surveillance data are high quality, timely, simple to understand and representative of the population.17, 18

In New South Wales (NSW), medical practitioners and pathology laboratories are required to report positive cases of COVID‐19 under the Public Health Act 2010 (NSW). As per NSW health guidelines, local public health units have the responsibility to follow up on all cases of COVID‐19 and record this information in the state's Notifiable Condition Information Management System (NCIMS).¹⁰ NCIMS is a secure, confidential, online database used to collect public health surveillance data in NSW. It contains information on patient demographic characteristics, laboratory results, clinical symptoms, medical management, likely source of infection, close contacts and the outcome of the disease. NCIMS is continuously updated in real‐time with information from public health staff across the state, the Ministry of Health and laboratories.

In March 2020, there was a rapid increase in the number of COVID‐19 cases in the Sydney Local Health District (SLHD). In response to the increased workload of the district's public health unit (PHU), 73 surge staff were seconded to the PHU in March and remained there for a period of 3–4 months. This increased the number of staff working at the PHU from 58 to 131. These surge staff came from the district's population health, community health and planning teams. The majority of surge staff had no prior training in communicable disease control, case investigation or contact tracing. PHU staff were responsible for supervising and training the surge staff in interviewing, contact tracing, managing close contacts, data collection and the use of NCIMS. PHU staff were also rostered to work on‐call after business hours (24/7), in addition to their usual work hours.

It was recognised from the outset that the rapid escalation in activity, combined with changes in the workforce members and responsibilities, could adversely affect the quality of public health data. To initially address this issue, the PHU COVID‐19 operations team ran a comprehensive staff training program that included completing HETI (Health Education and Training Institute) modules regarding emergency management, as well as surveillance and outbreak management, contact tracing videos, videos on how to use NCIMS, a review of Communicable Disease Network Australia (CDNA) national guidelines and the PHU‐specific standard operating procedures, Zoom tutorials and full case simulation. The training program went through phases of the PDSA (Plan Do Study Act) cycle and constant real‐time feedback from the staff and the audit team was incorporated.

In addition to the staff training program, and to ensure that the epidemiological data were of high quality and timely, a continuous quality improvement (CQI) process was commenced.17, 18 Shortell's CQI framework was applied to CQI processes as shown in Figure 1 .¹⁹ Shortell's CQI framework is comprised of four dimensions: strategic, cultural, technical and structural. The strategic dimension includes those activities and processes that are most important to the organisation and provide the greatest opportunity for improvement, such as vision, budget priorities and long‐term strategy. The cultural dimension represents the organisation's ‘beliefs, values, norms and behaviours’ that support or inhibit the CQI work. The technical dimension encompasses training and information infrastructure. The structural dimension refers to the ways that knowledge is acquired and dispersed throughout the organisation.

Figure 1.

Four dimensions of Shortell's CQI framework¹⁹ applied to the SLHD PHU.

The CQI was centred on daily handover meetings that included discussion of cases, management issues, dissemination of decisions and review of processes. Evidence‐based presentations on topics of interest were also included. These meetings were instrumental in identifying and understanding changes in guidelines and COVID‐19 operational processes and discussing the implications – implementing rapid changes in processes in the unit. To guide and evaluate changes and the performance of the unit, a real‐time audit was initiated. The audit was conducted by a team staffed separately to the core COVID‐19 operations team, which comprised of two supervisors (a clinical director and epidemiologist) and four other staff (an advanced paediatric trainee, clinical nurse consultant, public health unit trained personnel and registered nurse). In addition to assessing the quality of the data, any information that was identified as missing or incorrect was investigated by the audit team and rectified.

Methods

The audit was performed at the SLHD PHU, located at Royal Prince Alfred Hospital. All COVID‐19 cases managed by the SLHD PHU during the first wave (up until 23 May 2020) were included in the audit. These COVID‐19 cases resided within the SLHD, were quarantined in hotels located within the SLHD (including overseas travellers) and had most of their follow up done by the SLHD PHU. COVID‐19 cases were excluded if they were diagnosed after death, were subsequently found to have a false‐positive result, or their follow up was done by another local health district.

The following variables were audited for completeness: confirmed COVID‐19 case status; Aboriginal and Torres Strait Islander status; high‐risk occupation symptom onset date; clinical symptoms; linking (source of infection and their close contacts); hospitalisation status; outcome; and documentation (case questionnaires, contact tracing sheets, email communications regarding the case were attached). Accuracy of data was audited for variables: place of acquisition and source of infection. Missing data and accuracy were verified by comparing and reviewing: the COVID‐19 case questionnaire and contact tracing tables and summaries; the communicable disease notifications in NCIMS; email communication between the communicable disease team, Ministry of Health, and external agencies; SLHD patient electronic health records; and other electronic information stored by the PHU staff regarding the team's operations in the PHU shared drive and team emails. Where information remained missing or ambiguous, the relevant staff were contacted to clarify the information and cases were re‐interviewed as required to improve the completeness and accuracy of the data.

Aggregate ordinal variables for overall data quality, completeness, and accuracy were calculated for each case by summing the relevant variables listed above. A measure of PHU workload was also calculated by summing the number of other cases that were reported within three days of each case.

Individual logistic regression models were generated for each audit variable, and ordinal logistic regression models were generated for each aggregate variable. The only covariates included in each model were workload and date. These were both normalised using the Ordered Quantile (ORQ) transformation. Data were analysed and visualised using the statistical package, R version 4.0.3.²⁰

The audit was approved by the SLHD Human Research Ethics Committee (reference LNR X20‐0266).

Results

There were 304 COVID‐19 cases reported to the SLHD PHU from 3 March 2020 to 23 May 2020 (82 days or 12 weeks). The first case was audited on 24 March 2020 and the last case on 12 June 2020. Of the 304 cases, 11 were excluded from this study for the following reasons: one case had an indeterminate result followed by two negative results and was declared a false positive; three cases had a positive result by RT‐PCR with a subsequent negative serological result and declared to be false positives; one case was diagnosed after death; five cases were excluded due to cross‐jurisdictional issues as most of their follow up was done by other local health districts; and one case was lost to follow up after leaving the country. This left 293 COVID‐19 cases that were included in the audit.

There were 52 cases (17.7%) reported in the first quarter of the period audited, followed by a sharp increase to 217 cases (74.1%) in the second quarter, and 24 cases (8.2%) audited in the final two quarters (Table 1 ). The number of cases that were re‐interviewed by the audit team due to unknown source of infection was 41 (14.0%).

Table 1.

Completeness and accuracy of audited variables for each quarter. Numbers shown are the percentage of cases.

Audited variable	March 3 to March 22 n=52		March 23 to April 12 n=217		April 13 to May 2 n=19		May 3 to May 23 n=5
	Audit	After audit	Audit	After audit	Audit	After audit	Audit	After audit
Data Completeness
Aboriginal and Torres Strait Islander status	94.2	100	93.1	100	100	100	100	100
High‐risk occupation	76.9	100	78.3	100	89.5	100	100	100
Symptom onset date	96.2	100	97.2	100	100	100	100	100
Clinical symptoms	73.1	100	83.9	100	84.2	100	80.0	100
Linking	57.7	100	73.3	100	100	100	100	100
Hospitalisation status	94.2	100	89.4	100	100	100	100	100
Outcome	76.9	100	86.6	100	100	100	100	100
Documentation	75.0	100	89.9	100	100	100	60.0	100
Data Accuracy
Place of acquisition	98.1	100	89.9	100	100	100	100	100
Source of infection	96.2	100	88.5	100	100	100	80.0	100

Overall, there was an improvement in data quality over time, even after adjusting for workload (p<0.0001, ordinal logistic regression). On further examination, variables measuring completeness were associated with improvement over time (p<0.0001, ordinal logistic regression); whereas those measuring accuracy were inversely associated with increased workload (p=0.0003, ordinal logistic regression). Significant associations with individual audit variables are also shown in Figure 2 . Of note, completeness of linking was the most strongly associated with improvement over time (p=0.0001, logistic regression).

Figure 2.

Smoothed averages for the completeness and accuracy of each data quality variable measured by the audit (solid blue line), and after being corrected by the audit and PHU team (dashed solid line). Histograms show the number of cases over time.

Notes:

∗Significant association with date (p<0.05, logistic regression).

^Significant association with workload (p<0.05, logistic regression).

After assessing the data for completeness and accuracy, the audit and public health teams corrected the data, resulting in 100% complete and accurate data (Table 1, the dashed green line in Figure 2).

Discussion

Our study confirms the challenges of achieving high‐quality recorded data in a public health unit at the start of a pandemic. High‐quality data are required to give an accurate epidemiological picture of cases and contacts to allow for effective and timely decisions to be made regarding controlling the spread of disease. The major challenges faced by the PHU included increased workload on existing staff, changing roles and responsibilities of these staff, new staff not trained in communicable disease control or the existing systems, increased demand on information systems, and evolving guidelines. Realising the importance of accurate epidemiological data early in the pandemic and anticipating that there could be an issue with the quality of the data, a CQI process was employed early to address this. The audit confirmed that there were issues with initial data quality and allowed a measure of the effectiveness of interventions to improve the epidemiologic data that were guiding the public health response. For example, the recording of clinical symptoms was identified as low early in the pandemic because data was being collected manually by one team member who then handed it over for data entry by another team member who prioritised it as low. Information such as this was fed back to the team during the unit's daily meetings to help improve operational processes.

Not unexpectedly, our audit showed the lowest quality data at the beginning of the pandemic when these challenges were at their peak and when the number of reported cases was increasing rapidly. However, overall, there was an improvement in data quality over time, even after adjusting for the workload. This likely reflects changes due to quality processes like a comprehensive staff training program and ongoing support and supervision, leading to increased staff experience with the contact tracing and operational processes associated with the COVID‐19 pandemic. Interestingly, some variables were not associated with improvement over time but were inversely associated with the workload. Variables measuring accuracy (place of acquisition and source of infection) worsened with the increased workload, whereas variables measuring completeness were generally associated with improvement over time. This suggests that accuracy is a better measure of stress, while completeness is a better measure of improvement. Of note, the variables measuring linking and clinical symptoms improved over time and were not associated with the workload.

A major challenge for the PHU was the shortage of skilled workforce trained in communicable disease control at the beginning of the pandemic. To limit the impact of this in the future, it would seem sensible that identified surge staff were trained in public health concepts and the local public health systems prior to the next wave of the pandemic.¹⁶ However, this may not be easy to achieve politically and may come at a high opportunistic cost during normal business. Using the PDSA model, the extensive staff training program developed at the PHU helped to train the surge staff continuously to a high standard in COVID‐19 operations. The staff training program would likely meet the future need to train the surge staff if there were to be another wave of the pandemic. This evaluation is the subject of a separate paper. However, our study results demonstrate that in addition to this training, having an integrated but separate audit team at the beginning of a pandemic to actively review and correct data and maintain high‐quality data allows for appropriate delineation of staff roles and can ensure accurate surveillance data at a time of high activity. The audit team was able to achieve 100% complete and accurate data in near‐real‐time, with an investment of only 4% of the unit's staff.

Limitations

There were some limitations to this study. The audit covered a period of unusual activity for a PHU, with a lot of uncertainty, rapidly changing information, and staff without a public health background working for the first time in a public health unit. Therefore, the findings from this study may not be generalisable to times with more normal activity. The study did not report on timeliness, which is an important aspect of data quality. In addition, the audit only reviewed data within one local health district of NSW. While other health districts in NSW all use the same database system to record notifiable diseases such as COVID‐19 according to the same national guidelines, this information was not analysed. Therefore, our findings may not reflect the quality of data in other districts. However, the audit does provide a reference to compare other NSW local health districts. The audit was also unable to differentiate between the quality of the data entered by the PHU staff versus the surge staff, as many staff members were involved at various stages of contact tracing and data entry.

Recommendations

CQI processes can result in improved data quality during times of rapid growth and evolving knowledge. The effectiveness of these interventions can be monitored and guided by real‐time auditing. Data identified as incomplete or inaccurate from a real‐time audit can be corrected with minimal resources, resulting in high quality and timely data.

Conclusion

The SLHD PHU was able to provide high‐quality contact tracing services and good quality public health data on COVID‐19 cases in challenging times due to early deployment of surge staff with a comprehensive training program in COVID‐19 operational tasks, implementation of CQI processes guided by real‐time auditing, and an auditing team that actively corrected missing and inaccurate data. The CQI processes and the real‐time audit team enabled us to make more timely decisions regarding case management and contact tracing, contributing to the state‐wide reduction in COVID‐19 cases.

Acknowledgements

We are indebted to the staff at the public health unit and the surge staff from the population health, community health, and planning teams who were seconded to assist in the notification and tracing of patients. We extend our thanks to Dr Penelope Fotheringham for helping prepare the ethics protocol.