Evaluation and comparison of the National Tuberculosis (TB) Surveillance System in Ireland before and after the introduction of the Computerised Electronic Reporting System (CIDR)

Abstract

We evaluated and compared the completeness, timeliness, simplicity, usefulness and flexibility between the former National Tuberculosis (TB) Surveillance System (NTBSS) and the newer Computerised Infectious Disease Reporting System (CIDR). Completeness was assessed by examining the field completion of key variables and median time from diagnosis to notification was calculated to evaluate timeliness. Differences between the two systems on completeness and timeliness were statistically assessed using χ² and Wilcoxon rank-sum test, respectively. An online questionnaire on simplicity, flexibility and usefulness was sent to key stakeholders. Time and diagnosis-related variables were more complete in NTBSS, while variables on drug susceptibility, HIV and laboratory tests were more complete in CIDR (P < 0.05). The median time notification interval increased significantly in CIDR (P < 0.05). Stakeholders thought that CIDR is simpler (37.5%), more useful (41.7%) and more flexible (29.2%) than NTBSS. This study demonstrated that CIDR did not improve data completeness and decreased timeliness of notification. Simplicity, usefulness and flexibility were improved but qualitative methods should be applied to further explore these results.

Introduction

Since 1950, there has been a decline in the number of tuberculosis (TB) notifications in Ireland. The rates have steadily decreased since 1991 from 18.2 to 6.6 per 100 000 population in 2015 [1]. The highest rates have been noticed in the Health Service Executive (HSE) areas in the east (HSE-E) and south (HSE-S) [1]. Males and persons aged 65 years and older have higher rates [1]. Migration patterns have affected TB rates in Ireland as foreign-born cases in 2015 accounted for 42.6% of notifications [1]. Two large TB outbreaks were reported in 2007 and 2011 [2, 3].

The World Health Organization (WHO) has set a goal to eliminate TB by 2050, which is defined as a TB incidence rate of less than one sputum smear-positive case per 1 000 000 population [4]. In Ireland, this would correspond to four to five new sputum smear-positive cases per year [4]. To assist in achieving that, an effective TB surveillance system and a good TB control programme should be in place.

In Ireland, enhanced TB surveillance data have been collected since 1998. In 2000, an Epi Info™/MS-Access-based national TB surveillance system was developed at the Health Protection Surveillance Centre (HPSC). This system was operated regionally and data were collected in paper format, entered into an Epi Info™ database. Electronic files were sent quarterly to HPSC where they were collated and validated in MS-Access. This system was called the National TB Surveillance System (NTBSS) [5]. Since 2011, the TB-enhanced surveillance system has been incorporated into the Computerised Infectious Disease Reporting System (CIDR), a web-based system on a secure network, which is available nationwide to the departments of public health, public hospital laboratories and HPSC and which aims to improve the surveillance of all notifiable diseases [6]. CIDR holds all case-based infectious diseases and integrated clinical and laboratory data [7, 8].

CIDR was used for TB surveillance to help improve data completeness, timeliness and flexibility for users but it was never evaluated. In addition, a comparison between NTBSS and CIDR has never been measured. There is very limited literature on the comparison between integrated electronic and non-centralised paper-based surveillance systems and most are not recent [9–11].

The aim of the current evaluation was to assess the Irish national electronic TB surveillance system in terms of completeness, timeliness, simplicity, flexibility and usefulness and compare it with the old non-centralised system (NTBSS) in order to identify the areas of improvement that would support interventions ultimately aimed at TB elimination in 2050.

Methods

We followed the guidelines for evaluating public health surveillance systems developed by the European Centre for Disease Prevention and Control and the US Centre for Disease Control and Prevention [12, 13]. For the current evaluation, only variables existing in both systems were compared in terms of completeness and timeliness. Data were available for the years 2002–2015, and the statistical analysis was conducted in STATA 14.1^® [14].

The completeness of key variables was assessed by examining field completion (percentage of blank responses). We exported the 2002–2010 data from the MS-Access database and 2011–2015 data from CIDR. We assessed data completion for both core (basic data collected on all notifiable diseases) and enhanced variables. The χ² test was used to determine the significance of proportions of completed data fields in each system.

To assess the timeliness of the systems, we calculated the time interval between the date of diagnosis and the date of notification to the system. Median time intervals for completed records were calculated and expressed with an interquartile range (IQR). The distribution of time intervals between both systems was compared using the Wilcoxon rank-sum test after being tested for normality with the Shapiro–Wilk W test.

All data exported from NTBSS were imported into the MS-Access database where they were validated and stored. The validation process was manual and included logical checks based on completed data that helped to fill missing values. For that reason, only validated NTBSS data were available for this evaluation. The CIDR data were also imported into the same MS-Access database and validated on an annual basis; for our analysis, we used the crude data exported from CIDR. In an effort to cross-validate our results, we also conducted the same analysis using the MS-Access validated data for both systems and compared the results with the initial analysis.

The attributes of simplicity, flexibility and usefulness were evaluated by creating an online questionnaire (on Demographix^®). The questionnaire included multiple choice questions with responses measured using a Likert scale. We sent the questionnaire to all public health staff in Ireland who enter or use TB data in CIDR, some of whom had also used the NTBSS. A descriptive analysis of the results was conducted.

Results

There were 4007 TB notifications before and 1776 after the introduction of CIDR. Completeness of core questions was higher in the NTBSS system but variable for enhanced questions (Table 1). There was no significant difference in completeness for most core demographic variables. The variable ‘organism’ was more complete in CIDR but date variables were more complete in NTBSS. The completeness of the variable indicating the cause of death was very low in CIDR compared with NTBSS. When comparing the validated data at both systems in MS Access, the completeness of this variable significantly improved but did not change for the remainder variables (Table 2).

Table 1.

Completeness^a of core and enhanced variables in NTBSS (2002–2010) and CIDR (2011–2015)

	NTBSS		CIDR
	No of reports	% of total reports	No of reports	% of total reports	P value
Core variables
ID	4007	100	1776	100	N/A
Event date	4007	100	1776	100	N/A
Place (health board)	4007	100	1776	100	N/A
Sex	4002	99.9	1772	99.8	0.371
Date of birth	3972	99.1	1774	99.9	0.001
Age	4001	99.9	1774	99.9	0.726
Country of birth	3945	98.5	1735	97.7	0.043
Organism	2599	64.9	1399	78.8	<0.001
Date of onset	2946	73.5	1066	60.0	<0.001
Date of diagnosis	3778	94.3	1501	84.5	<0.001
Date event notified	3998	99.8	1776	100	0.046
Death cause	1813	45.3	103	5.8	<0.001
Enhanced variables
Employment	2792	69.7	960	54.1	<0.001
Living status	3852	96.1	1530	86.2	<0.001
Race and ethnic group	3850	96.1	1512	85.1	<0.001
Refugee and asylum seeker	3833	95.7	1388	78.2	<0.001
Year entered Ireland if born outside	2754	68.7	1392	78.4	<0.001
TB history	3302	82.4	1317	74.2	<0.001
TB previous year diagnosed	3885	97.0	1755	98.8	<0.001
Previous treatment (>1 month)	3879	96.8	1732	97.5	0.139
Previous treatment completed	3814	95.2	1725	97.1	0.001
Vaccination	1821	45.5	594	33.5	<0.001
Presence of risk factors	3274	81.7	1339	75.4	<0.001
HIV status	501	12.5	529	29.8	<0.001
If extrapulmonary, which site?	1447	36.1	675	38.0	0.168
X-ray	3561	88.9	1457	82.0	<0.001
Case finding	3799	94.8	1657	93.3	0.022
Diagnosis	3992	99.6	1748	98.4	<0.001
Treatment outcome	3379	84.3	1004	56.5	<0.001
DSM1	3619	90.3	1476	83.1	<0.001
DSM1 date	3175	79.2	1668	93.9	<0.001
DSM2	1286	32.1	860	48.4	<0.001
DSM2 date	3829	95.6	1737	97.8	<0.001
Histology	1528	38.1	628	35.4	0.044
Culture	3697	92.3	1530	86.2	<0.001
Treatment outcome	3379	84.3	1004	56.5	<0.001
Isoniazid	2615	65.3	1228	69.1	0.004
Rifampicin	2615	65.3	1230	69.3	0.003
Pyrazinamide	2585	64.5	1216	68.5	0.003
Ethambutol	2560	63.9	1224	68.9	<0.001
Streptomycin	1159	28.9	1084	61.0	<0.001
DSM during treatment (2 months)	797	19.9	520	29.3	<0.001
DSM at end of treatment	559	14.0	415	23.4	<0.001
Culture during treatment (2 months)	727	18.1	453	25.5	<0.001
Culture at end of treatment	545	13.6	384	21.6	<0.001

^aThe highest percentage of completeness is highlighted in bold for each variable.

Table 2.

Completeness^a of core and enhanced variables of the validated MS-Access data before (2002–2010) and after (2011–2015) the introduction of CIDR and significance of their differences

	Access before 2011		Access 2011–2014
	No of reports	% of total reports	No of reports	% of total reports	P value
Core variables
ID	4007	100	1462	100	N/A
Event date	4007	100	1462	100	N/A
Place (health board)	4007	100	1462	100	N/A
Sex	4002	99.9	1459	99.8	0.491
Date of birth	3972	99.1	1460	99.9	0.003
Age	4001	99.9	1460	99.9	0.912
Country of birth	3945	98.5	1447	99.0	0.148
Organism	2599	64.9	1155	79.0	<0.001
Date of onset	2946	73.5	894	61.2	<0.001
Date of diagnosis	3778	94.3	1246	85.2	<0.001
Date event notified	3998	99.8	1462	100	0.07
Death cause	1813	45.3	1434	98.1	<0.001
Enhanced variables
Employment	2792	69.7	814	55.7	<0.001
Living status	3852	96.1	1284	87.8	<0.001
Race and ethnic group	3850	96.1	1264	86.5	<0.001
Refugee and asylum seeker	3833	95.7	1173	80.2	<0.001
Year entered Ireland if born outside	2754	68.7	1150	78.7	<0.001
TB history	3302	82.4	1112	76.1	<0.001
TB previous year diagnosed	3885	97.0	1443	98.7	<0.001
Previous treatment (>1 month)	3879	96.8	1428	97.7	0.094
Previous treatment completed	3814	95.2	1425	97.5	<0.001
Vaccination	1821	45.5	501	34.3	<0.001
Presence of risk factors	3274	81.7	1189	81.3	0.748
HIV status	501	12.5	450	30.8	<0.001
If extrapulmonary, which site?	1447	36.1	571	39.1	0.046
X-ray	3561	88.9	1228	84.0	<0.001
Case finding	3799	94.8	1394	95.4	0.420
Diagnosis	3992	99.6	1459	99.8	0.334
DSM1	3619	90.3	1240	84.8	<0.001
DSM1 date	3175	79.2	1392	95.2	<0.001
DSM2	1286	32.1	739	50.6	<0.001
DSM2 date	3829	95.6	1220	83.5	<0.001
Histology	1528	38.1	548	37.5	0.661
Culture	3697	92.3	1283	87.8	<0.001
Treatment outcome	3379	84.3	944	64.6	<0.001
Isoniazid	2615	65.3	1048	71.7	<0.001
Rifampicin	2615	65.3	1047	71.6	<0.001
Pyrazinamide	2585	64.5	1040	71.1	<0.001
Ethambutol	2560	63.9	1044	71.4	<0.001
Streptomycin	1159	28.9	914	62.5	<0.001
DSM during treatment (2 months)	797	19.9	472	32.3	<0.001
DSM at end of treatment	559	14.0	389	26.6	<0.001
Culture during treatment (2 months)	727	18.1	415	28.4	<0.001
Culture at end of treatment	545	13.6	363	24.8	<0.001

^aThe highest percentage of completeness is highlighted in bold for each variable.

The completeness of enhanced demographic variables was higher in NTBSS compared with CIDR (Table 1). Variables on risk factors and vaccination were significantly more complete in NTBSS, but the completeness of the variable on HIV status was better in CIDR. The variables on the chest x-ray results and on diagnosis were more complete in NTBSS but completeness of diagnosis improved in CIDR after the data validation (Table 2). Variables for histology and culture were more complete in NTBSS. On the other hand, all variables related to drug sensitivity and laboratory testing after treatment initiation (follow-up) were significantly more complete in CIDR (Table 1).

The time interval between diagnosis and notification to the system significantly increased after the introduction of CIDR from a median of 3 days (IQR: 0–11) to 4 days (IQR: 1–12) (Wilcoxon rank-sum rest P < 0.001) indicating more delayed notification time. Figure 1 shows the medians and IQRs of the time interval between diagnosis and notification of TB by HSE area for both NTBSS and CIDR. For most areas, the median time from diagnosis to notification seemed to increase after the introduction of CIDR.

Fig. 1.

Days* between diagnosis and notification of TB on NTBSS between 2002 and 2010 (a) and CIDR between 2011 and 2015 (b) by health authority area. *Extreme values above 50 days were removed to allow the medians and interquartile ranges to be visible in the graph.

Twenty-four out of 27 people responded to the online questionnaire regarding simplicity, usefulness and flexibility of the systems (89% response). Most respondents were surveillance scientists (n = 9), followed by senior medical officers (n = 6), specialists in public health medicine (n = 6), infection control nurses (n = 2) and administration staff (n = 1). Twenty out of the 24 respondents had used both CIDR and NTBSS. Table 3 shows that based on a Likert scale, more people found that CIDR was easy to use (20–40%) and user-friendly (33%), while less people thought that it is difficult to use (4–17%) and not user-friendly (21%). Respondents agreed that the system (54%), the necessary variables (63%) and reports (29%) were available for the effective surveillance of TB. One third rated the ability of CIDR to describe TB on a weekly basis as good but were less happy with its ability to detect outbreaks. On the other hand, users thought that the system is not very flexible in terms of adding new variables (25%) and producing or modifying existing reports (33%). When people were asked to compare the two systems, the highest percentage thought that CIDR is simpler than NTBSS (38% vs. 25%), more useful (41% vs. 13%) and more flexible (29% vs. 25%).

Table 3.

Results from the online questionnaire on simplicity, usefulness and flexibility of CIDR, N = 24^a

Question
	Very easy (%)	Easy (%)	Neutral (%)	Difficult (%)	Very difficult (%)	N/A (%)
Simplicity
How would you rate organising the cross-cover for TB surveillance on CIDR?	2 (8.3)	1 (4.2)	8 (33.3)	1 (4.2)	0	11 (45.8)
Is creating a new TB event	5 (20.8)	7 (29.2)	0	2 (8.3)	0	10 (41.7)
Is viewing TB event information	2 (8.3)	10 (41.7)	7 (29.2)	3 (12.5)	2 (8.3)	0
Is updating TB-enhanced data	2 (8.3)	9 (37.5)	3 (12.5)	4 (16.7)	0	6 (25.0)
Is linking laboratory data to clinical information	3 (12.5)	7 (29.2)	2 (8.3)	3 (12.5)	0	9 (37.5)
Is creating outbreaks and entering outbreak data	2 (8.3)	7 (29.2)	7 (29.2)	2 (8.3)	0	6 (25.0)
Is producing TB reports	3 (12.5)	5 (20.8)	4 (16.7)	3 (12.5)	2 (8.3)	7 (29.2)
	Very user-friendly (%)	User-friendly (%)	Neutral (%)	Not user-friendly (%)	Not at all user-friendly (%)	N/A (%)
How would you rate the user friendliness of CIDR?	1 (4.2)	8 (33.3)	7 (29.2)	5 (20.8)	3 (12.5)	0
Usefulness	Strongly agree (%)	Agree (%)	Neutral (%)	Disagree (%)	Strongly disagree (%)	N/A (%)
CIDR is available to me when needed for TB surveillance	6 (25.0)	13 (54.2)	2 (8.3)	1 (4.2)	0	2 (8.3)
The necessary variables for effective TB surveillance are available on CIDR	4 (16.7)	15 (62.5)	1 (4.2)	2 (8.3)	0	2 (8.3)
The necessary reports for effective TB surveillance are available on CIDR	4 (16.7)	7 (29.2)	5 (20.8)	4 (16.7)	1 (4.2)	3 (12.5)
	Very good (%)	Good (%)	Fair (%)	Poor (%)	Very poor (%)	N/A (%)
How would you rate the ability of CIDR to describe TB activity on a weekly basis?	3 (12.5)	8 (33.3)	2 (8.3)	5 (20.83)	1 (4.2)	4 (16.7)
How would you rate the ability of CIDR to detect TB outbreaks from the current data collected?	1 (4.2)	7 (29.2)	9 (37.5)	5 (20.8)	1 (4.2)	1 (4.2)
Flexibility	Very easy (%)	Easy (%)	Neutral (%)	Difficult (%)	Very difficult (%)	N/A (%)
How easy do you think it is to request and add new fields to CIDR?	1 (4.2)	1 (4.2)	4 (16.7)	6 (25.0)	5 (20.8)	6 (25.0)
How easy do you think it is to request and produce new reports/modify existing reports in CIDR?	0	2 (8.3)	8 (33.3)	6 (25.0)	2 (8.3)	5 (20.8)

In bold, highlighted the response with the highest percentage for each question.

^aThe table presents only the questions related to CIDR and not those on the comparison between CIDR and NTBSS.

Discussion

This is the first study that assesses the completeness, timeliness, simplicity, flexibility and usefulness of the NTBSS after the introduction of CIDR and also compares these attributes to the NTBSS. The WHO TB elimination plan highlights the need for having precise and timely clinical and microbiological data to monitor the disease effectively [4].

In our study, we compared the completeness of the most important core and enhanced variables between CIDR and NTBSS. The core demographic variables were well completed in both systems, while the enhanced demographic variables were more complete in NTBSS. A better knowledge of these variables (employment, living status, ethnic group, refugee/asylum seeker) will allow us to better evaluate the trends of the disease among vulnerable populations and identify public health actions targeted to specific groups [15–19]. The core dates were more complete in NTBSS even when compared with the validated CIDR data, which highlights that not all variables for notified cases are entered consistently.

Variables related to risk factors and vaccination have better completion in NTBSS apart from the HIV status for which the completion improved after the introduction of CIDR; that could be a result of either the integration of clinical and laboratory data in CIDR or of the fact that HIV became a notifiable disease in 2011 [20]. Having good quality data on the HIV status of the TB patients can support collaborative TB/HIV activities [21].

The completeness of all variables related to drug sensitivity has significantly improved in CIDR. That may be a function of both awareness of multidrug-resistant-TB among public health professionals and the integration of clinical and laboratory data within CIDR. TB surveillance requires data on levels of TB drug resistance in the country and routine testing of all TB patients helps us better understand and monitor the trends [22]. All variables related to the outcome of treatment are better completed in CIDR, although the completion still remains low. Follow-up questions after treatment initiation can be better captured using a centralised electronic system that can be accessed by any person and updated at any time.

The time interval evaluated as an indicator of the timeliness of the surveillance system was the time between diagnosis of the disease and notification to the system. The 2010 Irish guidelines on the prevention and control of TB state that, ideally, TB cases should be notified not later than 1 working day following diagnosis but this is very hard to achieve in reality [23]. From our evaluation, the median time from diagnosis to notification was more than 1 day and it increased after the introduction of CIDR. This increase was observed in all HSE regions irrespective of their TB incidence. The difference in the timeliness of notifications observed between regions could be a function of human resource capacity for entering data both in laboratories and departments of public health. Although the NTBSS data appear more timely, it should be noted that the NTBSS date of notification may not reflect the date the case was entered onto the local database and it was also not the date that was available nationally. As CIDR is intended to be a ‘real-time’ surveillance system, the provision of timely data should be a priority and this needs to be further explored with users.

A weakness of the current system is that all data from CIDR are imported into an MS-Access database at HPSC that contains all national validated TB surveillance data since 2002. This database is used to facilitate report production and statistical analysis (the same also happens for other diseases) due to the limited capabilities of the reporting tool within CIDR. This additional validation is a consequence of having the data offline where a natural step is then to spot gaps or inaccuracies and augment the data as the analysis is in progress. This results in increased completeness of certain variables (e.g. cause of death). A centralised national electronic system though should have better business intelligence and reporting tools and should not require this process, which increases the workload.

The results of the stakeholder survey showed that many respondents rated CIDR as ‘user-friendly’ and ‘easy’ to use. When comparing the simplicity between the two systems, almost half of the respondents thought that CIDR was simpler to use. The usefulness of CIDR was also rated higher. Having a centralised surveillance system makes it simpler and faster to view and update data. In addition, CIDR keeps free-text fields to a minimum and uses standardised drop-down menus, which minimises errors and creates a dataset in consistent format that is easier to analyse [11]. A high percentage though did not rate highly the ability of CIDR to detect outbreaks but CIDR is a retrospective surveillance tool and was not designed to detect outbreaks (although if outbreaks are detected, they are reported to CIDR). Despite the fact that CIDR was considered to be more flexible than NTBSS, its flexibility alone was not rated very highly and that is an area that should be improved.

In conclusion, this study demonstrated that centralised electronic surveillance did not improve completeness of data and reduced the timeliness of the notification procedure. Simplicity, usefulness and flexibility were improved but qualitative methods should be applied to further explore these results. HPSC needs to explore ways with CIDR users to ensure optimal use of the system in order to improve its levels of completeness and timeliness.

Conflict of interest

None.