Quality Assurance in Trichiasis Surgery: a methodology

SUMMARY

Trachoma remains a significant cause of blindness in many parts of the world. The major route to blindness involves upper lid entropion leading to trachomatous trichiasis (TT) which promotes progressive corneal opacification. The provision of surgery to correct TT in the populations most severely affected is a major challenge for the global effort to eliminate Trachoma blindness by the year 2020.

Most attention has been paid to increasing the quantity of TT surgery performed, and large numbers of non-doctor operators have been trained to this end. Surgical audit by those performing TT surgery is not a routine part of any national trachoma control programme, and no effective mechanism exists for identifying surgeons experiencing poor outcomes.

We propose a methodology for surgical audit at the level of the individual surgeon based on Lot Quality Assurance. A set number of patients operated on previously for upper eyelid TT are examined to detect the recurrence of TT. The number of recurrent cases found will lead to categorisation of the TT surgeon to either “high recurrence” or “low recurrence” with reasonable confidence. The threshold of unacceptability can be set by individual programmes according to previous local studies of recurrence rates or those from similar settings. Identification of surgeons delivering unacceptably high levels of recurrent TT will guide managers on the need for remedial intervention such as re-training.

INTRODUCTION

Trachoma is the leading infectious cause of blindness worldwide. Recurrent infection by Chlamydia trachomatis leads to progressive tarsal conjunctival scarring, entropion and trichiasis. Ultimately corneal blindness develops as a result of the trauma from trichiasis and secondary bacterial infection. Trachoma control programmes (TCP’s) in endemic countries seek to prevent blinding trachoma through implementation of the SAFE Strategy.¹ The “S” stands for surgery for trichiasis.

Current estimates suggest there are 8 million people with trachomatous trichiasis (TT) in need of surgery.² In response to this large backlog of un-operated TT the emphasis has understandably been on quantity rather than quality of surgery performed. Most trachoma endemic countries do not have enough ophthalmologists to deliver TT surgery and therefore trained many non-doctor trichiasis surgeons. A randomised trial comparing ophthalmologists with integrated eye care workers found comparable results.³

The usual outcome measure for TT surgery is the proportion of operated eyes with recurrent TT, reportedly ranging from 10% at one year to 60% at three years.^4-10 Two categories are distinguished: early (0-3 months) recurrence due to surgical factors and late recurrence due to progressive cicatricial disease. Several factors may affect the risk of early recurrence: quality of surgery, specific technique, suture type and pre-operative disease severity. One study, conducted under “operational” conditions, reported significant inter-surgeon recurrence rate variability (0 to 80%).⁶ However, despite this, little attention has been paid by TCP’s to the need for surgical quality assurance. To our knowledge no formal methodologies have been published to guide this quality assurance process.

In 2003-04, evaluations were conducted of the TCP’s of eight countries. The most frequently proposed method of surgical quality assurance found was supervisory visits by experienced surgeons to observe TT surgery in progress. However, no documentation of any such supervisory process was found in any setting.¹¹ Such a method of quality assurance may have some validity, particularly if the use of a checklist is employed,¹² but is subject to numerous logistical constraints and does not evaluate actual surgical outcomes.

Here we propose a simple quantitative method of surgical audit which can be used by individual surgeons or programme managers to determine with reasonable certainty whether a recurrence rate higher than that deemed acceptable is encountered.

METHODOLOGY

Overview

We have used the Lot Quality Assurance Sampling (LQAS) methodology¹³ to develop an audit tool for assessing the outcome of upper eyelid TT surgery in terms of post-operative recurrence of TT. This method does not estimate the actual recurrence rate experienced, but merely assigns the surgeon or group of surgeons performing the surgery to one of two groups: “high recurrence” or “low recurrence”. This may provide programme managers with useful information to trigger remedial measures like additional training and supervision where “high recurrence” is identified.

Definitions of Recurrence

Standardisation of the definition of recurrence is essential for the audit process. The common definition of recurrent upper eyelid TT after surgery is: one or more upper lid eyelashes touching the globe, evidence of epilation or a history of repeated TT surgery.^6,7,10 Some TCP’s have a policy of operating only on those with more extensive trichiasis (“minor” vs “major” TT).⁶ Such programmes could use an alternative definition of recurrence: “TT requiring surgery”, similar to that used to drive the initial management decision regarding surgical intervention.

Trichiasis Surgery Outcome Standards

A wide range of TT recurrence rates has been reported ^4-8,10,14. Various local factors such as variations in the relative severity of pre-operative TT , ongoing exposure to C. trachomatis infection, quality of surgical training, volume of surgery being performed, surgical technique and suture material may influence the long term results. ^{4,9, 10} It is therefore difficult to propose a benchmark level for a universally “acceptable” recurrence rate, or suggest categorised benchmarks based on a combination of these factors. Each individual programme would need to determine their own benchmark for maximum acceptable recurrence rate, preferably based on previous studies of recurrence rates or those from similar settings.

Recurrence rates increase with time hence the interval between surgery and follow-up heavily influences the recurrence rate observed. The lowest published recurrence rates are around 10% at 1 year.^7,15 However, most series under operational rather than trial conditions are between 20-40%. Taking a pragmatic stand-point, we would suggest that everyone performing surgery for TT should be aiming to achieve a recurrence rate below 20% in the first year post-operatively. This figure does not represent the desired average result for surgeons within a programme, but rather a recurrence rate above which it would be reasonable for a surgeon to be required to undergo a re-training and certification process.

Assessment Method

LQAS employs a relatively simple statistical method, which has been reported previously. ^13,16,17 Software (SampleLQ v1.10) using published algorithms is freely available on-line (www.brixtonhealth.com). In brief, the proportions of the variable under consideration in a given population (recurrent TT) are chosen that are deemed definitely acceptable and unacceptable.

If the benchmark TT recurrence rate is set at 20%, as suggested above, these levels could be set at 10% (definitely acceptable) and 30% (definitely unacceptable). A sample size is then chosen that is small enough to be practicable, but large enough to give an acceptable degree of confidence for categorising a population as experiencing “acceptable” or “unacceptable” recurrence. Utilising SampleLQ v1.10, we determined that sample (n) of 40 eyes would give around 95% confidence in correctly categorising a sample if the threshold number of cases of recurrence (d) was set at 7. Thus, a sample of up to 40 operated eyes would need to be examined for recurrent TT until either: (i) 8 cases with recurrence have been found, or (ii) all 40 have been examined with 7 or less recurrent TT cases found. Where 7 (d) or less have recurrent TT, the prevalence of recurrence in the sampled population can be classified as 10% or less with a probability of 96%. When an 8th case of recurrent TT is found, the prevalence of recurrence in the sampled population is classified as 30% or more with a probability of 94%.

The risk of wrongly classifying a poorly performing surgeon who is actually experiencing high recurrence rate (30% or more) as having a low recurrence rate (10% or less) is 4%. This is the “consumer probability of error”. The risk of wrongly classifying a surgeon with a low recurrence rate (10% or less) as having a high recurrence rate is 6% (table 1). This is the “provider probability of error”.

Table 1. Lot Quality Assurance sampling for TT recurrence assessment.

Bench- mark	Categorisations		Expected error		Stopping rules
	“Low” Rate of Recurrent TT	“High” Rate of Recurrent TT	False allocation to “Low Recurrence”	False allocation to “High Recurrence”	N Φ	d Θ
15%	10%	20%	9%	7%	100	14
20%	10%	30%	4%	6%	40	7
30%	20%	40%	6%	5%	50	14

^ΦN is the maximum number of patients required to be examined

^Θd is the number of identified recurrences above which designation of the subject as experiencing “high recurrence” is triggered

In the bottom two rows of table 1 (Benchmark 20% and 30%), the gap between the categorisation prevalence classes is set at 20%. As can be seen for the example with a benchmark of 15%, narrowing this gap increases N, and therefore increases the logistical problems of conducting the audit. Widening the categorisation gap will reduce N, but leads to an increase in the expected errors. The exact change in accuracy can only be calculated by simulation models or field trials.

Follow up Interval

The time between surgery and the follow-up is important.¹⁸ Therefore, efforts should be made to ensure that the mean follow up interval is approximately one year after surgery, and that cases are not included that are outside an acceptable range, such as 10-14 months. In low surgical volume practices, where insufficient cases fall into the 12 month follow-up window (10 to 14 months), the audit could be performed in more than one blocks, for example, leaving a four month gap between each round of recruitment until sufficient cases have been examined.

Patient Record Keeping

An audit process in any setting is dependant on sufficiently detailed and accurate surgical records to allow tracing of patients. Where adequate record keeping has not been maintained, commencement of this practice would be of first importance. A record should include: name, contact information, age, pre-operative disease severity, date of surgery, surgeon name, type of surgery and complications.

Selection of Patients to Audit

Ideally, the method for selecting cases for audit should minimise potential for bias, whilst being as tightly clustered around one year post-surgery as possible. As a minimum standard, 40 consecutive cases, which tightly cluster around one year post-surgery, should be selected for audit. Cases lost to follow up should be replaced in the sample by a pre-determined method, such as the construction of a reserve list of a size sufficient to cover the anticipated number of cases lost to follow up.

Alternatively, to reduce potential bias further, a list of all cases eligible for inclusion could be constructed, and names selected at random by use of random number tables or drawing lots. The measures that should be taken to trace cases should be pre-determined to minimise bias from excess loss of cases who may be at risk of higher recurrence rates, for instance due to living in more remote areas or having persistent symptoms and being reluctant therefore to re-attend for examination. Cases from the original selection that are lost to follow up should be replaced by the same method of random selection.

Removing the responsibility for case selection and follow-up examination from the surgeon being audited is highly desirable, to ensure greater accuracy in the diagnosis of recurrent TT and to reduce selection bias. This may require training of specific individuals to act as audit workers within a programme.

Transportation and other logistical considerations may favour the audit of the most accessible cases. Unfortunately, this may introduce significant bias towards finding better surgical outcomes. People with easier access to surgical services at health centre may tend to have surgery when the disease is milder (which has a better outcome) and their communities may have benefited from more interventions against trachoma in general.

Many health care workers who perform TT surgery are involved in other programmes such as health education, outreach activities, vaccination programmes or antibiotic distribution for trachoma. Data collection for this TT audit process could be combined with activities whereby communities are visited from which cases of TT were drawn, and would permit assurance that all cases were operated by the same surgeon.

DISCUSSION

Community based trichiasis surgery programmes have been established in many countries to provide treatment to millions of people with potentially blinding trichiasis, often in remote rural settings. This has generally involved the training and equipping of health care professionals who usually have little or no previous surgical experience. Inevitably, to date the emphasis has been on delivering high volume surgery to address the backlog. In recent years results from operational settings indicate that the outcome of surgery, in terms of recurrent trichiasis, are not as good as anticipated. Therefore, the important issue of surgical quality needs to be seriously addressed by trachoma control programmes. Efforts are being made to improve the quality of the surgical technique and the training programmes. A certification process, conducted at the completion of surgical training, has been taken up by some countries.¹⁹ However, after this point, there are no formal guidelines on estimating the level of recurrence.

Published data from operational settings indicate that the outcome of TT surgery can vary widely between surgeons.⁶ The impact of poor quality surgery may be profound. Repeat TT surgery is technically more difficult. Recurrence probably leads to an increased risk of blindness, increasing reluctance of communities to accept surgery and increased cost to the patient and the health service. Audit is standard practice in modern health care systems worldwide. It supports the ongoing improvement and maintenance of standards in services. Therefore, it seems appropriate for trachoma control programmes to develop mechanisms for the ongoing audit of results. However, this aspiration comes with many challenges, not least the logistical difficulty of following up patients in inaccessible rural communities. Any proposed methodology should be simple and where possible minimise the sample size required.

Methodologies that allow categorisation without absolute definition of rates have previously been used to good effect in blindness prevention. Rapid Epidemiological Mapping of Onchocerciasis (REMO) surveys in communities suspected to be endemic for Onchocerciasis utilised the LQAS method to determine whether mass distribution of ivermectin is indicated.²⁰ A method based on LQAS has also been employed to prioritise communities in trachoma endemic areas for implementation of trachoma control interventions.^13,17 Here we use a similar methodology to stratify the outcome of an individual surgeon’s surgery into low and high recurrence groups. We believe that this is a practical solution permitting targeted interventions to improve quality. At the same time it minimises the amount of patients that would need to be followed up.

Probably the biggest challenge in introducing any surgical audit tool would be in empowering the TT surgeons themselves to own the process as a mechanism to facilitate either affirmation of good results or the opportunity to improve. It is important to avoid it being viewed as a tool to identify those who are “failing”. This is very important as most TCP’s are working with limited human resources and maintaining staff motivation is essential. However, TCP’s have a duty of care to provide surgical services with acceptable recurrence rates. For this it is necessary to monitor results and provide additional training, support and supervision where results are less good. This proposed methodology would allow audit to be undertaken in a relatively easy but statistically robust fashion, at the level of the individual surgeon. The methodology might also be applied at the level of a geographic region or for the TT surgical programme as a whole, if more thorough evaluation of post-operative TT recurrence rates where not possible, although it would be more difficult to target remedial action to improve the situation if a higher recurrence rate were identified.