Practice Audit in Gastroenterology (PAGE) program: A novel approach to continuing professional development

Abstract

BACKGROUND:

Practice audit is an important component of continuing professional development that may more readily be undertaken if it were less complex. This qualitative study assessed the use of personal digital assistants to facilitate data collection and review.

METHODS:

Personal digital assistants programmed with standard questionnaires related to upper gastrointestinal endoscopies (Practice Audit in Gastroenterology-Endoscopy [‘PAGE-Endo’]) and colonoscopies (PAGE-Colonoscopy [‘PAGE-Colo’]) were provided to Canadian gastroenterologists, surgeons and internists. Over a three-week audit period, participants recorded indications, and the expected (E) and reported (R) findings for each procedure. Thereafter, participants recorded compliance with reporting, the ease of use and value of the PAGE program, and their willingness to perform another audit.

RESULTS:

Over 15 to 18 months, 173 participants completed PAGE-Endo (6168 procedures) and 111 completed PAGE-Colo (4776 procedures). Most respondents noted that PAGE was easy to use (99%), beneficial (88% to 95%), and that they were willing undertake another audit (92% to 95%). In PAGE-Endo, alarm features were prevalent (55%), but major reported findings were less common than expected: esophagitis (E 29.9%, R 14.8%), esophageal stricture (E 8.3%, R 3.6%), gastric ulcer (E 17.0%, R 4.7%), gastric cancer (E 4.3%, R 1.0%) and duodenal ulcer (E 11.5%, R 5.7%). In PAGE-Colo, more colonoscopies were performed for symptom investigation (55%) than for screening (25%) or surveillance (20%). There were marked interprovincial variations with respect to sedation, biopsies and technical aspects of colonoscopy.

CONCLUSION:

Secure, real-time data entry with review of aggregate and individual data in the PAGE program provided an acceptable, straightforward methodology for accredited practice audit activities. PAGE has considerable potential for continuing professional development in gastroenterology and other specialties.

Continuing professional development (CPD) is essential for physicians to acquire up-to-date information, incorporate learning into practice and, in many jurisdicitons, maintain certification (1,2). In Canada, the Canadian Association of Gastroenterology (CAG) is the national body approved by the Royal College of Physicians and Surgeons of Canada for accreditation of gastroenterology CPD activities.

Currently, practising Canadian specialists must earn 400 credits in six CPD categories over five years to maintain certification (FRCPC/FRCSC). One of these categories (Section 5) provides two credits per hour for activities (including practice audit) that involves direct evaluation and improvement of clinical practice (3). Practice audit requires physicians to select an aspect of care for evaluation, identify audit goals (4) and develop target criteria against which clinical performance is judged. Data are collected and compared with target criteria to identify practice gaps that are addressed by the implementation of new management strategies. After a suitable time period, data are collected again to evaluate the success of implemented changes.

Despite their recognized value in learning and improving health care, practice audits are not widely used. Perceived barriers, including lack of expertise or time to develop a protocol or carry out an audit, might be overcome if there were independently developed protocols that were relevant to the individual’s practice, straightforward to complete and provided outcomes that were easy to evaluate in comparison with peer practice.

The present paper reports an innovative practice audit program developed to overcome these perceived barriers. Using personal digital assistants (PDAs) preprogrammed with audit questions, physicians enter data as they work. After a specified period, data are downloaded to a secure central Web site database, allowing physicians to review their own data and to compare them with anonymous aggregate data reported by their peers.

To determine whether a structured, PDA-based practice audit program can provide a well-accepted, easy-to-use learning tool, endoscopy practice audit programs were developed as an educational initiative of the CAG and AstraZeneca Canada Inc. Participants collected data from consecutive endoscopic procedures over a three-week period in two programs: the first was for upper gastrointestinal endoscopy –Practice Audit in Gastroenterology-Endoscopy (PAGE-Endo) and the second, started one year later, was for colonoscopy –PAGE-Colonoscopy (PAGE-Colo). A descriptive analysis of the data provides an overview of endoscopic practice, nationally and provincially, across Canada as a basis for future practice audits by Canadian endoscopists.

METHODS

Questionnaires for both programs were developed by a steering committee with expertise in gastroenterology, education and software. To simplify the practice audit and subsequent analyses, questionnaires were designed for data entry using list boxes and drop-down menus rather than free text. In addition, the stem for each question and all response options were visible at the same time on one screen without the need to scroll through options, and the programming ensured that participants could not proceed without completing the current question. The questionnaires were programmed in Visual Basic (Embedded Visual Basic 3.0, Microsoft, USA); after pilot testing, they were revised and downloaded to the PDAs (iPAQ Model 3670, Compaq, Taiwan), which were lent, for three-week periods, to physicians, who entered demographic and procedure-specific data in three sections.

Section I:

Demographic data, entered once at the start of the practice audit, included details of the participant’s specialty, practice and endoscopy suite locations, referral population size and an estimate of the number of endoscopies performed as a percentage of all endoscopic procedures. In PAGE-Colo, participants also entered practice details with respect to screening colonoscopy and sigmoidoscopy.

Section II:

Before each procedure, participants entered patient historical data, without any identifiers and, afterwards, procedure-specific information.

In PAGE-Endo, participants answered up to 17 questions per procedure. Preprocedural data included symptoms and signs warranting patient referral, the main reason for performing the endoscopy, whether they expected to find a clinically significant lesion, the presence of specific alarm symptoms, data on previous endoscopies and their predictions as to the specific endoscopic findings or diagnoses (‘expected’ findings). Postprocedural data included the use and type of conscious sedation, endoscopic findings and diagnoses (‘reported’ findings), details of biopsies, therapeutic interventions and the effects of the findings on subsequent management.

In PAGE-Colo, participants answered up to 52 questions per procedure. Preprocedural data included patient characteristics, the indication for colonoscopy and the preparation used. Then, depending on the indication for colonoscopy, one of three sets of questions was answered before the procedure: Path A – investigation of symptoms; Path B – screening for colon cancer or polyps; or Path C – surveillance. Data recorded included: Path A – reason for referral and expected diagnoses; Path B – reason for referral, details of previous normal screening colonoscopies, the number of relatives with colorectal cancer or polyps and the endoscopist’s estimated likelihood of finding colorectal cancer; and Path C –reason for surveillance and details of previous colonoscopies. Postprocedural data included adequacy of preparation, extent of examination, landmarks used to identify the cecum, time required to complete the procedure, monitoring procedures, medications used and plans to recolonoscope the patient. Participants also recorded findings and diagnoses, including information on polyps, therapeutic interventions, immediate complications and the effects of the colonoscopic findings on subsequent management.

Section III:

Participants reported the percentage of procedures for which they had entered data, their evaluation of the program, their wishes to complete another practice audit and topics of interest for future programs.

The PDAs were collected after audit completion and data were downloaded to personal computers for transmission to the central server/database, where an automated quality control system checked for data validity, duplication or loss. Central server data were validated against the original PDA data with random manual checks to ensure data integrity. Following data validation, participants could review their own data and aggregate data from their peers at a secure, password-protected, encrypted CAG Web site.

Data analysis

Data were subject to a descriptive analysis, presented as the proportions of procedures for which specific features were reported, nationally and provincially. In the event that there were no more than two participants per province, these data were not presented separately.

RESULTS

Endoscopist demographics

PAGE-Endo was completed by 173 participants over 18 months and PAGE-Colo by 111 participants over 15 months (Table 1). Most participants were community-based, and approximately 80% were gastroenterologists, with greatest representation from Ontario and British Columbia (both PAGE programs), Quebec and Nova Scotia (PAGE-Endo); 89 (51%) of those who completed PAGE-Endo also completed PAGE-Colo.

TABLE 1.

Demographics related to participants in the Practice Audit in Gastroenterology (PAGE) programs

	PAGE-Endo	PAGE-Colo
Reporting period	January 2002 to June 2003	January 2003 to March 2004
Number of participants	173	111
Specialty
Gastroenterologist	137	90
Surgeon	18	18
Internist	6	3
Other specialty	0	0
Data not collected	12	–
Province
British Columbia	34	29
Alberta	4	6
Saskatchewan	4	1
Manitoba	5	3
Ontario	74	51
Quebec	27	9
New Brunswick	0	1
Nova Scotia	20	7
Newfoundland	5	4
Practice location
Community hospital	49%	66%
University/teaching hospital	30%	20%
Private office	14%	14%
Data not collected	7%	–
Endoscopy suite location
Community hospital	61%	75%
University/teaching hospital	29%	21%
Private office	3%	4%
Data not collected	7%	–
Procedures that are	Endoscopies	Colonoscopies
≤20%	2%	0%
21%–40%	57%	8%
41%–60%	32%	43%
61%–80%	1%	44%
81%–100%	1%	5%
Data not collected	7%	–
Average number of patients endoscoped/physician	36	43
Total number of procedures	6168	4776

PAGE-Colo PAGE questionnaire related to colonoscopy; PAGE-Endo PAGE questionnaire related to upper endoscopy

Audit compliance

In the PAGE-Endo, participants reported an average of 36 procedures. Section III was completed by 80 participants (46%); 22% reported documentation of all endoscopies, while 47% reported documentation of 91% to 99% of procedures (Figure 1). In PAGE-Colo, participants reported an average of 43 procedures. Section III was completed by 43 of participants (39%); 56% reported documentation of all colonoscopies, while 30% reported documentation of 91% to 99% of procedures (Figure 1).

Figure 1).

Estimated percentage of upper gastrointestinal endoscopies and colonoscopies documented for audit purposes by participants in the Practice Audit in Gastroenterology (PAGE)-Endoscopy (80%) and PAGE-Colonoscopy (43%) programs

Technical aspects of the PDA system

Data loss due to short PDA battery life occurred in 10% to 15% of audits during the first four months of PAGE-Endo. This was prevented in later audits by adding a memory card on which every procedure record was saved immediately on completion. If the battery failed, the database and program were restored automatically once the battery had been recharged. Subsequent data validation procedures confirmed data integrity during transfer from the PDA to the central server.

Evaluation of the PDA practice audit experience

The audit evaluation (Section III) was completed by 46% and 39% of participants in PAGE-Endo and PAGE-Colo, respectively. Of these participants, most reported that the program was beneficial (PAGE-Endo 88%, PAGE-Colo 95%), easy to use (99% and 99%, respectively), not burdensome (91% and 80%, respectively) and did not pose too many questions (95% and 79%, respectively). Overall, 92% and 95% of PAGE-Endo and PAGE-Colo participants, respectively, were willing to undertake another practice audit on a different topic.

PAGE-Endo

Overall, alarm features other than age over 50 years (5) were identified as reasons for endoscopy in 55% of 6168 endoscopy patients: anemia or evidence of gastrointestinal blood loss (27%), dysphagia (18%), vomiting (14%), weight loss (14%) and an abdominal mass (1%). Furthermore, 68% of all upper endoscopies were performed in patients who were over 50 years of age. Anemia or evidence of gastrointestinal blood loss were more common in patients over 50-years-old (33% versus 16% in those 50 years or younger), while other alarm features were equally prevalent in both age groups. The proportions of procedures in which the different alarm features were noted varied markedly between provinces (Figure 2).

Figure 2).

The percentage of procedures for which patients were over the age of 50 years or had alarm features reported preprocedure (Practice Audit in Gastroenterology-Endoscopy program). The triangles indicate percentages in each province (Alberta, British Columbia, Manitoba, Newfoundland, Nova Scotia, Ontario, Quebec and Saskatchewan), and horizontal lines show the overall percentage for each feature

Endoscopy was normal in 1865 cases (30%). The most prevalent (greater than 5%) findings were gastritis (18%), hiatus hernia (16%), erosive esophagitis (14%), gastric erosions (7.2%), endoscopic Barrett’s esophagus (6.8%), esophageal stricture (6.4%) and duodenal ulcer (5.4%). For 4945 endoscopies, participants were presumed to have had no prior knowledge of possible findings, because the reason for endoscopy was neither to perform a therapeutic procedure nor to follow up on a previous lesion. Among these procedures, there was agreement between expected findings (recorded pre-procedure) and those reported (recorded postprocedure) (Figure 3) for the presence of a normal endoscopy (expected 31.7%, reported 34.7%). In contrast, nearly all positive findings and diagnoses were less prevalent than expected, except hiatus hernia, which was slightly more prevalent.

Figure 3).

Most common expected findings compared with reported findings from 4945 endoscopies for which participants had no apparent advance knowledge of existing lesions or need for therapeutic intervention. NSAID Nonsteroidal anti-inflammatory drug

Overall, alarm symptoms were not strong predictors of underlying disease, although esophageal stricture was more common in patients with dysphagia (31.9%) than in those without dysphagia (0.95%), as was a Schatzki’s ring (11.6% versus 0.83%). Of the 50 patients who had an abdominal mass, 11 (22%) had gastric cancer.

Conscious sedation was used in 76% of the 6168 procedures; most endoscopists used a benzodiazepine alone (53%) or with an opiate (42%), but there were significant regional differences in sedation practice (Figure 4). Testing for Helicobacter pylori was performed in 54% of patients, and mucosal biopsies from the esophagus, stomach or duodenum were taken in 69% of patients, although there were marked interprovincial variations in the proportions of patients who were tested (Figure 5). Similarly, there were marked variations with respect to the proportions of procedures during which a therapeutic intervention was performed (14%) and the proportions of procedures that were considered to have modified the participant’s treatment plan (60%) (Figure 5).

Figure 4).

Use of conscious sedation in upper gastrointestinal endoscopy. Results are shown for provinces in which there were more than 15 participants (British Columbia [BC] – 34, Ontario [ON] – 74, Quebec [QC] – 26 and Nova Scotia [NS] – 20)

Figure 5).

The percentages of procedures in which patients underwent testing for Helicobacter pylori, mucosal biopsy or a therapeutic intervention, in addition to the procedures that were reported to have led to a change in planned management (Practice Audit in Gastroenterology-Endoscopy program). The triangles indicate percentages for each province (Alberta, British Columbia, Manitoba, Newfoundland, Nova Scotia, Ontario, Quebec and Saskatchewan), and horizontal lines show the overall percentage for each item

PAGE-Colo

Most colonoscopies (67%) were performed in individuals older than 50-years-old; the remainder was performed in individuals who were 41- to 50-years-old (20%) and those 18- to 40-years-old (13%). The most common indications for colonoscopy were Path A (2639 procedures [55%]) – previous rectal bleeding (34%), abdominal pain (28%), diarrhea (24%), anemia (18%), previous abnormal gastrointestinal investigation (12%) and acute loss of bright red blood per rectum (10%); Path B (1204 procedures [25%]) – patient family history (84%), patient age (44%) and media publicity about colon cancer (10%); Path C (933 procedures [20%]) – previous colonic polyps (64%), prior colon cancer (22%) and a history of colonic inflammatory bowel disease (14%).

Conscious sedation was used in 96% of patients, and 91% received both a benzodiazepine and an opiate. There were marked regional variations in endoscopy, with 89% to 97% of patients receiving some form of sedation in different provinces. Similarly, antispasmodics were used in 21% of colonoscopies overall, ranging across provinces from 0% to 55%, and oxygen was administered in 42% of colonoscopies (ranging from 11% to 87%) (Figure 6). Technically, there was also marked variation in the proportions of colonoscopies in which an assistant advanced the colonoscope (17%, ranging from 5% to 75%), counter-pressure was applied (49%, ranging from 25% to 73%) and the patient’s position was changed during the procedure (52%, ranging from 22% to 82%) (Figure 6).

Figure 6).

The percentages of procedures in which patients received no sedation, received an antispasmodic or received oxygen, in addition to the percentages of procedures in which an assistant advanced the colonoscope, an assistant applied counter-pressure or the patient’s position was changed (Practice Audit in Gastroenterology-Colonoscopy program). The triangles indicate percentages in each province (Alberta, British Columbia, Manitoba, Newfoundland, Nova Scotia, Ontario and Quebec) and the horizontal lines indicate the overall percentage for each item

Colonic preparation was reported as excellent (clean, little residue) in 51%, good (generally clean, residual fluid) in 30%, adequate (most of the mucosa visible) in 12%, poor (washing required to clear mucosa) in 6% and very poor (inadequate to complete procedure) in 1%. The cecum was visualized in 92% of procedures (ranging across provinces from 86% to 94%), including 29% of procedures (ranging from 21% to 45%) in which the terminal ileum was intubated. Ileal intubation was attempted in 35% of procedures (ranging from 16% to 62%) with a reported success rate of 83% (ranging from 56% to 100%). Features used to identify the cecum included visualization of the ileocecal valve (87%), appendiceal orifice (64%) and triangular folds (52%).

Colonoscopy was normal in 38% of cases, with polyps (27%), diverticulosis (23%) and hemorrhoids (13%) being the most common (greater than 10%) abnormalities. Colon cancer was identified in 3% of procedures.

DISCUSSION

The present study suggests that a structured, PDA-based practice audit is feasible. Of those participants who completed the evaluation section after the program, the vast majority reported that the PAGE program was beneficial, easy to use and not a burden to complete in clinical practice, and that they would be willing to complete another program. However, because fewer than half of participants completed this postaudit evaluation, these findings must be interpreted with caution.

Endoscopy and colonoscopy were selected for the first practice audit programs because these practice events are generally grouped in discrete sessions with sufficient time between procedures to allow data entry over a finite period rather than sporadically over a longer period. Additionally, endoscopy was expected to be an attractive audit focus because it is a fundamental component of gastroenterology practice. Findings indicated marked variations in practice with respect to conscious sedation, endoscopic biopsy practice, colonoscopic technique and the extent of colonoscopy, as well as between expected and reported findings. These data do not necessarily signify practice deficits, but they do allow participants to evaluate their own practice and determine whether changes are required. The prevalence of alarm features (5) was greater than in previous Canadian endoscopic studies (6,7), while esophagitis and peptic ulcers were somewhat less prevalent, probably because patients undergoing investigation in the PAGE programs were unselected.

The PAGE program was developed for PDAs for several reasons. The standardized protocol precludes the need for physicians to develop their own audit questions. The PDA-based, structured questionnaire facilitates data entry, reduces paperwork and incorporates practice audit into physicians’ increasing use of PDAs. Because they are portable, PDAs permit real-time data entry and eliminate the need for potentially erroneous data recall after the fact. The standardized protocol allows compilation of results so that physicians can compare their practice with that of their peers. Web-based data access then allows physicians to review their own audit and assess its relationship to clinical guidelines and peer practice. This is important because most physicians work alone, particularly in an ambulatory care setting. Multiple levels of security ensure confidentiality: aggregate data are accessible only by participants and designated investigators or programmers, and individual physicians’ data are accessible only to themselves.

These data present an overview of endoscopic practice in Canada, based on over 10,000 procedures, that bears comparison with previous endoscopic studies (6,7). However, the PAGE programs did not, and were not intended to, provide a detailed description of disease prevalence, and data entry was not controlled to the extent necessary for a comprehensive data repository (8). Rather, the intent of the PAGE programs was to provide a means for practising clinicians to record a representative description of their endoscopic procedures, such that they could then review their own practice and determine the extent to which it differed from that of their peers. The results of the PAGE programs clearly showed that there was considerable variation between provinces in the prevalence of different diagnoses and, more importantly, in endoscopic practice. It is therefore likely that there is even greater interindividual variation in practice. The opportunity to compare the results with aggregate national or provincial data allowed participants to determine which specific aspects of their practice differed most from the norm and should thus be reassessed. In addition to providing information to the individual participant, the PAGE programs also provided aggregate data that may indicate to professional organizations, such as the CAG, specific topics for which practice guidelines should be developed. They may also indicate whether interprovincial variations in practice are related to differences in access to specialists or the availability of other health care resources.

Although the PAGE programs were well received, several aspects of the audit process require improvement and refinement. For example, PAGE-Colo was noted to be longer and more time-consuming than PAGE-Endo, and it could reasonably be abridged. Incorporation of data entry time stamps would allow quantification of time spent on the audit. This would permit automated calculation of available CPD credits and help to determine the optimal length for future audits. Subsequent practice audits may benefit from incorporating specific target criteria to provide a standard against which participants can evaluate their performance. Participants should also have the opportunity to repeat an audit, after a suitable period of time, to determine whether evaluation of the initial audit has led to any changes in practice. Future programs may also permit needs assessments for general CPD activities, development of practice guidelines or consensus conferences with respect to, for example, the indications for H pylori testing or mucosal biopsy at upper endoscopy, the use of medications and oxygen during endoscopy or colonoscopy, and the availability of assistance during colonoscopy.

Although the direct costs of PDAs, programming, distribution, data collection, analysis and management are considerable, the traditional practice audit is also costly for the individual; however, a PDA-based approach should become more attractive as PDAs become less expensive and more widely used. The development of multiple, structured practice audits, with specific evaluation tools and educational resources, may augment current programs from professional organizations, which could then provide additional support.

In the future, this technology may allow evaluation of many aspects of practice and health resource utilization for physicians and other health care professionals from the perspective of professional organizations and health care providers. The CAG encourages other professions to consider this approach, and to begin collecting data to document and evaluate practice in their specialties. The latest PAGE program – Improving Access to Gastroenterology Services in Canada – has already reported ground-breaking findings on national and provincial waiting times between primary care referral and the receipt of gastroenterological care (9). Such data are an essential first step to support health care decisions, in line with the federal government’s directive to provide patient wait time guarantees.

In summary, PDA-based practice audit is a novel and powerful real-time learning tool for CPD that confers significant advantages and holds enormous promise for further development.