Adverse drug reaction monitoring in a secondary care hospital in South India

Abstract

Aims

To ascertain the current burden of ADRs at a Government hospital in Ooty and to assess the severity of reported ADRs and the additional financial burden associated with ADRs.

Methods

A prospective, spontaneous reporting study was conducted over a period of 9 months of inpatient admissions to the medical wards, co-ordinated by clinical pharmacists. The WHO definition of an ADR was adopted. The Naranjo algorithm scale was used for causality assessment. Confirmed ADRs were classified according to the Wills & Brown [7] method and assessed for severity and patient outcomes. The average cost incurred in treating the ADRs was calculated.

Results

Of the total of 187 adverse drug events (ADEs) reported, 164 reports from 121 patients were confirmed as ADRs, giving an overall incidence of 9.8%. This included 58 (3.4%) ADR related admissions and 63 (3.7%) ADRs occurring during the hospital stay. About two thirds of the reactions (102, 62.2%) were classified as probable. The majority of the reactions (88, 53.7%) were mild. Most patients (119, 72.6%) recovered from the incidence. The majority of the reactions were of type H (100, 61%) which indicates that they were not predictable and not potentially preventable. An average cost of 481 rupees (£6) was spent on each patient to manage ADRs.

Conclusions

The incidence and severity of ADRs documented in our study are lower than those reported in comparable populations in Western studies but more than those reported in India.

What is already known about this subject

• The benefits of adverse drug reaction (ADR) monitoring are well-known.

• Poor awareness and nonavailability of a central co-ordinating body resulted in lack of ADR monitoring in India.

• The National Pharmacovigilance Programme was recently initiated, encouraging ADR monitoring in selected centres, including our centre.

What this study adds

• This is the first study of its kind at GHQH, Ootacamund that has provided insight into the burden of ADRs here.

• The incidence and severity of ADRs documented in our study is lower than those reported in comparable populations in Western studies but more than those reported in India.

Introduction

Pharmacovigilance is an integral part of drug therapy. Still, it is not widely practiced in Indian hospitals. In various studies, adverse drug reactions have been implicated as a leading cause of considerable morbidity and mortality [1]. The incidence of adverse drug reactions (ADR) varies with studies which show incidences ranging from as low as 0.15% to as high as 30% [1–3]. Elderly and hospitalized patients are reported to be more susceptible to ADRs than the adult population (16.6% vs. 4.1%) [1]. Indian reports on ADR monitoring have been very few. This may be because ADR monitoring is still evolving here. After decades of hibernation, the need for an efficient pharmacovigilance programme was felt, the result of which was the institution of National Pharmacovigilance Programme in November 2004 [4]. Under this programme, the Central Drugs Standards Control Organization, New Delhi officiates as the central co-ordinating body under which two zonal, five regional and 24 peripheral centres have been established. The objective of this programme is to create awareness among the health professionals on ADR monitoring and to encourage a reporting culture.

Hospital-based ADR monitoring and reporting programmes aim to identify and quantify the risks associated with the use of drugs. This information may be useful in identifying and minimizing preventable ADRs while generally enhancing the knowledge of the prescribers to deal with ADRs more efficiently. The participation of pharmacists in national pharmacovigilance programmes is not a common feature [5]. The pharmacists' involvement in such programmes is seen only in some countries. In India, clinical pharmacy is still evolving and hence, pharmacists' involvement in such activities has been low. The aim of the present study was to undertake ADR monitoring in a government hospital where a clinical pharmacy programme is well established. The primary objectives included monitoring and documenting ADRs and evaluating them according to set criteria. The secondary objective was to analyze the cost burden involved in managing ADRs.

Methods

The study was conducted at Government Head Quarters Hospital (GHQH), Ooty, India. This is a 420-bedded secondary care hospital catering for the poorer sections of the society. The hospital does not have specialties and has general medical and surgical units. Like all the government hospitals in the state, this hospital also receives drugs from the Tamil Nadu State Medical Services Corporation, Chennai, which is the central body that purchases drugs and surgicals for the entire state based on the Essential Drug List of the state government. This list consists of 122 drugs in 266 formulations.

A prospective study was conducted over a period of 9 months from June 2004 to February 2005 at GHQH. The study was co-ordinated by clinical pharmacists. A spontaneous reporting technique was followed. Approval of the Institutional Human Ethics Committee and permission from the superintendent of the hospital were obtained. Informed consent was obtained from all the patients suspected of ADRs before documentation. Patients in the medical wards (one male and one female) and intensive care unit of the hospital were included in the study. Patients with intentional and accidental poisoning and patients with drug abuse were excluded from the study. Prior to the study, there was no organized pharmacovigilance programme in the hospital.

Awareness on ADR monitoring was created through clinical meetings with health and allied healthcare professionals of the hospital. Clinical pharmacists attended ward rounds with the doctors as part of the regular clinical pharmacy services. During the ward rounds, these pharmacists encouraged the doctors to report suspected adverse drug events (ADEs). Also, when clinical pharmacists suspected ADEs, they were promptly brought to the notice of the treating doctors, who, if they also felt that there could be an ADR issue, filled in the notification forms. Nurses also filled in the notification forms. However, clinical pharmacists themselves did not fill in notification forms. The ADRs were defined according to the WHO definition of an ADR. Various forms were designed for the purpose of the study. These included a notification form, a patient and reaction details documentation form, an ADR assessment form and an ADR classification form. Notification forms were kept in the participating wards. All the in-patients were assessed for ADRs during the study period. In the suspected cases, past medical/medication history of patients were collected. Patients were interviewed, monitored daily throughout their hospital stay and their medical records were reviewed. The suspected ADRs were carefully analyzed and documented. All relevant data including all drugs the patients received prior to the onset of the reaction, their respective dosage, route of administration with frequency, date of onset of reaction and the patients' allergy status (to drugs and foods) were noted. In addition the patient medication history and other comorbidities were also identified. A panel of judges comprising of three clinicians and one clinical pharmacist were formed in GHQH to evaluate the ADE reports for causality assessment to confirm ADRs. The panel met once every month to assess the ADE reports generated. The ADRs thus confirmed were classified and subjected to severity assessment. When there was a disagreement between the reviewers on confirming any particular event as an ADR, the case was discussed until a consensus was reached. In this regard, the treating doctor's remarks were more seriously considered. When consensus was not achieved, the report was designated as ‘unconfirmed’. The causality relationship between the ADR and the suspected drug therapy was assessed using the Naranjo probability scale [6]. No rechallenge was attempted in any patient.

The ADRs were classified according to the Wills & Brown classification [7] (Appendix 1). The severity of the reaction was determined according to Hartwig et al.[8] as given below:

Mild reactions which were self limiting and able to resolve over time without treatment and did not contribute to prolongation of length of stay.

Moderate ADRs were defined as those that required therapeutic intervention and hospitalization prolonged by 1 day but resolved in <24 h or change in drug therapy or specific treatment to prevent a further outcome.

Severe ADRs were those that were life threatening, producing disability and those that prolonged hospital stay or led to hospitalization, required intensive medical care, or led to the death of the patient.

Patient outcomes were reported as:

Fatal

Fully recovered (Patient fully recovered during hospitalization)

Recovering (Patient recovering, but not fully recovered during hospitalization)

Unknown (not documented after initial report in chart)

The cost incurred in managing the documented ADRs was calculated based on the total amount spent on the patients divided by total number of patients (n = 121). In the cases where the offending drug was stopped and where the treatment was continued without any change, the cost of treatment was considered as nil. All the cases which involved expenditure on drugs, laboratory tests, syringes, etc., were considered for the calculation of the cost incurred for the hospital. If the patient was transferred to the intensive care unit from the ward to which he/she was admitted in order to manage ADRs, this additional cost of care was added to the total cost. However, the hospital room rent was not included in the cost calculation as this is variable depending on the type of room in which the patient stayed. Also, physician care and nursing care were not included in the cost calculation.

All those who filled in the notification forms were sent ‘Thank you’ cards and were periodically provided with information on the ADRs generated at the hospital through printed information sheets.

Statistical analysis

Rates of ADR related admissions and ADR occurrence during the hospital stay were calculated as percentage of in-patient population treated. Student's t-test was used to compare means. For other variables, the χ² test was used. A two-tailed P value of less than 0.05 was considered statistically significant.

Results

During the study period, 187 ADE reports were received of which 164 ADRs occurring in 121 patients were confirmed. This included 58 (3.4%) ADR related admissions and 63 (3.7%) ADRs occurring during the hospital stay. There were 1682 in-patient admissions during the study period with a male to female ratio of 0.89. The overall incidence was 9.8%. Females experienced a significantly higher incidence of ADRs (78, 64.5%) than males (43, 35.5%) (P < 0.005). The male to female ratio according to occurrence of ADRs was 0.55. This trend was observed in both ADR related admissions and ADRs occurring during the hospital stay. Paediatric patients (<18 years) experienced 14 (17.3%) ADRs, followed by geriatric patients (>60 years) 23 (14.4%) ADRs and adults 84 (6.3%) ADRs (Table 1). The highest percentage of ADRs was seen in paediatric and geriatric patients, both being statistically significant when compared with adult patients (χ² with three degrees of freedom = 20; P < 0.001).

Table 1.

Demographic characteristics of the patients. These include classification of patients according to age and sex and total number of admissions vs. patients with ADRs. Percentage of patients affected due to ADRs was calculated based on the total admissions in the concerned wards in the particular category, viz. children, adults and geriatrics

Characteristics (stages)	Number of patients with ADR/Number of patients hospitalized	Number (%) (n = 121) ADR related admissions	ADRs occurring during hospital stay
Male	43/792 (5.4%)	20 (2.5%)	23 (2.9%)
Female	78/890 (8.8%)	35 (3.9%)	43 (4.8%)
Paediatric	14/81 (17.3%)	6 (7.4%)	8 (9.9%)
Adult	84/1342 (6.3%)	40 (3%)	44 (3.3%)
Geriatric	23/159 (14.5%)	11 (6.9%)	12 (7.5%)
Total	121/1682 (7.2%)	57 (3.4%)	64 (3.8%)

Details regarding classification and assessment of ADRs are given in Table 2. Classification of ADRs showed that most of the reactions (100, 61%) were of type H followed by type A (61, 37.2%). According to the Naranjo algorithm scale, 102 (62.2%) reactions were assessed to be probable, 52 (31.7%) as possible and 10 (6.1%) as definite. Severity assessment of the ADRs showed that the majority of the reactions reported were mild (88, 53.7%), followed by moderate (58, 35.4%) and severe (8, 9.5%). There were no fatal reactions. In 119 (72.6%) ADRs, complete recovery was achieved. Nine (5.5%) ADRs were classified as ‘unknown outcomes’ in which the outcomes could not be assessed as the patients sought voluntary discharge from the hospital.

Table 2.

Classification and assessment of ADRS. Classification was done according to Wills & Brown [7] and causalty assessment was done according to Naranjo et al.[6]. Outcomes were assessed according to Hartwig et al. [8]. The treatment details were assessed based on the number of patients (n = 121) where as for all other assessments, the n = 164 (total number of ADRs)

Parameter	Number (%) (n = 164)
Type of reaction#
Type A Augmented reactions	61 (37.2%)
Type B Bugs reactions	–
Type C Chemical reactions	–
Type D Delivery reactions	3 (1.8%)
Type E Exit reactions	–
Type F Familial reaction	–
Type G Genotoxicity reactions	–
Type H Hypersensitivity reactions	100 (61%)
Type U Unclassified reactions	–
Causality*
Definite	10 (6.1%)
Probable	102 (62.2%)
Possible	52 (31.7%)
Onset of ADRs
Acute (<1 h)	7 (4.3%)
Sub-acute (1–24 h)	91 (55.5%)
Latent (>2 days)	62 (37.8%)
Unknown	4 (2.4%)
Severity	88 (53.7%)
Mild	88 (53.7%)
Moderate	58 (35.4%)
Severe	18 (10.9%)
Outcomes†
Fatal	0
Fully Recovered	119 (72.6%)
Recovering	36 (%)
Unknown	9 (%)
Treatment (n = 121)
Stopped the medication	37 (30.6%)
Reduced the dose	9 (7.4%)
Added another drug	16 (13.2%)
Substituted another drug	27 (22.3%)
No change	32 (26.4%)

^#Classification according to Wills & Brown [7]

^*classification according to Naranjo et al. [6]

^†classification according to Hartwig et al. [8].

In 37 (30.6%) patients, the offending drug was stopped. The offending drug was substituted with another drug in 27 (22.3%) patients, another drug was added to relieve the symptoms in 16 (13.2%) patients and the dose was reduced to ameliorate the symptoms in 9 (7.4%) patients. No change in treatment was attempted in 32 (26.4%) patients.

The most common drugs causing the ADRs and their reaction details are shown in Table 3. Antibiotics were associated with about one third of all the ADRs reported (55, 33.5%). Ampicillin produced the highest number of reactions (18, 11.1%) followed by ciprofloxacin (16, 9.8%) and nifedipine (11, 6.7%). Rashes (35, 21.3%) were the most common ADR reported followed by oedema, itching and diarrhoea (13, 7.9% each). The organ systems affected due to ADRs are presented in Table 4. Accordingly, skin was found to be the most commonly affected organ system (56, 34.1%) followed by the central nervous (31, 18.9%) and gastrointestinal (29, 17.7%) systems.

Table 3.

Drugs most commonly Involved in ADRs and their reaction details. The drugs are presented along with their ATC codes. The classification of drugs is based on AHFS Drug Information 2005. Individual reactions reported for each drug and the total number of reactions reported for each drug is presented

Drug	ATC code	Reaction details	Total number (%) (n = 164)
Ampicillin	J01CA01	Giddiness 2, rashes 10, erythema 1, itching 4, diarrhoea 1	18 (1.1%)
Ciprofloxacin	J01MA02	Rashes 7, chest compression 1, itching 3, giddiness 1, oral thrush 4	16 (9.8%)
Nifedipine	C08CA05	Oedema 4, burning sensation 2, palpitation 2, headache 2, restlessness 1	11 (6.7%)
Cefotaxime	J01DD01	Diarrhoea 5, rashes 3, oral thrush 2	10 (6.1%)
Diclofenac	M01AB05	Rashes 4, shock 4, nausea/vomiting 1	9 (5.5%)
Atenolol	C07AB03	Fatigue 2, cough 2, oedema 4	8 (4.9%)
Theophylline	R03DA04	Tremor 5, supraventricular tachycardia 1, giddiness 2	8 (4.9%)
Salbutamol	RO3CC02	Palpitation 2, muscle cramps 2, tremor 1, dry mouth 2	7 (4.3%)
Insulin	A10AE01	Burning sensation 2, rashes 2, itching 2	6 (3.7%)
Furosemide	CO3CA01	Electrolyte imbalance 3, muscle cramps 2, gastritis 1	6 (3.7%)
Metronidazole	G01AF01	Rashes 3, melaena 1, diarrhoea 2	6 (3.7%)

Table 4.

Organ systems affected due to ADRs. The documented ADRs were classified according to the organ systems involved. The numbers represent the total number of ADRs that occurred involving the particular organ system

Organ system	Number (%) of ADRs n = 164
Skin	56 (34.1%)
Central nervous system	31 (18.9%)
Gastrointestinal	29 (17.7%)
Cardiovascular	28 (17.1%)
Eyes, ears, nose and throat	8 (4.9%)
Musculoskeletal	4 (2.4%)
Metabolic	3 (1.8%)
Haematology	2 (1.2%)
Genito-urinary	2 (1.2%)
Respiratory	1 (0.6%)

Out of 121 patients experiencing ADRs, only 43 (35.6%) incurred additional expenditure in managing their ADRs. The minimum cost incurred for managing ADRs was 3 rupees (£0.04) and the maximum cost incurred was 2550 rupees (£32). Patients with mild reactions had the lowest expenditure on managing ADRs. Eight patients in this category incurred an average expenditure of 56 rupees (£0.7) each. Twenty-five patients had moderate reactions and incurred an average expenditure of 215 rupees (£2.7) each. Ten patients had severe reactions and incurred the highest expenditure, with an average cost of 1487 rupees (£19) each. The average cost of managing an ADR at the hospital was found to be 481 rupees (£6).

Discussion

In our study 3.4% of ADRs were associated with hospital admissions. Our findings are similar to other reports generated elsewhere which estimated that 2.9–6.7% of all hospital admissions are caused by ADRs [9–13]. However, ADRs experienced by hospitalized patients gave an incidence of 3.7%, which is lower than other studies in Western populations but more than the reports generated in India and other developing countries [10–16]. Although our study used a spontaneous reporting system for ADR monitoring, the presence of clinical pharmacists in the wards and their constant encouragement might have helped clinicians and nurses to notify ADRs that resulted in better reporting than comparable studies in India.

The demographic details of our study showed female gender predominance over males, which was similar to that of other studies reported in the literature [16]. Previous studies have shown that a larger percentage of ADRs was reported from geriatric and paediatric populations which was similar to our results [17, 18]. In our study paediatric (17.3%) and geriatric patients (14.4%) experienced a higher percentage of ADRs than the adult population.

Under-reporting by doctors is well known, and in India also, the spontaneous reporting system has produced lower rates of reporting [16, 19]. Clinical pharmacy was introduced to the hospital in 1998 but the ADR monitoring and reporting programme was not introduced until 2004 because pharmacovigilance was poorly developed in our country. At the same time, as part of the routine clinical pharmacy services, ADR monitoring was done by the clinical pharmacists in the hospital without further documentation and reporting. In the present study, pharmacists were involved in ADR monitoring by way of creating awareness, documentation and assessment of the reports but did not report the suspected ADEs themselves. In addition, pharmacists also assessed the patients for ADR related issues during drug therapy monitoring and when such issues were identified, they were brought to the notice of the treating clinician for further evaluation, thus effectively addressing the problem of under-reporting. Pharmacists, of late, have been encouraged to participate in the ADR monitoring programme globally and our efforts show that it will be beneficial to involve pharmacists in such programmes in India also [16, 20].

We did not formally assess the preventability of ADRs. At the same time, we have observed a significant number of ADRs falling into the type H category which may potentially not be preventable. This may indicate that drug therapy is fairly well managed. This view is also supported by the fact that only 3.7% of the hospitalized patients had ADRs. The hospital follows the essential drugs concept and has a list of essential drugs (n = 126) based on the WHO list of essential drugs. This restricted list may also have contributed to the better understanding and therapeutic management of the patients. Also, since most of the patients are repeat patients to the hospital, their therapeutic issues are fairly well known to the clinicians.

The most common systems associated with ADRs in our study were skin and the central nervous system. This finding is consistent with many studies which have reported a higher percentage of dermatological manifestations than others [9, 19, 22, 23]. The gastrointestinal system has also been reported to be involved in the majority of ADRs [21]. In our study, this formed the third largest report on ADRs. In our study, antibiotics (55) and cardiovascular drugs (33) were the most commonly involved drug classes in ADRs. This finding is consistent with the studies reported by Murphy et al.[9] Suh et al.[21] and Prosser et al.[23]. Bordet et al.[24] reported the highest percentage for cardiovascular drugs, which was second in our study. The most common drugs involved in ADRs were old drugs such as ampicillin, ciprofloxacin, nifedipine, etc. Since the hospital uses drugs that are included in the essential drug list which does not include many recently introduced drugs, ADRs of such drugs could not be generated here.

The costs incurred in managing ADRs in our patients seem to be lower than those reported by various authors in India and elsewhere [16, 24, 25]. This may be because the room rent, medical care and nursing care were not included in the total cost incurred in managing ADRs. Also, drugs are purchased for the entire state by the government resulting in huge cost savings. It may be inferred that the patients would have incurred an expenditure of about three times the expenditure incurred at this hospital if they were treated in private hospitals.

ADR monitoring was introduced in the hospital in the year 2004. However, the programme has so far been implemented only in the in-patient medical wards of the hospital. With the encouraging support of the hospital authorities and clinicians of the hospital, we believe that it will be possible to expand the programme to other departments of the hospital in future.

Appendix I

Classification of adverse drug reactions (Wills & Brown [7])

Type A: Augmented reactions

Type A reactions are dose related actions of a medicine upon the human body, which could have been predicted based upon a knowledge of the mode of action and pharmacology of a drug or excipient. These reactions can only occur while the subject is still receiving the preparation and improve partially or completely when the causative agent is withdrawn or the dose reduced.

Type B: Bugs reactions

These are adverse reactions that rely upon promoting the growth of certain microorganisms. These type B reactions are pharmacologically predictable events, but they are not type A according to the definition used in the preceding section, since the direct and principal pharmacological action is on the bodies of microorganism rather than on the human body. Examples include sugar-containing medicines promoting dental caries, antibiotics causing overgrowth of resistant bacterial species in the intestine, broad spectrum antibiotics causing oral thrush and over use of one agent stimulating the development of resistance among a specific species of microorganism rendering further use of the agent ineffective.

Note then an infection arising as a result of drug-induced immunosuppression would not be a type B reaction. The primary adverse event in such a case would be suppression of the human immune system, which is usually a type A reaction. Infections arising as a result of this would be a secondary event.

Type C: Chemical reactions

A number of adverse reactions depend upon the chemical nature of a drug or excipient rather than pharmacological properties. They are all basically forms of chemical irritation, which makes it likely that, when exposed to the preparation, most people could experience a similar reaction. The severity of a type C reaction is more a function of concentration of the offending substance than dose. Typical side-effects in this category include extravasation reactions, phlebitis, pain at the site of an injection owing to the irritant action of a drug or excipient, acid or alkali burns, contact (irritant) dermatitis and gastrointestinal mucosa damage caused by local irritant action.

These reactions are not pharmacologically predictable, but a knowledge of the physicochemical characteristics of the causative agents may enable them to be foreseen.

Type D: Delivery reactions

A variety of adverse reactions occur as a specific consequence of the method of drug delivery. These reactions do not depend upon the chemical or pharmacological properties of the constituents of the preparation, but occur because of the physical nature of the formulation and/or the method of administration. These reactions will be heterogeneous. Methods of delivery vary and so the specific nature of the adverse reactions must also vary.

The unifying characteristic is that, if the method of delivery is changed, the adverse reaction will cease to occur. Examples include inflammation or fibrosis around implants, particles in injections causing thrombosis or blood vessel occlusion, a tablet lodging in the throat, inhaling the ‘dust cap’ of an inhaler, cough after using a dry powder inhaler, infections at the site of an injection (owing to the opening of a port of entry for bacteria) and infections due to contamination of injection solution with microorganisms.

Type E: Exit reactions

These are known as withdrawal reactions, and are a manifestation of physical dependence. It is only possible for them to occur after administration of the medicine has ceased or the dose suddenly reduced. Unlike all other adverse reactions, which typically worsen if the causative agent is continued, reintroduction of the drug will actually ameliorate symptoms. The likelihood of a reaction is linked more to duration of administration than dose. In addition, although these reactions are pharmacologically predictable to an extent, the development of withdrawal reactions is not universal. Many patients do not experience them despite continuous high dose exposure.

Type F: Familial reaction

Certain adverse drug reactions occur only in susceptible individuals with genetically determined, inherited metabolic disorders. Some of the more common familial disorders include phenyl ketonuria, glucose 6-phosphate dehydrogenase deficiency; esterase inhibitor deficiency, porphyria and sickle cell anaemia.

These reactions must not be confused with those that occur because of the normal variation in ability to metabolize a drug among the population. For example, up to 10% of the population of the western world are deficient in CYP 2D6. However, this does not make them liable to suffer unique adverse effects compared with the rest of the population.

Type G: Genotoxicity reactions

A number of drugs can produce genetic damage in humans. Notably, some are potentially carcinogenic or genotoxic. Some, but not all, teratogenic agents damage genetic material within the fetus.

Type H: Hypersensitivity reactions

These are side-effects caused by allergy or hypersensitivity. They are probably the most common adverse reactions after Type A reactions. There are many different types, but all involve activation of an immune response. They are not pharmacologically predictable, and neither are they dose related according to the definition of ‘dose dependent’ given above (although very small doses can sometimes be used for desensitization). Accordingly, reducing the dose does not usually lead to amelioration of symptoms; the drug must be stopped. Some examples are anaphylaxis, allergic skin rashes, Stevens–Johnson syndrome, photoallergy, acute angioedema, hypersensitivity, cholestasis, and hypersensitivity mediated blood dyscrasias.

Type U: Unclassified reactions

Some ADRs have a mechanism that is not understood and these must remain unclassified until more is known about them. This may necessitate the introduction of new adverse reaction categories in the future. Examples include drug induced taste disturbance, muscular adverse effects of simvastatin, and nausea and vomiting after a gaseous general anaesthetic.

We wish to thank Dr P. Chidambaranathan, Dr V.P. Karthikeyan and Dr Mahesh for serving on the panel of judges to assess ADRs. We also thank Dr Raghuram, Dr Suresh, Dr Balasubramaniam and the hospital authorities for the support and encouragement extended for this work.