Key Performance Indicators for the Assessment of Pediatric Pharmacotherapeutic Guidance

Abstract

Given the paucity of actual guidance provided for managing pediatric drug therapy, prescribing caregivers must be able to draw on the limited published information in pediatrics and/or guidance provided in adults with some account for expected pediatric response. Guidance for managing drug therapy in children is clearly desirable. Our objectives were to construct key performance indicators (KPIs) for pediatric pharmacotherapy guidance to identify drugs where pharmacotherapy guidance would be most beneficial. A pilot survey to assess variation in caregiver appreciation for pediatric dosing guidance has also been constructed to provide a complementary subjective assessment. Three KPI categories, drug utilization (based on hospital admission and billing data collected from 2001 through 2006), medical need, and guidance outcome value along with a KPI composite score have been proposed. Low scores are favored with respect to prioritization for pharmacotherapy guidance. The pilot survey consisted of 15 questions to assess 1) physician knowledge regarding dosing guidance, 2) attitudes toward dose modification and patient individualization, 3) the accessibility, ease of use and appropriateness of existing data stores, and 4) frequency of dosing modification, consultation of dosing compendiums and estimate of success rate in dosing guidance. Pilot results suggest that dosing guidance is generally viewed as important and that the existing resources are insufficient to guide recommendations for all drugs. While the majority of respondents check more than one resource less than 25% of the time, at least 25% of the respondents check more than one resource 25–50% of the time. The majority viewed the relevance of dosing guidance very important to the management of drug therapy. The questionnaire is being extended to the primary care centers, the Kids First Network and specialty care centers. Results will guide the development of decision support systems (DSS) that provide patient-specific pharmacotherapy guidance as part of our pediatric knowledgebase initiative. For the top 25 most utilized agents at our institution over the last 6 years, KPI score ranged from 35 (dexamethasone) to 77 (cefazolin and ampicillin) with an average score of 55. Prototype DSS for tacrolimus and methotrexate are strongly supported by the KPI scoring which ranks their selection in the top 5% of drugs on formulary. KPI metrics provide an objective means of ranking agents for which pediatric pharmacotherapeutic guidance is clearly needed.

INTRODUCTION

Pediatric pharmacotherapy can be challenging due to developmental changes that may alter drug kinetics, pathophysiologic differences that may alter pharmacodynamics, disease etiologies that may be different from adults, and other factors that may result in great variation in safety and efficacy outcomes. The paucity of information available to health care providers to derive dosing guidance in children is likewise problematic and places an additional burden on the management of pharmacotherapy within a hospital setting where polypharmacy issues prevail as well. The reauthorization of both the Best Pharmaceuticals in Children Act and the Pediatric Research and Equity Act in 2007 highlights the appreciation for the gains made to close this knowledge gap¹ and the support for the continuation of efforts to further improve child health outcomes.² The recent adoption of the Paediatric Investigation Plan proposal by the European Medicines Agency also indicates that this is a global perspective that will hopefully broaden research efforts and expedite the closure of these knowledge gaps.³

Getting from label to practice is not a foregone conclusion, however. An important final step in this evolving process will be the mechanism by which such information is conveyed to the pediatric prescribing community and ultimately the patient. Improving drug monographs with pediatric data alone will be inadequate in this regard and practical guidance on the relationship between pharmacotherapeutic management and outcomes must ultimately be derived by caregivers relying on their own experience in conjunction with “tools” which incorporate current clinical pharmacologic knowledge regarding drug therapy in children and that have the potential to “learn” from accumulated, individual patient histories. In attempt to address many of these issues we are developing a Pediatric Knowledgebase at The Children's Hospital of Philadelphia (CHOP).⁴ Our goals for the Pediatric Knowledgebase include: 1) to provide dosing guidance consistent with formulary standard of care, 2) to examine patient pharmacotherapeutic indices with respect to individual agent performance relative to historical controls derived from the hospital data warehouse, 3) to explore treatment— diagnosis—drug correlation in conjunction with utilization, and 4) to educate health care providers on clinical pharmacologic principles specific to populations and drug combinations of interest. Static compendial information (Lexi-Comp, Physician's Desk Reference, etc.) can be searched, indexed and summarized for easy viewing; forecasting of relevant drug exposure or clinical markers (laboratory values, pharmacodynamics, adverse events) is made available in the “Drug Dashboard” module. Drug dashboards are designed for and by the physician therapeutic area in collaboration with clinical pharmacology and information technology.

Key Performance Indicators (KPI) are generally thought of as metrics (usually financial) used to help an organization define and measure progress toward organizational goals. In our setting we are attempting to define a multimetric KPI score to focus the prioritization of decision support systems to address the most critical deficiencies in the management of drug therapy for children. Our approach to the definition of the KPI scoring variable was based on the methodology described by Blocksom⁵ and is guided by the need to recognize that prioritization for pharmacotherapy guidance needs to reflect several components and be as objective as possible. The goals for the research presented herein were to define and develop a KPI score to be used as a baseline assessment for pediatric pharmacotherapy outcomes. A survey to assess the appreciation for pharmacotherapy in general as well as the necessity of dosing guidance and modification among our prescribing community was also constructed to support the requirements for additional resources to provide pharmacotherapy guidance and serve as a complementary subjective assessment of pharmacotherapy prioritization. The defined KPI will also provide a measuring stick by which the benefit of such drug dashboards can be judged once implemented.

METHODS

Questionnaire Development

A survey consisting of 15 questions was constructed to assess 1) physician knowledge regarding dosing guidance, 2) attitudes toward dose modification and patient individualization, 3) the accessibility, ease of use and appropriateness of existing data stores, and 4) frequency of dosing modification, consultation of dosing compendiums and estimate of success rate in dosing guidance. The questionnaire was developed with the Survey Monkey software to make the questionnaire available in a web-based format (http://www.surveymonkey.com/Default.aspx). The survey questions were developed from informal interviews with physician, nursing and pharmacy stakeholders that interface with the Division of Clinical Pharmacology & Therapeutics at the Children's Hospital of Philadelphia. Survey categories included demographics, pharmacotherapy resources, dosing adjustment and modification, and valuation of additional tools to provide improved pharmacotherapy guidance. Table 1 contains the various survey questions and response options developed from our initial interviews.

Table 1.

Pilot questionnaire design features to assess pediatric caregiver prescribing attitudes and opinion regarding pharmacotherapy management

The pilot survey was submitted to 30 testers representing the various stakeholders comprising the pharmacotherapeutic caregiver community at our institution. The intention of this pilot survey was to gauge the appropriateness of the questions to this community, uncover any potential bias in the questions or response options and assess the relevance of the initial results to define the baseline needs assessment for our pediatric knowledgebase project. Post survey interviews were conducted with initial stakeholder team that provided input into the pilot questionnaire design to review the survey output, assess the performance of the questionnaire and suggest changes before creation of a final questionnaire to be submitted to a broader prescribing community within our institution. Survey output was summarized in Microsoft Excel and plotted in Graphpad Prism.

Drug Utilization

Drug utilization has been assessed based on the review and summarization of hospital admission and billing data collected from 2001 through 2006. From the transaction data provided from the Siemen's accounting database, we have constructed a separate utilization database, which contains 12,656,008 records and 85 fields (approximately 81 tables, 3393 MB Oracle 8i database). Relevant fields of interest for query and data summarization include Disposition, Admission, Service, Physicians, Procedures, Treatment, Patient, Nursing Station, and Diagnosis. From this database, drug exposure can be calculated as total drug administered and as administration per patient encounter. The entire CHOP formulary consists of 998 drugs with 7,553 formulation-dose entries (unique formulations and dosage strengths) at present. As part of the continuous review of the therapeutic standards committee, the formulary is periodically re-evaluated and drugs are added or removed from the list. Hence, utilization rankings over the 6-year interval were filtered by the current formulary list.

Utilization stratification across the evaluation period and within therapeutic area and age strata has been summarized. Each inpatient or outpatient visit/admission was considered a patient encounter and, therefore, patients that were hospitalized more than once during each year were included for each separate encounter. Likewise, patient encounters were summarized with age strata and correlated with diagnosis to assess the association of drugs with therapeutic areas. The assignment of therapeutic area has been based on the most common indication for which the drug is utilized; drugs have not been cross-listed in this analysis across multiple areas in which an individual agent may be utilized. Utilization data was queried directly in Oracle. Data summarization (merging, counting, quintile assignment and ranking) was performed using SAS version 9.1 (PC/Windows platform). Graphical presentation of utilization data and trend analysis over time was performed using S-PLUS and SAS.

Assigning Attributes for Prioritization, Scoring and KPIs

Key Performance Indicators (KPI) proposed to assess the value of targeted pharmacotherapeutic intervention for drugs administered to children have been defined in three categories: drug utilization, medical need, and guidance outcome value. Within each category a set of attributes has been defined from which each drug candidate can be scored to measure the comparative subset and composite scores against each drug evaluated. Attribute definitions and the scoring criteria within each category are provided in Table 2. These raw sets of values (indicators) are combined into category scores for drug utilization, medical need, and guidance outcome value, respectively, and an overall KPI score which is the transformed sum of the various category scores. In the proposed scoring system, category and KPI scores which are low in number reflect drugs for which the value of pharmacotherapeutic guidance is high. Likewise, higher scores are consistent with drugs that are well managed, understood and/or not extensively utilized in children in general. The equation for KPI score is shown below:

Table 2.

Attributes of KPIs to Assess Pediatric Pharmacotherapy Guidance

The component attributes are all unitless numbers so transformation of conversion is not required. The scaler multipliers for Medical Need and Guidance Outcome are intended to equalize the range of the maximum scores for each of the three categories so that the KPI composite reflects equal weight for each category. As the various component attributes are newly proposed as well, their distributional properties have not been previously characterized. As Blocksom has pointed out, keeping the variation in response of the component metrics similar should improve the repeatability and reliability of the composite metric.⁵

Medical need and drug utilization are relatively straightforward in their meaning. Within the Medical Need category, the first three attributes (life saving intervention, treatment options, and pediatric dosing guidance availability) represent ordinal response variables on a scale of 0 to 5. The remaining 4 attributes in this category (titration to effect, subtherapeutic exposure, supratherapeutic exposure and chronic toxicity) are dichotomous variables scored as 0 or 1 (present or not). The range of scores in the Medical Need category will span 0 to 19 which will be scaled to 0 to 38 within the KPI score as illustrated by the equation above. Utilization criterion are based on actual patient exposures to drug over a 6-year period (2001 – 2006) and are defined for the entire inpatient population, within therapeutic area classes and subset into 6 different age strata (neonates, infants, children, adolescents, young adults and adults). The yearly and total (6-year) patient encounters for each of these age strata are shown in Table 3. For each attribute, drugs are ranked based on utilization within each stratum. Each of the utilization attributes is scored from 1 to 5 based on the quintile of the ranks within each category. The range of scores in the Utilization category will span 7 to 35 and is not scaled within the KPI composite score.

Table 3.

Patient encounters stratified by age group at the Children's Hospital of Philadelphia from 2001 through 2006

Guidance outcome value refers to the extent to which clinically relevant dosing guidance can be derived for target agents. Within the Guidance Outcome Value category, the 6 attributes (adult activity concentration-effect [C-E] relationship, adult toxicity C-E relationship, pediatric activity C-E relationship, pediatric toxicity C-E relationship, adult TDM, pediatric TDM) are all dichotomous and are identified based on the existence of these relationships or monitoring practice based on the Lexi-Comp compendium (http://www.lexi.com/) or the published literature. The assignment of TDM attributes were not based on CHOP practices per se but the recommendation to monitor drug or biomarker exposure within Lexi-Comp.

RESULTS

Survey of Attitudes on Pharmacotherapy in Children

The complete results of the 15-question pilot survey are contained in the Appendix. As mentioned previously, the survey was distributed to 30 testers, including 23 attending physicians, 5 fellows, one clinical pharmacist and one nurse. Of the testers surveyed, most (56.7%, n = 17) were from Oncology. The remainder of the survey community represented Critical Care (17.7%, n = 5) and Allergy and Immunology (6.7%, n = 2). Six other therapeutic groups / areas of specialization were represented by one person each. Most of the respondents (96.7%, n = 29) work in the hospital inpatient campus as opposed to our specialty care centers and outpatient Kid's First Network.

Survey results describing pediatric caregiver response to pharmacotherapy resources, the value of existing, available compendial resources, the frequency with which dosing guidance is sought by the caregiver and the relevance of dosing adjustments to individual caregiver practice are presented in Figure (panels A through D, respectively). As the results suggest, survey respondents use a diverse array of compendial sources to provide guidance on pediatric pharmacotherapy with the online Lexi-Comp application being the most popular resource across the various caregiver communities. These resources were more commonly referred to as somewhat informative as opposed to very informative. In addition, while the majority of respondents check more than one resource less than 25% of the time, at least 25% of the respondents check more than one resource 25–50% of the time. Not surprisingly, the majority of respondents viewed the relevance of dosing guidance very important to the management of drug therapy for patients in their care.

Figure.

Pilot questionnaire results describing pediatric caregiver response to (A) pharmacotherapy resources, (B) the value of existing, available compendial resources, (C) the frequency with which dosing guidance is sought by the caregiver and (D) the relevance of dosing adjustments to individual caregiver practice.

SCM = Sunrise Clinical Manager, PDR = Physician's Desk Reference.

▪ Attending physician Fellow physician □ Clinical pharmacist Nurse

Drug Utilization

The summarization of utilization results across age and therapeutic area strata are presented in Tables 4 and 5. Table 4 shows the ranking of agents based on administration exposure across the 2001 to 2006 evaluation period hospital wide (overall) and within age strata. As the table indicates, the ranking of agents is not the same within the various age strata. While there are agents which are clearly highly utilized across all age strata (e.g., acetaminophen, midazolam, etc.), others are unique with respect to their utilization pattern within a particular age strata (e.g., nystatin for neonates and infliximab for young adults). Of course, some of these trends are due to the nesting of therapeutic areas within the age strata and the overall utilization pattern for our institution which also reflects local / regional therapeutic bias based on the patient indications for which the institution is most established in treating. It is also clear that the utilization exposures are strongly influenced by the admissions within each age strata as shown in Table 3; the ranking within age strata for KPI scoring was an attempt to reflect the percentage of utilization within age grouping.

Table 4.

Drug Utilization Ranking Assessed by Administration Exposure from 2001 Through 2006 Hospital Wide (Overall) and Within Age Strata

Table 5.

Utilization outcomes for top 5 agents (based on total utilization over the 6 year observation period) within therapeutic area from 2001 through 2006

Table 5 shows the utilization outcomes for the top 5 agents (based on total utilization over the 6-year observation period) within therapeutic areas from 2001 through 2006. Only the therapeutic areas representing the top 25 most utilized agents are shown, though the assessment and ranking were completed for all drugs on formulary. These data also show time-dependent utilization patterns which may reflect experience/confidence with any agent, the introduction of a newer agent with perceived benefit or superiority or simply a reflection of changes in the density or admission of particular patient population. Most of the top agents within the therapeutic area sub-strata are remarkably consistent in their utilization over this 6-year observation period. These utilization data strongly support the decomposition of utilization into attributes which more appropriately reflect the dispersion in patterns across age and therapeutic area strata.

KPI Metrics

Table 6 shows the results of the various medical need, utilization, and guidance outcome value category and KPI scoring. The KPI scores for methotrexate and tacrolimus are shown for reference as they represent drugs for which prototype decision support systems have been developed. In the list of top 25 agents, the KPI score ranges from 35 (dexamethasone) to 77 (cefazolin and ampicillin) with an average score of 55 (standard deviation, 11.8). By contrast, the KPI scores for methotrexate and tacrolimus were 21 and 26, respectively, and rank among the top KPI scores based on our criteria. Based on overall utilization ranking, methotrexate ranked 52 and tacrolimus ranked 99 which still put these agents in the top third of the 345 drugs on formulary included in this analysis.

Table 6.

KPI scores for top 25 prescribed agents (overall exposure) at the Children's Hospital of Philadelphia relative to agents for which decision support systems have been developed

While more formal sensitivity analyses of the criteria are ongoing, it is obvious that the Guidance Value category scoring which has zero as a lower limit can strongly influence the KPI scoring as can be shown from the tacrolimus and methotrexate values. This was an intentionally derived property as it was felt that, while balanced scoring across the categories was desirable, a data driven bonus for agents for which guidance value was highest should be made.

DISCUSSION

Increased appreciation of the need to propose and implement child health quality measures is slowly evolving. While Shaller⁶ has recently provided a perspective on the state of practice in this area, the major issues and challenges as well as the obstacles cited in this interview-based analysis still exist. Of the top 5 needs that Shaller identified in this analysis, the investment in building research capacity via a trained pool of users of quality measures and the capacity to use and understand quality improvement methods and tools requires the dedicated engagement of the end-user community.

The benefit of surveys to provide baseline assessment of performance initiatives around hospital informatics has been previously demonstrated.^7–9 Our pilot survey is now being modified for submission to our broader caregiver community with the main hospital, specialty care centers and outpatient (Kid's First) network. Based on post-survey interviews with our initial survey development team, it was clearly viewed as a priority that we expand the survey access to include broader representation of our caregiver community ensuring that the final survey elicits greater participation from the nursing and pharmacy community at CHOP. The pilot survey was largely representative only of our physician community with 28 of 30 test responders being either an attending physician or fellow. Recent studies by Beuscart-Zephir¹⁰ and Handler¹¹ examining differences in caregiver attitudes to computerized physician order-entry systems suggest that such differences may extend to dosing and pharmacotherapy guidance as well. We also intend to expand our survey questions in order to obtain more detail on the satisfaction with existing compendial resources and the identification of drugs and drug classes which are most difficult to manage. Our intention is to compare the prioritization for pharmacotherapy guidance suggested from our survey community with the KPI score in order to evaluate the overlap between subjective and presumably more objective measures.

While the proposed KPI scoring system was envisioned to provide an objective means of prioritizing medicines for which more extensive pharmacotherapy guidance is warranted, we also recognize the need to further refine the KPI proposal to assess the sensitivity of the KPI score to the various sub-attributes. The need to prioritize is valid for many areas of drug research as well as project planning and resource allocation. Particularly in the area of pediatric drug development and clinical pharmacologic investigation, prioritization has been the goal of the NICHD, the FDA and law makers seeking to address the gaps in drug knowledge in children. Prioritization methods put forward have been largely based on sequential filtering criteria and have been viewed as highly suggestive. Our KPI approach may offer a more objective alternative and, with minor modification, could be applicable to other areas in which prioritization of pediatric pharmacotherapy research and development may be valuable. Likewise, it would also seem to be relevant for other institutions seeking to develop decision support systems to manage pharmacotherapy in children. Ongoing efforts include the further investigation of the KPI components with exploration of the sensitivity of the KPI score to the various sub-attributes.

One of our goals for the KPI scoring was to facilitate the identification of drug candidates for development of drug dashboards within our pediatric knowledgebase initiative.⁴ These drug-specific decision support systems are envisioned to provide patient-specific pharmacotherapy guidance based on the integration of hospital-based electronic medical records systems with data visualization tools, forecasting algorithms and drug-specific clinical pharmacologic guidance. Our intention was to create a balanced approach to the selection of these agents so that their selection was not entirely based on either medical need or utilization. Initial efforts in this area have produced prototype dashboards for methotrexate and tacrolimus.⁴ While their selection as dashboard candidates was not based on the KPI score, the KPI scoring system does support their selection, as both drugs are ranked higher than the top 25 most utilized drugs at our institution. We are also in the process of externally qualifying this scoring criterion via external collaborators. Drug utilization is most certainly geographically and population dependent and is likely influenced by other factors as well.¹² Likewise, medical need, while presumably defined via objective, quantitative attributes, would yield different results if applied to countries where access to medicines and disease conditions differ from the United States.¹³

There exists a heightened awareness for pharmacotherapy guidance and the development of algorithms across therapeutic areas and pediatric indications. Excellent examples for both pharmacotherapy management and patient outcome benefit exist in the areas of pediatric immune thrombocytopenic purpura,¹⁴ hyperlipidemia in pediatric heart transplant recipients,¹⁵ pediatric migraine,¹⁶ lung disease,¹⁷ insomnia in primary care,¹⁸ and bipolar disorder.¹⁹ It is also clear that facilitating the guidance discussed in these examples would be greatly enhanced with an integrated, hospital informatics solution. Our emphasis with the pediatric knowledgebase is to build such a system. Future emphasis for this project will focus on broadening collaborations with global thought leaders in this area and increasing the pool of informed end-users who would both benefit and utilize such a system. The proposed KPI scoring system should provide a means of focusing efforts in this area.

ABBREVIATIONS

CHOP

Children's Hospital of Philadelphia

DSS

decision support system

KPI

key performance indicator

TDM

Therapeutic Drug Monitoring

APPENDIX, Pilot Questionnaire Results, n = 30

1. What is your status as a clinical caregiver?

   

2. What is your training expertise (Area of specialization, e.g., Neonatology, Pediatric Oncology)?

   

3. What is your location within the CHOP system?

   

4. How do you currently obtain dosing information when prescribing a medication?

   

5. How informative are the existing dosing compendiums?

   

6. How often do you have to check more than one source to obtain the dosing guidance you require?

   

7. How important is dose adjustment in the care of your patients?

   

8. The criteria upon which you would scale adult doses or patient groups in which you would make dosing adjustments?

   

9. How convenient is it for you to obtain information pertaining to dosing guidance?

   

10. How often do you modify or are asked to modify dosages beyond the standard dose requirements for your patients?

    

11. Can you give an estimate of how effective (success rate) is your dose adjustment history?

    

12. Do you use any tools (software, calculator, etc) that allow you to check dose requirements? If so what tool(s)?

    

13. Would it be valuable for you to have an online tool to provide dosing guidance that would let you individualize patient dosing and examine dosing history relative to best practices?