Estimating Development Cost for a Tailored Interactive Computer Program to Enhance Colorectal Cancer Screening Compliance

Abstract

The authors used an actual-work estimate method to estimate the cost of developing a tailored interactive computer education program to improve compliance with colorectal cancer screening guidelines in a large multi-specialty group medical practice. Resource use was prospectively collected from time logs, administrative records, and a design and computing subcontract. Sensitivity analysis was performed to examine the uncertainty of the overhead cost rate and other parameters. The cost of developing the system was $328,866. The development cost was $52.79 per patient when amortized over a 7-year period with a cohort of 1,000 persons. About 20% of the cost was incurred in defining the theoretic framework and supporting literature, constructing the variables and survey, and conducting focus groups. About 41% of the cost was for developing the messages, algorithms, and constructing program elements, and the remaining cost was to create and test the computer education program. About 69% of the cost was attributable to personnel expenses. Development cost is rarely estimated but is important for feasibility studies and ex-ante economic evaluations of alternative interventions. The findings from this study may aid decision makers in planning, assessing, budgeting, and pricing development of tailored interactive computer-based interventions.

Introduction

Health professionals and behavioral researchers are engaged in developing computerized health screening promotion programs for clinic and community settings. While promising, the technology requires intense development and testing and development costs are difficult to estimate. There are few systematic studies of development cost of this or other health promotion technologies. ^1,2

Estimating the cost of computer-based intervention development is useful whether decisions about investing in health promotion are viewed from a public or private perspective. Development cost is an important part of the total investment to be assessed in an ex-ante cost-effectiveness analysis to inform the decision makers about the full consequences of moving forward with the colorectal cancer screening (CRCS) program as compared with other competing health promotion interventions that may or may not require technology development. ³ Once the intervention technology is developed, the development costs would be excluded from a societal evaluation of using it. Accessing an existing technology does not use additional social resources, and therefore does not represent a cost to society. A societal evaluation would include all costs from the time of the decision forward, to whomever they accrue. A private investment assessment would simply consider the costs and revenues that would accrue to an institution such as a group practice or an HMO. In this case, the institution may be required to purchase the technology or a license to use it and this would be part of the private cost. This is a more narrow view of the cost, but highly pertinent to an institution and the decisions regarding resource allocation. If a private developer were to market the intervention technology, the development cost estimate could serve as a guide to pricing, such that the investment plus an expected rate of return could be recouped. Implementing the technology does require resources and this would be included in economic evaluation from both the private and social perspectives. Future reports from this study will focus on the issue of cost-effectiveness of the new technology, which will include estimates of the effects of the intervention.

This paper describes and illustrates an actual-work method of cost estimation for a study to evaluate a tailored interactive computer-based intervention to increase compliance with CRCS among patients 50 years of age and older in a primary care clinic. Colorectal cancer is an important public health concern with estimates of more than 72,000 new cases and over 27,000 deaths for 2006. ⁴ Studies have shown that regular screening facilitates earlier detection of colorectal cancer and lowers mortality. ^5–8 Public health professionals have attempted to increase rates of CRCS through different types of behavioral interventions. ^9,10

To date, most studies evaluating different methods to increase CRCS have focused on patient populations, ^9,11 and most were directed at improving fecal occult blood test (FOBT) adherence. The most common strategy was to offer study participants a minimal prompt such as a letter or phone call to encourage completion of CRCS. In general, minimal or relatively impersonal forms of contact (e.g., asking individuals to pick up a FOBT or mail in a response card to obtain a kit) yielded rates of adherence between 10% and 30%. ^12,13 More personalized strategies such as a letter signed by one's own physician that included a FOBT kit in the mail-out ^14,15 or personalized mailed interventions with telephone follow-up increased adherence up to 50%. ^16–18 To date, there have only been a few computer or video based interventions to increase CRCS ^19–21 and only one of these has been shown to successfully increase completion of CRCS screening. ¹⁹ Tailored materials provide individualized feedback compared to generic materials that are used to give general information to patients. A computer algorithm is employed to select tailored messages based on information provided by the individual in real-time as they move through the interactive computer-based education session. This intervention is being developed in collaboration with The Kelsey Research Foundation. The Foundation collaborates with university-based researchers and Kelsey-Seybold Clinic (KSC) to develop research projects, patient care and patient education programs, and quality improvement initiatives. The tailored interactive computer-based health promotion intervention is currently being implemented in a randomized trial of over 1,000 patients who use primary care in the KSC. KSC is a large multi-specialty medical organization with 300 physicians that provides care to an ethnically-diverse population of over 350,000 patients at 20 clinics in Houston, Texas. KSC offers physician services in 34 medical specialties and sub-specialties and has over 1.2 million patient visits annually. The ethnic composition of the patient population is 55% White, 23% African American, 19% Hispanic, and 3% Asian. The following is a description of the process of intervention development, and a cost model for estimating intervention development costs and cost estimates by steps in the development process.

Interactive Technology

The role of computer-assisted education in inducing specific and measurable behavioral change has been documented. ^22–27 Revere and Dunbar ²² evaluated 37 studies that used computer-generated interventions designed to impact patient behavior and concluded that such interventions are effective. In a recent report, the Science Panel on Interactive Communication and Health highlighted the potential of interactive health communications and the use of new media to simultaneously improve health outcomes, decrease health care costs, and increase consumer satisfaction. ²⁸

Several studies have used computer-based interventions to provide patients with information about their risk of cancer, ^29,30 educate women about breast cancer screening, ^31,32 assist women in making decisions about breast cancer treatment, ^33,34 and assist men with decisions about prostate cancer screening and treatment. ^35–37 Overall, these computer-assisted interventions have been successful in increasing knowledge about cancer risk, screening or treatment options, and in modifying patient behavior in the desired direction. One study successfully used a computer-based video to improve knowledge and patient stage of readiness to undergo CRCS by either FOBT or sigmoidoscopy. ¹⁹ However, this intervention did not allow participants to interact with the computer program, nor was it tailored to their initial stage of readiness to undergo CRCS. No other studies have used a computer-based, tailored intervention designed to improve rates of CRCS.

Developing Tailored Interventions

Tailored interventions employ theories of health behavior to change factors that influence a person's performance of a desired behavior. ³⁸ We developed the interactive computer program using Intervention Mapping, which is a framework for systematic health promotion program planning, implementation, and evaluation. Intervention Mapping incorporates theory and empiric evidence to identify determinants of a behavior, develop intervention objectives, and select methods and strategies for an intervention. ^39,40 Through Intervention Mapping, we ensured that the Transtheoretical (stages of change) Model was incorporated throughout the program to give participants tailored information about CRCS based on their current intention to be screened. The next section provides an overview of the theoretical basis and the components of the computerized, tailored health education program.

The Transtheoretical Model was used to inform the tailored computer intervention objectives. ^41–46 Our intervention builds on the manuals and expert systems used in other successful programs, ^47–49 including mammography screening. ⁴⁶ By assessing the stage of change, processes of change, pros and cons (decisional balance), and self-efficacy, immediate feedback is given to participants regarding their current intention for CRCS, in order to motivate them to move to the next stage of change and empower them to discuss CRCS with their physician.

▶ displays the four components involved in creating and delivering a tailored, interactive computer program. Included are the source, the message channel for conveying the intervention, the receiver of the intervention, and feedback, or program elements which present the tailored messages to the receiver. These components interact in a cyclical manner.

Figure 1.

Components of a tailored messaging system.

The source for our project includes a staging algorithm that determines each participant's current intention to be screened (stage of change) for colorectal cancer. The expert system translates the participant's gender and stage of change into tailoring variables that select the appropriate intervention messages to be presented to the participant through the computer program.

The receivers are men and women, aged 50–64 years, who receive their care from KSC. Participants are asked to view the computer program before attending a wellness visit with their primary care physician. Individually-tailored “feedback” messages are generated and delivered to the participant via the message channel (i.e., the computer program) in real time as the participant interacts with the program. The computer program includes videos showing similar participants discussing their experience with CRCS and interactive exercises that provide stage-appropriate messages and support for discussing CRCS with their physician.

To evaluate the effect of the tailored computer program on the receiver, two-week and six-month surveys are administered to participants, and chart reviews will be done six months after study visits to check on completion of CRCS.

Intervention Development Process

▶ displays the five steps in developing a tailored interactive computer program. The goal of the first step was to identify the factors most likely to influence a person's motivation or ability to make the appropriate screening behavior change. This was completed through a review of current literature on CRCS, conducting several focus groups with patients from the study site and consulting with study co-investigators with expertise in cancer prevention research. Step 2 involved identifying the determinants of behavior using Intervention Mapping to create matrices that cross-referenced performance objectives with learning and changed objectives for successful behavior change using the Transtheoretical Model. This development process lasted nine months, from February 2003 to November 2003 and involved select project team members. In Step 3, the development of the program framework and messages began through flowcharting of the computer program. A patient's tailored messages were determined by their intention regarding CRCS, which was monitored throughout the program and by their gender. The flowcharting allowed the project team to make decisions about the structure of the program, such as where tailored information would be given and what theoretical constructs would be included in the program elements. This part of the process lasted nine months (August 2003 through April 2004) and ran concurrently during the end of Step 2. Project team members met regularly during Steps 2 and 3 to review and refine the matrices and the messages. Step 4 consisted of production of the program elements, including filming and editing of the role model videos and audio taping of all narration and feedback messages. An experienced producer of health promotion programs supervised this process and assisted the project director. All aspects of video and audio production were involved including: casting actors, scouting locations, supervising filming or recording, editing and post-production. This process was completed in four months from May 2004 through August 2004. Step 5a consisted of programming the computer program and combining it with the algorithms and program elements to ensure users are given the tailored information appropriate for their stage of readiness in regards to CRCS. Macromedia Flash MX 2004 Professional was used to build the program and code interactivity. The following Adobe products were also used in the development of the computer program: Photoshop for image creation and illustration, Illustrator for titles and typography, Premier for video editing, scaling and conversion, and Audition for sound editing and conversion. Microsoft Access is used to track and convert all data from the program into a database. The main computer programmer and the project director completed the majority of work in this stage of development, which lasted from August 2004 to January 2005. Step 5b consisted of testing and retesting the message channel for accuracy of message content and delivery. All project team members and the computer programming team were involved in this process. The computer program was tested in-house to identify flaws in the program and then beta tested with people who were representative of the study population to ensure that the program was accessible and understandable. In the final stages of development, an expert consultant in the Transtheoretical Model reviewed the program to confirm that the tailored messages were stage appropriate. This step was conducted towards the end of 2004 and early 2005.

Figure 2.

Steps to develop a tailored messaging system.

Cost Methodology

The actual-work estimate of the intervention development cost was based on both societal and clinical perspectives. The societal perspective includes all costs, to whomever they accrue, including the volunteer time cost. Use of the societal perspective for the base case is recommended by guidelines for economic evaluation in order to make evaluations comparable. ⁵⁰ The clinic perspective can be determined by subtracting the cost of volunteers from the total cost.

The cost estimate consists of two major components: 1) conceptualization, instrument development and planning were conducted by the research team at the University of Texas, School of Public Health (UTSPH) and the Kelsey Research Foundation; and 2) video and software development was completed by staff at Baylor College of Medicine. Micro costing (tracking resource use and their unit values) is utilized in component 1 while the cost of component 2 is based on the amount paid to Baylor for services rendered. Intervention development requires personnel time, volunteer time, supplies, equipment, and postal services. Time logs, payroll data, invoices, and project records were used to itemize, quantify, and value these resources. Space, utilities, and general administration were estimated as a percent of direct costs. The cost model applies current local prices, but all data were disaggregated so the model can be recalculated with other prices and overhead rates. Sensitivity analysis of the overhead rate, the expected life of the technology, the interest rate, and the size of the population that may use the intervention was used to assess uncertainty of the parameters.

Baylor Contract Costs

A contract was negotiated with The Center for Collaborative and Interactive Technologies (CCIT) at Baylor College of Medicine for development and testing of the computer program for the intervention. The CCIT project team included a director, a manager of web development, a lead designer and information architect, and an account manager. The program cost consists of four parts: 1) Design, development and production; 2) Internal and user testing and modification; 3) Loading, testing, installation and maintenance; and 4) project management. The end product was a 799 MB program (1 MB database, 3.9 MB executable, 56 MB audio, and 739 MB video. The graphics were incorporated into the executable. There are 22 separate “screens” but each screen may display from one to three embedded modules (with nested “screens” of their own) depending on the participant's stage of change for CRCS and point in the program (what constitutes a “screen” is variable given that participants may or may not choose to look at certain animations/barriers/activities etc.) The application was written in Macromedia (now Adobe) Flash MX and scripted in ActionScript 2.0. The cost of creating interfaces to load data into the patient record was incorporated in the Baylor contract. This cost could not be broken out from other aspects of their work.

The most time was spent on the development phase, which consisted of: developing the look and feel of the program, producing elements such as text pages, images, animation, and programming the algorithms and logic layer used to navigate the program. CCIT billed at a flat rate of $125 per hour. The hourly rate covered the contributions of a wide array of people who worked on this project either directly or indirectly as well as infrastructure costs such as administrative support, equipment, supplies, and the unexpected costs of a project. Overhead at Baylor was at 50% of direct cost.

UTSPH Micro Costs

Time Costs

The UTSPH staff including research scientists, consultants, graduate assistants, data programmers and administrative staff reported their time spent on the intervention development process. Time data and wage and benefits per hour were used to determine personnel costs. Time was recorded in daily time logs, which were collected on a weekly basis. If staff failed to record their time in the weekly logs, they estimated the time retrospectively. All task time logs were completed; however, approximately 20% of the time logs were estimated retrospectively. Hourly personnel costs were derived from annual salary plus fringe benefit rates and adjustments for productivity and time actually worked (▶). An 85% productivity rate was used to adjust salary per minute. ⁵¹ Adjustments were made for time not working, including non-intervention development meetings, vacations, and holidays. ⁵² Personnel cost data were collected from January 2003 to February 2005.

Table 1.

Table 1. Annual Salaries, Fringe Benefit Rates, and Adjusted Salary per Minute by Job Category

	Base Salary ($)	Fringe Rate	Salary and Fringe ($)	Salary per Min.	Non-intervention Related Training	Non-intervention Related Meetings	Vacation and Holidays	Annual Hours at Work	Annual Hours Available for Tasks	Prop. of Paid Time Available for Tasks	Adj. Salary per Minute
Professors	110,000.00	0.205	132,550.00	$1.06	10	32	188	1850	1572.5	0.76	$1.40
Associate professors	85,500.00	0.205	103,027.50	$0.83	10	32	188	1850	1572.5	0.76	$1.09
Assistant professors	67,500.00	0.205	81,337.50	$0.65	10	32	188	1850	1572.5	0.76	$0.86
Project director	47,712.00	0.205	57,492.96	$0.46	10	32	188	1850	1572.5	0.76	$0.61
Database manager	45,012.00	0.205	54,239.46	$0.43	10	32	188	1850	1572.5	0.76	$0.57
Consultants	500.00		500.00	$1.04			N/A				$1.04
Graduate research assistants	22,400.00	0.205	26,992.00	$0.22	10	32	188	1850	1572.5	0.76	$0.29
Administrative assistant	42,492.00	0.205	51,202.86	$0.41	10	32	188	1850	1572.5	0.76	$0.54
Research assistant (KSC ∗ )	33,000.00	0.3	42,900.00	$0.34	10	32	188	1850	1572.5	0.76	$0.45
Data programmer (KSC ∗ )	63,000.00	0.3	81,900.00	$0.66	10	32	188	1850	1572.5	0.76	$0.87
Researcher (KSC ∗ )	80,000.00	0.3	104,000.00	$0.83	10	32	188	1850	1572.5	0.76	$1.10
Post-doctoral fellow	45,000.00	0.205	54,225.00	$0.43	10	32	188	1850	1572.5	0.76	$0.57

^∗ KSC- Kelsey-Seybold Clinic

Total hours per year: 2080, Productive rate: 0.85.

Volunteer time was estimated from documentation of focus group sessions, travel time, and time spent reviewing an early paper-based prototype of the computer education program. Volunteers were assigned wage rates to represent the value of time expended on discussion of knowledge and opinion on colorectal cancer and CRCS, travel to focus group meetings, and review of a paper-based computer education program. For the volunteers who were employed, the mean hourly wage by age, gender and ethnicity of full time workers in the 2004 Current Population Survey was used in calculating the value of their time. ⁵³ For retired persons, the federal minimum wage was used. A weighted average of these two variables was used for the working volunteers with unknown employment status. MapQuest was used to calculate the travel time and distance between the volunteers' homes and the focus group location (The website is available at: http://www.mapquest.com.) Travel time was weighted by the wage rate to estimate travel time cost for volunteers. Travel distance was weighted by 2004 standard mileage rate to estimate the cost of operating a vehicle. ⁵⁴

Supply Costs

Supply costs included the materials used for the pre-pilot, pilot, and focus group documents and computer and software items. The cost for materials was extracted from project records and valued at year 2004 market prices.

Overhead

Accounting data were inadequate to fully allocate all overhead costs to specific project activities. A hypothetical overhead rate of 35% of direct costs was used to estimate total overhead costs for the intervention development; however, an overhead rate of 50% was applied to the subcontract with CCIT at Baylor College of Medicine. ⁵¹ These costs included general administration, space and housekeeping, and utilities. The cost of volunteer time was included in the overhead cost calculation because managing volunteers requires overhead costs that are associated with an organization.

Cost Amortization

Cost was amortized over the expected life of the technology. ⁵⁵ This is identical to computing a fixed rate mortgage for N months and R rate of interest. The amortized cost AC (in that case the monthly mortgage payment) is enough both to pay off the principal value of the loan plus the interest. The interest represents the opportunity cost of capital. The opportunity cost is the cost of utilizing the resources in this way rather than an alternative way that would yield the average real (adjusted for inflation) return of R% per annum (see formula below).

where

The annual amortized cost of the program (AC) was obtained by amortizing the development cost (DC) at the real rate of interest (R) over the expected life of the technology (N).

Sensitivity Analysis

The indirect cost rate is a norm that is uncertain in this application. Our base case estimate is 35 percent with a range of 30 percent to 40 percent. These values are from the preventive health care literature. ^51,56,57 In amortizing the development costs, there was uncertainty about the interest rate, the length of time before the technology becomes obsolete, and the number of patients who may utilize the intervention. We use an interest rate of 3 percent with a range of 0 to 5 percent. ⁵⁰ Given the rapidly changing health promotion field, it was assumed the technology would have a seven year life, with a range of five to ten years. To show how development cost per participant would vary, per participant cost was estimated for cohorts of 500 to 1,500 people age 50-64 years old. The base case was 1,000.

Results

Unit Costs Estimates

Personnel cost elements are shown in ▶. Adjusted wage rates ranged from $.29 per minute for graduate assistants to $1.40 for senior research scientists. Wage rates for volunteers were based on 2004 Current Population Survey. The year 2004 minimum federal wage of $5.15 an hour was the time value assigned to retired, disabled, or unemployed volunteers. Of 36 volunteers who completed the pre-pilot and pilot surveys, 13 failed to answer their employment status. They were assigned weighted average wage rates based on their gender and ethnicity.

Total and Average Cost Estimates

Intervention development cost was $328,866, of which $237,250 was direct cost. Personnel and overhead items comprised 97% of the cost (▶). The 23 personnel contributed a total of 3,733 hours to intervention development. The research scientists spent 876 hours, the consultants spent 16 hours, the project manager spent 914 hours, the programmer spent 10 hours, the graduate research assistants spent 1,806 hours, and the administrative assistant contributed 111 hours. About 14 percent of the GRA time was devoted to programming; constructing a website, designing electronic forms and database development. The 36 volunteers devoted approximately 60.5 hours participating in focus groups, discussing knowledge and opinions about colorectal cancer and CRCS and providing advice and feedback on the computer education program. The total cost of volunteer time was $1,258.

Table 2.

Table 2. Total Intervention Development Costs by Resource Category

Resource Category	Cost ($)	Percent of Total Cost (%)
Total personnel time	$227,957.15	69.32%
—UTSPH ∗∗	$149,727.82	45.53%
— Baylor estimated	$78,229.33	23.79%
Travel	$3,541.32	1.08%
Volunteer time	$1,258.18	0.38%
Supplies & equipment	$4,493.15	1.37%
Total overhead	$91,615.83	27.86%
— UTSPH	$52,501.16	15.96%
— Baylor estimated	$39,114.67	11.89%
Total	$328,865.63	100.00%

^∗∗ UTSPH-University of Texas School of Public Health at Houston

∗“Total” may not add & “percent” may not total 100 due to rounding.

Approximately 10% of the $237,250 in direct costs were incurred in the preliminary literature review and focus group (Step 1), 11% in intervention mapping (Step 2), 24% in computer program design development (Step 3), 19% in producing the computer program elements (Step 4), 33% in developing computer education program (Step 5a), and 4% in refining computer program (▶). When amortized over a seven year technology life and utilized by a cohort of 1,000 50–64 years old participants, the per-participant cost was $52.79.

Table 3.

Table 3. Total Intervention Development Costs by Intervention Step

	Project Cost	Percent of Total Cost
Step 1: Preliminary Background Research
Literature Review Time Cost - Personnel	$11,435.92	3.48%
Focus Groups Supply Cost	$1,048.93	0.32%
Focus Groups Time Cost - Volunteer	$663.44	0.20%
Focus Groups Time Cost - Personnel	$9,879.73	3.00%
Total Step 1	$23,028.02	7.00%
Step 2: Intervention Mapping
Theoretical Intervention Development Time Cost - Personnel	$8,636.26	2.63%
Intervention Development Meeting Time Cost	$11,029.84	3.35%
General Meetings Time Cost	$4,790.68	1.46%
Computer and Software Cost	$1,082.84	0.33%
Total Step 2	$25,539.62	7.77%
Step 3: Computer Program Design Development
Intervention Material Design Time Cost - Personnel	$18,305.07	5.57%
Intervention Material Meeting Time Cost	$6,617.91	2.01%
General Meetings Time Cost	$4,790.68	1.46%
Computer and Software Cost	$1,082.84	0.33%
Focus Group Supply Cost	$137.82	0.04%
Focus Group Time Cost - Volunteer	$594.74	0.18%
Focus Group Time Cost - Personnel	$2,478.86	0.75%
Photocopying Cost	$1,140.72	0.35%
Personnel Subcontract & Consultant Fee & Travel	$22,532.20	6.85%
Total Step 3	$57,680.83	17.54%
Step 4: Production of Program Component
Production	$10,181.12	3.10%
Casting	$697.59	0.21%
Filming	$4,320.37	1.31%
Editing	$10,727.04	3.26%
Permit Cost	$85.19	0.03%
Voice Talent	$4,381.51	1.33%
Miscellaneous	$56.79	0.02%
Computer Program Component Production Time Cost - Personnel	$14,107.72	4.29%
Total Step 4	$44,557.32	13.55%
Step 5a: Computer Programming
CCIT Contract	$71,562.67	21.76%
UTSPH Computer programming Time Cost - Personnel	$5,709.60	1.74%
Total Step 5a	$77,272.27	23.50%
Step 5b: Computer Program Testing
CCIT ∗ Contract	$6,666.67	2.03%
UTSPH Time Cost - Personnel	$2,505.08	0.76%
Total Step 5b	$9,171.75	2.79%
Total Direct Cost	$237,249.80	72.14%
Overhead Cost	$91,615.83	27.86%
Total Development Costs	$328,865.63

^∗ CCIT-Center for Collaborative and Interactive Technologies, Baylor College of Medicine

Sensitivity Analysis

Cost per person was most sensitive to the number of years of life of the technology and the number of patients to which it is applied (▶). When the overhead is varied between 30% and 40%, the range of the amortized cost per participant for the intervention was $49.50 to $55.27. The amortized cost per participant ranged from $38.55 to $71.81, when technology life was varied from five to ten years. A cost per participant of $47.00-56.83 was obtained when the interest rate was varied from 0 to 5 percent. When the cohort size was also tested with a range from 500 to 1,500 participants, the amortized cost per participant ranged from $35.19 to $105.57.

Table 4.

Table 4. Sensitivity Analysis of Overhead Rate, Years of Life of the Technology, Discount Rate, Productivity Rate, and Size of the Target Population

Variable	Amortized Cost per Participant ($)
Overhead
0.30	49.50
0.35	52.79
0.40	55.27
N (yrs of life of the technology)
5	71.81
7	52.79
10	38.55
R (discount rate, %)
0	47.00
3	52.79
5	56.83
Productivity rate, %
75	55.63
85	52.79
95	50.52
Cohort size
500	105.57
1,000	52.79
1,500	35.19

The total cost increased by 5.4% or $17,721 when the productivity rate is 75%, whereas the total cost decreased by 4.3% or $14,137 when the productivity rate is 95%. The amortized cost per participant for the intervention at 75% and 95% productivity rate were $55.63 and $50.52, respectively.

Discussion

Based on our knowledge, this report is the first attempt to estimate the full cost of developing a computer-generated and computer delivered tailored intervention. Researchers have estimated the cost of multi-media patient weight loss education software development, but they have not included the translation of theory into text and media for influencing patient behavior. ² While the projects are only roughly comparable, the inflation adjusted ⁵³ cost estimate for the multi-media weight loss program based on a conservative retrospective analysis of actual work was $286,000, a similar order of magnitude compared to our estimate of $329,000.

Our cost estimates are subject to several limitations. This was a case study, and results may not generalize to other interventions and locations. For example, time prices for staff were based on Houston area wages and fringe benefits. However, the methods and the order of magnitude of costs may guide other efforts to assess development costs. While the staff was provided with charts and an Internet-based system to record their time spent on various activities related to intervention development and reminded frequently of the importance of keeping accurate logs, there was no independent observation of time. When reports lagged, staff was prompted to estimate time retrospectively. Although compliance for completion of time logs was 100%, approximately 20% of staff time logs were estimated retrospectively.

Researchers have found self-reporting an easy and low-cost method to measure time data of work activity, but its inherent bias of under- or overestimating time and the burden of continuing self-reporting may lead to inaccurate data. Burke et al. compared nurse executives' total work time obtained from a self-reporting method and a time and motion method. The results showed the amount of total time is comparable between the two methods. However, the results indicated significantly fewer activities and longer mean activity time were reported when using the self-reporting method. ⁵⁸ The authors suggested identifying a selected list of activities to minimize perceptual differences among participants and the self-reporting burden. We provided a list of activities for time recording and we used total time of each activity to calculate total time cost.

Overhead and productivity rates were based on rates found in the preventive health care literature. A fully allocated cost model and direct estimation of productivity was beyond the scope of the study, given the many complex services provided in an academic health care setting. A sensitivity analysis of rates 10 percent above and below the 85 percent base rate showed little impact on the estimated development cost per patient.

Similar to mammography intervention development cost findings by Lairson et al. ¹ the majority of the cost was attributable to personnel costs and personnel costs were affected by the size and complexity of the development team. Total development cost in the current study of $328,866 is about 24 percent higher than the $264,390 cost of developing a print based computer generated tailored intervention for mammography screening. ¹ If both the CRCS and mammography screening promotion development costs are amortized over seven years for a population of 10,000 persons, the development cost per person is $5.28 and $4.24, respectively. It may also be instructive to compare these development costs to estimates of the cost of implementing screening promotion programs. For example, Chirikos et al. estimate an average implementation cost of $3.25 for FOBT for a community-based intervention for an underserved population to promote CRCS. ⁵⁹

In summary, this report presents the cost of developing a multi-media tailored CRCS promotion intervention in the context of a research project based in a large multi-specialty group medical practice. Volunteer patients were actively involved in developing and testing the program. Computer education program costs comprised the majority of the development costs and these were affected by the complexity of the program itself. The number of tailoring variables is directly related to the complexity of the program. Tracking personnel time was challenging, however it was helpful to have staff who worked on the project every day report their time each week and have everyone else report on a monthly basis. Our system was also enhanced by the use of a Web-based time log entry system that was easily accessible by all project team members. Patient participation in program development and testing was valuable compared to the relatively small incremental cost of volunteer time and transportation.

As experience and methods build, we should be in a better position to predict and retrospectively assess development costs for multi-media health education programs. This information is valuable to decision makers concerned with investment decisions and pricing of products and services.