Abstract
The authors compare alternative methods of cost estimation for a patient multimedia education (PME) program, using a computerized weight-reduction PME project as an example. Data from the project planning and budgeting process and actual costs of the completed project are analyzed retrospectively to calculate three different estimates—pre-work, post-work, and actual work. Three traditional methods of estimating the cost of computer programs (the lines-of-code, function point, and task ratio analyses) underestimate costs in this example. A commercial program (Cost Xpert) that calculates the cost of developing a graphical user interface provided a better estimate, as did a tally reflecting the complexity and quality of media material in the project.
Health care professionals involved in multimedia education must design innovative, high-quality programs that are cost effective, easily modifiable, and can adapt to changes in content material and computer technology.1 It is challenging to estimate the cost of health education multimedia projects. The “literature on software cost estimation is somewhat sparse.”2 For example, Jones lists the book Software Engineering Economics3 as the sole source of detailed published algorithms for software cost estimation. Many algorithms remain largely proprietary and have been described only in general terms. Most cost estimation methods are expensive, limiting the choices for how to estimate.
Cost estimation algorithms can be manual or automated. Automated methods are generally more accurate. However, automated methods tend to overestimate costs, whereas manual methods underestimate them.2 Basic estimation programs depend directly on the cost data for accuracy. Manual methods of software cost estimation require mainly the developer's time; there are no proprietary data to purchase and (usually) no licensing fees to pay. Until the demand for automated multimedia software cost estimation increases, important background data will remain missing from estimation databases.
The manual methods fall into three general, progressively more accurate categories—lines of code, task ratios, and function point metrics. The lines-of-code method estimates how much programming (printed lines of code) is needed for the discrete tasks in a project2 and derives costs by multiplying the “total lines” by a cost factor. This method is fast but relies heavily on how the discrete tasks are defined, the programming language (i.e., object-oriented language vs. procedural language), the experience of the programmers, and the derivation of the “cost factor.”
The task ratio method assumes that every programming project can be divided into specific tasks (e.g., programming, debugging, and management).2 The proportion of effort devoted to each of these tasks creates a ratio. Different types of projects (e.g., commercial or military) have standard sets of ratios. A cost estimate is generated by choosing the appropriate set of ratios for a specific project. By defining the cost of one task in the project, the costs of other tasks and the total cost can be estimated. One advantage of this approach is that a good estimation of one task generates a more accurate estimation for the entire project. The cost estimate still depends on how discrete tasks are defined and on the program design. A change in design can lead to a different cost estimate.
The function point metric is both the most accurate and the most complex approach.2 Multiple revisions of rules and examples of how to apply the metric have helped clarify complexities, but many of the rule revision schemas are proprietary.4 A small group planning a patient multimedia education (PME) project on a limited budget could use an older and publicly available explanation of this method7 at the risk of losing accuracy. The method defines a programming project in terms of atomic “functions” common to all algorithms—data elements, movement of data, and interactions within each use—in an attempt to measure the complexity of the project.2 In addition, the function point metric provides information that can guide the design and management of the project. The impact of changes in project specifications can also be estimated.
A final impediment to the estimation of development costs for a PME project is judgment of the cost of the “media” for the multimedia project. As an extreme example, consider the difference in cost between 30 seconds of video produced for network television and 30 seconds of video produced by a developer using a hand-held video camera. The final cost estimate for the project should include consideration of the desired quality for the final product. Multimedia cost estimates have been published, but they are exclusively retrospective and do not suggest any method for making estimates prospectively.6 An alternative approach might be to estimate the cost of media to be used and then to add this to the programming and project management costs.
Method
Setting
We retrospectively analyzed the planning process for a computerized weight reduction PME project. The PME program was part of a clinical trial funded by the National Heart, Lung and Blood Institute, focusing on cost-effective approaches to weight reduction with weight loss (pounds lost), cardiovascular risk factors, and quality of life as the program's primary outcome measures. Previous experience using video PME for weight reduction suggested that computer-guided cognitive behavioral programs could be highly effective7–10 in promoting lifestyle changes while reducing overall medical costs through decreasing staff time.
Materials
The following materials were reviewed and evaluated for their relevance to calculation of PME development costs—grant proposal, grant budget information, minutes of group meetings, purchase orders, check requests, and the contract between the software company and the research institution.
Pre-existing materials incorporated into the program included a 300-page weight reduction workbook, two hours of video footage from the previous weight reduction project, clip art materials on nutrition, and 30 food slides from the American Heart Association. Additional materials included a 700-page resource book documenting each screen layout in the PME program and a record of correspondence between the software company and the research institution.
The PME end product was a 341-MB program (database 2 MB, audio 147 MB, video 123 MB, graphics 23 MB, code 46 MB). The program had a total of 671 “screens.” This program was analyzed for lines of code and amounts of video, audio, and text. A software cost estimation package called Cost Xpert (Cost Xpert Group, Jamul, California) was used to provide cost estimations.
Analysis
Three different retrospective cost estimates were used—“pre-work,” “post-work,” and “actual work.” Cost Xpert produced pre-work estimates based on the project's grant proposal, including lines-of-code and task ratio cost estimates. Cost Xpert generated post-work estimates based on the PME end product including lines of code, function points, and graphical user interface elements.
The estimate of the graphical user interface cost was based on elements (menu items, dialog boxes, windows) used in the program. The actual-work estimate was based on external costs (determined by the review of purchase orders, check requests, and consulting contracts) and internal costs (based on meeting minutes and budget information). The resource book and final project media were analyzed to measure total media used and were combined with estimates of media cost to produce an overall media cost estimate.
Results
The pre-work estimate was $187,015 for lines of code and $224,900 for the task ratio analysis. The post-work estimate used 9,213 as the total number of lines of code and was $212,790. The PME program had 21 external inputs, 17 external input files, 2 external outputs, 1 external query, and 3 logical internal tables. This resulted in an estimate of $240,470 using the function point analysis. The Cost Xpert software graphical user interface method of analysis produced an estimate of $286,135 (4 dialog boxes, 4 menu selections, 2 reports, 18 tables, and 2 windows).
Purchase orders, check requests, and contracts documented the following actual work costs—$76,000 to the software company, $1,700 for additional video development, $50 for AHA slides and clip-art books, and less than $1,000 for miscellaneous software (e.g., word processing, graphics, clip art). Therefore, the actual-work external costs totaled $78,750.
Review of 400 pages of minutes and 741 electronic mail messages revealed that there were three key people involved in project management. They acted in the following roles—consultant company liaison, 24 months; project manager, 10 months; and tester, 2 months. The percentage effort for these people was estimated as 30 percent, 100 percent, and 50 percent, respectively. Using their salaries during that time, actual project management costs were estimated to be $73,555.86. Five experts produced and reviewed new program content, two for a 24-month period and three for a 12-month period, at an estimated cost of $115,469.66. Therefore the actual-work internal costs totaled $189,025.25.
Combined internal and external estimates generated an actual-work cost of $267,775.52. These costs did not include the reuse of material from prior projects. They also did not include the costs of overtime hours, compensation hours, unpaid staff (e.g., medical students), and consultant losses. Therefore, the true cost of the PME project is greater than our actual-work cost because of these limitations in data collection. Table 1▶ provides a summary of the pre-work, post-work and actual-work cost estimates for the PME program.
Table 1.
▪ Summary of Cost Estimates for the PME Project, Compared with Actual-Work Estimate
| Cost Estimation Method | Pre-work ($) | Post-work ($) | Actual Work ($) |
|---|---|---|---|
| Lines of code | $187,015 | $212,790 | – |
| Task ratio | $224,900 | $224,900 | – |
| Function points | ID | $240,470 | – |
| Graphical user interface | ID | $286,135 | – |
| Actual-work estimate | – | – | $267,775.52 |
Note: The data in the grant proposal were insufficient to formulate a pre-work function point analysis or graphical user interface analysis. ID indicates insufficient data.
Discussion
Our findings suggest that traditional cost estimation methods (lines of code, function points, and task ratio analyses) provide underestimates of true computer PME software development costs. Pre-work estimates were much lower than post-work estimates, because more accurate information was available for the PME program. The post-work estimates for lines of codes and function points both produced underestimates of the actual costs. Because of limitations in data collection, the actual cost estimate was a conservative estimate of the program's production costs. The graphical user interface estimate appeared to be the most accurate. However, this method is proprietary and inaccessible for analysis; it does not allow PME developers to adjust estimations to suit their specific program requirements.
Table 2▶ provides a comprehensive summary of the estimated cost of producing media products, based on the PME's program resource book. Using this table along with measures of pre-existing content, desired quantity of new content, and desired media quality, researchers can estimate the cost for the media portion of their project. On the basis of these results, the PME media portion of the project would have cost approximately $113,760 to produce. Using the post-work lines-of-code estimate ($212,790), the total project cost would be $326,550.
Table 2.
▪ Range of Production Costs for the PME Project
| Media | Totals by Screen | Estimated Cost | Media Development Cost | Transfer Estimate | Estimated Transfer Cost |
|---|---|---|---|---|---|
| Text | 671 | $50/screen | $33,550 | $10/screen | $6,710 |
| Graphic (new) | 355 | $100/graphic | $35,500 | $20/screen | $7,100 |
| Graphic (pre-existing) | 137 | NA | NA | $20/screen | $2,740 |
| Audio (3 second clip/screen) | 419 | $500/min | $17,460 | $20/screen | $8,380 |
| Video (new) (10 second clip/screen) | 3 | $5000/min | $1,700 | $20/screen | $40 |
| Video (pre-existing) (10 second clip/screen) | 29 | NA | NA | $20/screen | $580 |
| Totals | – | – | $88,210 | – | $25,550 |
| Media cost | – | – | – | – | $113,760 |
| Programming cost (post-work LOC) | – | – | – | – | $212,790 |
| Grand Total | – | – | – | – | $326,550 |
Note: The table includes total number of “screens,” cost estimates to produce each form of media, total costs for media development, and estimated costs to transfer existing media into a multimedia format. Breakdowns of totals at each level of production are also included. LOC indicates lines of code.
Table 3▶ provides researchers with cost ranges for producing low-quality to medium-quality media. Low-quality media may include the use of clip-art photos, nonprofessional voice-overs, or video footage created by a standard camcorder. Medium-quality media may include professional video footage, professional audio voice-overs, and graphics that include sophisticated animation. High quality, or broadcast quality, can cost ten times more than the amounts listed.
Table 3.
▪ Cost Ranges to Produce Specific Forms of Media, Including Estimates to Transfer Pre-existing Material to a PME Project
| Media | Cost to Produce ($) | Transfer Estimate ($) |
|---|---|---|
| Text | 10–100/page | 10/page |
| Graphic (new) | 50–500/graphic | 20/graphic |
| Graphic (pre-existing) | NA | 20/graphic |
| Audio | 10–1000/min | 20/clip |
| Video (new) | 500–5,000/min | 20/clip |
| Video (pre-existing) | NA | 20/clip |
Typically, published costs for the development of PME projects use final “presentation time” as the starting point for estimation. Because of the complexity of PME projects, the presentation time may also be difficult to estimate, limiting the usefulness of this approach. It is also difficult to draw a direct comparison between estimation methods based on presentation time and our media-based method. However, both Rockley's estimates for video production ($1,000 to $6,000/min) and professional rates ($50/hr)11 agree with our estimates. Similarly, Hillman's estimates for media authoring ($90/page) and audio material ($200/min)12 fall within our specified ranges.
In conclusion, a media-based method potentially offers a more accurate estimate of the production costs than traditional approaches. Thus, it is recommended that program developers consider the complexity and quality of program media when estimating costs and budgets for PME projects. This approach to cost estimation is applicable to Web-based PME projects, in which media accounts for a large percentage of overall development costs.
This work was supported in part by grant R01 HL50372 from the National Heart, Lung and Blood Institute.
References
- 1.Freeman K, Wynn-Jones J, Groves-Phillips S, Lewis L. Teleconsulting: a practical account of pitfalls, problems and promise: experience from the TEAM project group. J Telemed Telecare. 1996;1:1–3. [DOI] [PubMed] [Google Scholar]
- 2.Jones TC. Estimating Software Costs. New York: McGraw-Hill, 1998.
- 3.Boehm B. Software Engineering Economics. Engelwood Cliffs, NJ: Prentice Hall, 1981.
- 4.Group IFPU. IFPUG Counting Practices Manual, Release 4. Westerville, Ohio; 1994.
- 5.Dreger B. Function Point Analysis. Englewood Cliffs, NJ: Prentice Hall; 1989.
- 6.Canale R, Wills S. Producing professional interactive multimedia: project management issues. Br J Educ Technol. 1995;26(2):84–93. [Google Scholar]
- 7.Wylie-Rosett J, Swencionis C, Peters MH, et al. A weight reduction intervention that optimizes use of practitioner's time, lowers glucose level, and raises HDL cholesterol in older adults. J Am Diet Assoc. 1994;94(1):37–42. [DOI] [PubMed] [Google Scholar]
- 8.Caban A, Cimino C, Ginsberg M, Swencionis C, Wylie-Rosett J. Computer-guided intervention for weight control. Proc AMIA Annu Fall Symp. 1996:978.
- 9.Swencionis C, Peters M, Wylie-Rosett J, Cimino C. Profiling weight control interventions while maximizing the use of staff time. Diabetes Spectrum. 1994;7:130–2. [Google Scholar]
- 10.Wylie-Rosett J, Swencionis C, Cimino C, et al. Computerized weight loss intervention optimizes staff time: the clinical and cost results of a controlled clinical trial conducted in a managed care setting. J Am Diet Assoc. 2001, in press. [DOI] [PubMed]
- 11.Rockley A. Planning a multimedia documentation project. Tech Commun. 1994:653–661.
- 12.Hillman D. Multimedia Technology and Applications. New York: Del Mar, 1998.