Table 5.

Cross-case results validation tests.

Case	Construct validity	External validity	Reliability
1	To achieve data triangulation, the case study had multiple sources of cost data. (1) The project budget that was submitted to the project funder, (2) the actual costs submitted to the funder at the completion of the project, and (3) the timesheet log of hours captured by the course implementers. The final case report was reviewed, and feedback gathered from the course designers (BS, MT); any inconsistencies or inaccuracies were corrected.	By using Levin’s ingredients method for cost identification, the case followed an established costing procedure that is used as the basis for analytic frameworks for economic evaluation in education. This process based on a common analytic framework allows for the generalization of the study findings to similar use cases.	A study protocol was created at the commencement of the case; the protocol details the structure of the study and details how data were collected to ensure the reliability of the results.
2	Multiple sources of cost data and reporting data were used to validate that data sources were an accurate record of what occurred. (1) The project budget created at the project commencement, (2) the actual cost report submitted at the completion of the project, (3) the timesheet log of hours captured by each team resource, (4) a third-party work-log for course production and monitor of billable hours recorded charged to the program, (5) external audit reports on the course construction, and (6) review of notes from monthly reviews of budget spend. The final case report was reviewed, and feedback gathered from the course designers (BS, MT); feedback was provided and reviewed by the research team to ensure implementation accuracy.	The repetition of a model used in prior research [22], application of Levin’s ingredients method for education intervention analysis, and use of standard costing and variance calculation activity-based costing methods demonstrated a common analytic framework that is transportable to other studies.	To achieve this test, a study protocol was used and formed the governing basis for the study.
3	The data sources for each ingredient category were sourced from (1) the initial project budget, (2) reported submitted costs, (3) a time log of hours worked, and (4) a third-party work-log of the activities of subcontracted courses. The final case report was reviewed to ensure accuracy.	The same process that was used in the 2 previous cases was replicated [24], and application of Levin’s ingredients method for education intervention analysis demonstrated a common analytic framework transportable to other electronic learning studies.	A minor variation of the previous study protocols executed was used and stored as the governance framework for the study.