Verification, Validation and Sensitivity Studies in Computational Biomechanics

Abstract

Computational techniques and software for the analysis of problems in mechanics have naturally moved from their origins in the traditional engineering disciplines to the study of cell, tissue and organ biomechanics. Increasingly complex models have been developed to describe and predict the mechanical behavior of such biological systems. While the availability of advanced computational tools has led to exciting research advances in the field, the utility of these models is often the subject of criticism due to inadequate model verification and validation. The objective of this review is to present the concepts of verification, validation and sensitivity studies with regard to the construction, analysis and interpretation of models in computational biomechanics. Specific examples from the field are discussed. It is hoped that this review will serve as a guide to the use of verification and validation principles in the field of computational biomechanics, thereby improving the peer acceptance of studies that use computational modeling techniques.

1. Introduction

Accurate, quantitative simulations of the biomechanics of living systems and their surrounding environment have the potential to facilitate advancements in nearly every aspect of medicine and biology. Computational models can yield estimates of stress and strain data over the entire continuum of interest, which becomes especially advantageous for locations where it may be difficult or impossible to obtain experimental measurements. In addition, advancements in imaging techniques and geometry reconstruction have opened the door to develop and non-invasively analyze patient-specific models, which may revolutionize the way clinicians diagnose and treat certain pathologies. Finally, continuing improvements in computing hardware have allowed use of finely discretized geometries (e.g. high resolution representations of vertebral bodies [1]) and sophisticated constitutive models (e.g. cartilage poroelasticity [2,3]) with the hope that these added complexities will produce more realistic representations of biological materials.

The aforementioned positive aspects have likely been the driving force responsible for the rapid growth of the computational biomechanics field. However, model credibility must be established before clinicians and scientists can be expected to extrapolate information and decisions based on model predictions. Specifically, an analyst must convince his or her peers that: 1) the mathematical equations governing the model are implemented correctly, 2) the model is an accurate representation of the underlying physics of the problem, and 3) an assessment of error and uncertainty is accounted for in the model predictions. To accomplish these three tasks an analyst must be able to combine methodologies and data from both computational and experimental biomechanics. In other words, models should be verified and validated using a combined computational and experimental protocol.

Verification and validation (V&V) are processes by which evidence is generated and credibility is thereby established that a computer model yields results with sufficient accuracy for its intended use [4]. More specifically, verification is the process of determining that a model implementation accurately represents the conceptual description and solution to the model [5]. Validation is a process by which computational predictions are compared to experimental data (the ‘gold standard’) in an effort to assess the modeling error. Put simply, verification deals with ‘solving the equations right’ whereas validation is the process of ‘solving the right equations’ [6,7]. A plan to test a specific hypothesis or set of hypotheses with tools from computational biomechanics should include specific plans for both verification and validation in the overall design of the study, as the study design, verification and validation must be coupled (Fig. 1).

Figure 1.

Overview of the verification and validation process. Verification deals with assessing the ability of the model to solve the mathematical representation of the conceptual model correctly and can be separated into code verification and calculation verification. Validation assures that the model represents the true mechanical behavior of the physical system with sufficient accuracy. Model accuracy is assessed using quantitative comparisons between computational predictions and experimental results. The computational model and/or experiment are revised if the model is determined to be inaccurate for the intended use. Adapted from [4] with permission.

It has been argued that ‘verification and validation of numerical models of natural systems is impossible’ [8]. This line of thinking is analogous to the argument by Karl Popper [9] that, like scientific theories, correctness of model predictions cannot be proven but only disproven. To avoid this seemingly circular argument, the analyst must approach the problem by posing specific hypotheses regarding model V&V, along with appropriately chosen tolerances, and then test these hypotheses. Repeated rejection of the null hypothesis (that the model does not reproduce the underlying principles of mechanics or that the model cannot predict experimental data within some acceptable error) for tests of the model’s descriptive and predictive capabilities provides confidence in the use of the model for decision making.

There should be no doubt that proper V&V increases peer acceptance and helps to bridge the gap between analysts, experimentalists and clinicians. Appropriate V&V demonstrates if a particular model has adequate accuracy and detail for its intended use. Model V&V also allows for quantification of detection limits, which assists in determining the limits of model application and therefore prevents unjust extrapolation. If properly documented, the V&V process will provide a solid framework for future modeling efforts [4,5,10–12].

Computer simulations of physical processes have been used in the traditional engineering disciples as early as the 1950s (e.g. Monte Carlo simulation to study nuclear detonation [13]) and as early as the 1970s to model tissue biomechanics by use of the finite element (FE) method (e.g. [14–19]). The field of computational fluid dynamics (CFD) was the first to initiate formal discussions and requirements regarding V&V [5,10–12,20,21]. In 1986 the Journal of Fluids Engineering was the first to institute a journal policy statement related to V&V (but not completely encompassing the subject):

‘The Journal of Fluids Engineering will not accept for publication any paper reporting the numerical solution of a fluids engineering problem that fails to address the risk of systemic truncation error testing and accuracy estimation’.

A comprehensive text on the subject of V&V in CFD was published by Roache in 1998 [11]. Other formal work related to V&V in CFD was presented in 1998 by the American Institute of Aeronautics and Astronautics (AIAA) [5]. The latter document emphasized that only guidelines could be presented because the current state of the art did not permit the implementation of V&V standards [5]. The computational solid mechanics community was developing guidelines for use of model V&V around the same time as the CFD field. In 1999 an ASME committee was formed to institute guidelines regarding V&V from which a document was eventually published in 2006 [4].

The time lag between use of models in the traditional engineering disciplines (1950s) to that used by the biomechanics community (early 1970s) may also reflect the time delayed development of V&V policies and guidelines in the field of computational solid biomechanics. Nevertheless, discussions pertaining to V&V in this field have been actively underway [22–24]. For example, journals such as the Annals of Biomedical Engineering have instituted policies regarding modeling studies by stipulating that ‘modeling developments should conform to standard modeling practice’ and that ‘appropriate measures of variability should be specified for quantitative results based all or in part on a model and experimental data’ [24]. Recently, Viceconti published an editorial [23] that briefly discussed the importance of the V&V process in computational biomechanics. General guidelines were provided for evaluating the level of clinical utility of computational models. It was suggested that the degree of V&V performed by the analyst should be used as the primary criteria for assessing the clinical utility of a particular model. Weiss et al. discussed approaches for the verification and validation of ligament FE models, stressing the importance of sensitivity studies in the context of ligament modeling with the FE method [22].

While prior work has effectively outlined the importance of V&V [4,5,10,11,22,23,25], a review of the subject with an eye toward application to computational biomechanics has not been presented. The objective of this paper is to present the concepts of verification, validation, and sensitivity studies in the context of typical analyses in computational biomechanics. The paper focuses specifically on problems in solid biomechanics. Examples and critiques of biomechanics studies that have attempted model V&V are presented. It is hoped that implementation of these principles will improve the application range of biological simulations and peer acceptance of model predictions.

2. Accuracy, Uncertainty, and Error

Since error motivates the need for V&V procedures, it is crucial to understand the types of errors in experimental and computational studies. In the broadest sense error is the difference between a simulated value or an experimental value and the truth. Accuracy is defined as the closeness of agreement between a simulation/experimental value and its true value [12]. Therefore, accuracy and error share an inverse relationship.

Errors can be classified as either numerical errors or modeling errors. Numerical errors occur when mathematical equations are solved using computational techniques. Discretization error, incomplete grid convergence, and computer round-off errors are all examples of numerical errors [4,5,10,11]. Modeling errors are due to assumptions and approximations in the mathematical representation of the physical problem of interest [4,5,10,11]. Such errors occur due to inconsistencies between the model and physical system and include geometry, boundary conditions, material properties, and governing constitutive equations. Although discretization and geometry errors are often lumped together, they should be considered separately. Discretization error is a consequence of breaking a mathematical problem into discrete sub-problems, while geometry errors occur due to insufficient or incomplete surface or volumetric representation of the continuum of interest.

Although the terms error and uncertainty are generally associated with a loss in modeling accuracy, they should be defined separately. Uncertainty is only a potential deficiency which may or may not be present during the modeling process. Uncertainty arises due to 1) a lack of knowledge regarding the physical system of interest (e.g. unknown material data, insufficient initial and boundary conditions), or 2) the inherent variation in material properties [10]. In the latter case, sources of uncertainty can be singled out from other contributors of uncertainty by their representation as randomly distributed quantities (i.e. probability distributions) using Monte Carlo simulations [13] or they can be simulated using a known range of values by way of sensitivity analyses (see section 6). In contrast, errors are always present in the model and may be classified as either acknowledged or unacknowledged [5,10]. Computer round off errors, physical approximations (e.g. defining bones as rigid structures in joint models), and tolerances for iterative convergence are all examples of acknowledged errors. Unacknowledged errors, also known as ‘human error’, occur when modeling or programming mistakes are made [5,10].

The required level of accuracy for a particular model will depend on its intended use [4,5,10,11]. Since all experimental data have random and bias (systematic) errors, the issue of ‘correctness’, in an absolute sense, becomes impossible to address. From an engineering perspective, however, one does not require ‘absolute truth’ [10]. A statistically meaningful comparison of computational results with experimental measurements over the range of intended model use may be sufficient, assuming sources of uncertainties and errors are quantified and considered [10]. In summary, the terms ‘acceptable agreement’ or ‘accurate’ must be based on a combination of engineering expertise, repeated rejection of appropriate null hypotheses (as discussed above) and external peer review [4,5,10,11].

3. Development of the V&V Plan

The model V&V procedure begins with the physical system of interest and ends with construction of the computational model to predict the reality of interest (Fig. 2). The conceptual model is the simplified representation of the system and contains all of the partial differential equations (PDEs) and constitutive equations to describe the mechanical behavior of the continuum. Numerical algorithms are chosen and implemented to solve the mathematical equations. Finally, physical parameters (e.g., material coefficients) and discretization parameters (e.g., finite elements) are specified.

Figure 2.

Flowchart illustrating the path from conceptual to computational model. The conceptual model is the simplified representation of the reality of interest. Mathematical equations are used to describe the mechanical behavior of the conceptual model. Numerical algorithms are chosen to solve these mathematical equations and are coded appropriately. Physical parameters and discretization parameters are incorporated into the model. Adapted from [4] with permission.

Formulation of the conceptual model is the most important aspect of V&V [10]. The physical responses of interest must be captured in the computer model, so it is essential to identify a-priori which components are worthy of implementing and which are not. The Phenomena Identification and Ranking Table (PIRT) can be used to identify such key components prior to model development [10]. Factors that are considered during the development of computational models in biomechanics are similar to traditional engineering models and include boundary and loading conditions, material behavior and convergence criteria.

As discussed earlier, it is crucial to determine the intended use of the model prior to execution of the V&V plan. For example, when developing a model of the mechanics of a diarthrodial joint, it must be decided if the model will be used to predict overall displacements and joint kinematics or if localized strains and stresses are more important. Of particular importance is selection of validation experiments to complement the key response features since only a limited number of measurements can be made during these experiments. Limits in the experimental study may affect the overall applicability and utility of the computational model, not vice versa. In contrast, one may simply be interested in gaining an understanding of the potential physical a response of a system. This situation may arise in the study of systems for which there are many unknowns with regard to constitutive behavior, boundary conditions and important physical processes, such as in the study of cell mechanics. For this type of study the validation procedures may be substantially abbreviated. Rather, the investigator may focus on sensitivity studies to understand the mechanical response of the system.

A final but equally important component of the V&V plan is proper selection of validation metrics [10]. The term ‘validation metric’ must not be confused with ‘validation measurement’. Measurements are data that are recorded at the time of the experimental study whereas metrics are used after the computational model has been solved to measure differences between computational predictions and experimental results [4,10]. Although the actual implementation of validation metrics does not occur until the later part of the model validation process (see section 5.2) it is important to understand how accuracy of the model will be assessed prior to conducting the validation experiments. This will ensure that appropriate experiments are conducted and that high quality validation data are produced.

4. Verification

The American Society of Mechanical Engineer’s ‘Guide for Verification and Validation in Computational Solid Mechanics’ [4] defines verification as:

‘The process of determining that a computational model accurately represents the underlying mathematical model and its solution. In essence, verification is the process of gathering evidence to establish that the computational implementation of the mathematical model and its associated solution are correct.’

Figure 3 illustrates the fundamental attributes and flow of the verification process. Mathematical models usually consist of a set of PDEs and the associated boundary conditions, initial conditions, and constitutive equations [4,25]. Implementing the mathematical model in a computational code requires numerical discretization, solution algorithms, and convergence criteria [4,25]. By saying a FE code is ‘verified’ we simply mean that it gives the correct solution to a set of benchmark problems that consist of either analytical solutions or highly accurate numerical solutions [5]. There is no guarantee that the computational model will give accurate solutions to ‘real world’ problems [5].

Figure 3.

Flow chart of the verification procedure. During model verification computational predictions are quantitatively compared to analytical solutions, semi-analytical solutions, or numerical solutions. Adapted from [5] with permission.

A review of the literature demonstrates that, in the field of solid biomechanics, verification usually consists of implementing constitutive equations and assessing discretization error [1,26–31]. This is likely because most biomechanics based research studies use established and/or commercially available computational software for which code verification has already been completed [22,27,32–35]. However, a thorough verification process will become necessary when ‘custom’ or ‘in-house’ computational codes are developed.

4.1 Code Verification Versus Calculation Verification

The verification process is generally divided between code verification and calculation verification [4,10,25]. Code verification assesses whether the code is an accurate representation of the discretized model, whereas calculation verification determines whether the discretized model correctly represents the mathematical model [25]. Code verification is generally thought of as a software development activity [4,5,10,25] in which it is verified that there are no programming errors and that the numerical algorithms can reproduce known solutions. Calculation verification assesses the numerical errors in the simulation caused by temporal and spatial discretization error, iterative error, round-off error, coordinate transformation error, and symmetry error related to various types of boundary conditions [4,10]. Calculation verification is referred to by some authors as solution verification [10,25] or numerical error estimation [4].

4.2 Code Verification

Code verification is divided between the activities of numerical code verification and software quality assurance (SQA) [4,5,10]. Numerical code verification assesses the mathematical accuracy of the code and the implementation of the discrete algorithms for solving the PDEs [4,10]. This area of verification is where new constitutive models are implemented in computational codes (see section 4.2.1 below). SQA involves subjects such as configuration management, version control, code architecture, documentation, and regression testing [4,10].

4.2.1 Numerical Code Verification

Numerical code verification involves comparing solutions produced by the code’s algorithms to test problems for which the ‘right’ answer is known. The goal is to verify that numerical algorithms are implemented correctly [4]. In computational biomechanics, these numerical algorithms are often based on discretization with the FE method or the finite difference method. There are many issues that could lead to the code not producing the correct answer. These can include programming errors, insufficient mesh resolution to achieve the asymptotic range, mixed accuracy issues, singularities, discontinuities, contact surfaces, mesh clustering, inadequate iterative convergence, and over-specified boundary conditions [11].

4.2.2 Types of Benchmark Problems

The ASME and AIAA guides to V&V generally agree on the hierarchy of test problems to be used [4,5]. The list from highest to lowest accuracy is as follows: (1) exact analytical solutions (including manufactured solutions), (2) semi-analytic solutions (reduction to numerical integration of ordinary differential equations (ODEs), etc.), and (3) highly accurate numerical solutions to PDEs.

The most useful benchmark problems have exact analytical solutions. These are closed-form solutions to special cases of the PDEs that are represented in the conceptual model [5,10] and are either solutions to real-world physics problems or manufactured solutions to the PDEs [4]. Real-world physics problems will have realistic initial and boundary conditions. The easiest of these problems to solve will only require arithmetic evaluations of explicit mathematical expressions. Solutions to real-world physics problems often take the form of semi-analytical solutions which are not as accurate as analytical solutions and are harder to compute.

The method of manufactured solutions involves prescribing solution functions for the PDEs and finding the forcing functions that are consistent with the prescribed solution [4]. Once the prescribed solution function is inserted into the PDE, symbolic manipulation software (e.g., MACSYMA©, Mathematica©, etc.) can be used to find the derivatives. The forcing function is created by rearranging the equation such that all remaining terms in excess of the terms in the original PDE are grouped [10]. It is then added back to the original PDE so that the solution function satisfies the new PDE. The new PDE boundary conditions are either the value of the solution function on the boundary (Dirichlet condition) or a condition that can be analytically derived from the solution function (Neumann condition) [10].

Semi-analytical solutions to a set of PDEs are not as accurate as analytical solutions or manufactured solutions and either cannot be derived or are difficult to derive using symbolic manipulation software. These solutions usually consist of infinite series, complex integrals, or asymptotic expansions. When using semi-analytical solutions to perform code verification numerical error must be reduced to an acceptable level so that errors attributed to the code are not due to the solution [4].

The final and least accurate of the bench mark problems are numerical solutions to the PDEs. There are two types of numerical benchmark solutions: 1) solutions in which the PDEs have been reduced to one or more ODEs (e.g. using similarity transformations) that must be integrated numerically, and 2) solutions in which the PDEs have been solved directly by numerical methods [4]. Published numerical benchmark solutions must only be used if they meet three strict criteria: 1) the code used to produce the solution is thoroughly verified and documented, 2) a complete numerical error estimation is reported with the solution, and 3) the solution is accurately calculated by an independent investigator, preferably someone who has used different numerical approaches and computer codes [10].

4.3 Calculation Verification

Calculation verification in computational biomechanics is usually conducted through the use of mesh convergence studies, with the objective to estimate the error associated with model discretization. The literature in computational mechanics describes a priori and a posteriori methods for estimating error in a numerical solution to a complex set of PDEs [4,5,10]. A priori approaches use only information about the numerical algorithm that approximates the partial differential operators and the given initial and boundary conditions [4,10]. A posteriori error estimation approaches use all of the a priori information plus the results from two or more numerical solutions to the same problem that have different mesh densities and/or different time steps [4,10]. Thus, mesh convergence studies are an a posteriori approach to error estimation.

Discretization error is inherent in all numerical models that must discretize either the geometry of interest or the time evolution of the solution. For instance, a mesh convergence study is usually necessary to address spatial discretization error with the FE method. FE model predictions are usually ‘too stiff’ when compared to analytical solutions, and it is usually expected that mesh refinement will result in a ‘softer’ solution. Mesh convergence studies usually involve incrementally refining element discretization until parameter predictions of interest (displacement, strain, stress, etc.) asymptote [1,28–30]. It is always recommended that the intended validation parameter (e.g. strain measurements) be used as the primary criteria for determining mesh convergence [28,36]. Multiple-mesh solutions can be combined with Richardson extrapolation to establish an acceptable mesh refinement [4,5,10,25]. Mesh convergence studies do not guarantee that model predictions are accurate. Rather, they ensure that a finer discretization would likely not change the predictions significantly.

It is often assumed that solutions will be smooth for calculation verification studies. However, singularities, discontinuities, and buckling may occur. These issues are compounded in complex conceptual models, where multiple space and time scales may be important and strong non-linearities may be present [4]. An empirical approach to error estimation can be use if three or more meshes are created and a least-squares evaluation of observed convergence rates of functionals, rather than point values, is employed. Further problems can arise when there is coupling between numerical error and spatial and temporal scales. For example, when modeling the mechanics of ligaments that may buckle during load application (see section 4.4.2), insufficient mesh refinement will preclude the model from exhibiting higher modes of buckling [28]. More refined meshes may exhibit different buckling patterns, making it difficult or impossible to compare solutions. In this case, a minimum mesh refinement that exhibits a converged state of buckling should be solved first before additional refinements can be compared.

4.4 Examples of Verification in Computational Biomechanics

This section presents examples of verification of the implementation of a new constitutive model and mesh convergence studies.

4.4.1 Example of Constitutive Equation Verification

Biological soft tissues such as ligament and heart have been represented with transversely isotropic hyperelastic constitutive models [22,32,36–43]. A simple example is characterized by an isotropic solid matrix and single fiber family with the following strain energy function [31]:

$$W=F_1({Ĩ}_1)+F_2(\mathrm{\lambda ̃})+\frac{K}{2}{(\text{ln}(J))}^2$$ (1)

Here, Ĩ₁ is the first deviatoric invariant, λ̃ is the deviatoric part of the stretch ratio along the local fiber direction, and J is the determinant of the deformation gradient, F. When deformed along the fiber direction a material characterized with this strain energy will see a stress contribution from the F₂(λ̃) term (e.g. which may represent un-crimping of the collagen fibers in ligament for example) [31]. A complete description of the constitutive equation and its FE implementation can be found in Weiss et al. [31].

Ionescu et al. used the strain energy function from Equation 1 to model soft tissue failure using the material point method [43]. In this implementation, the fibers did not resist compression. Thus deformation transverse to the fibers is only resisted by the isotropic matrix (Fig. 4). To verify that the constitutive model was implemented correctly, an equibiaxial test was simulated and stresses along and transverse to the fiber direction were analyzed. An analytical expression for the stress-strain relationship was derived for this homogeneous deformation [43]. The analytical solution and computational predictions of Cauchy stress were plotted as a function of fiber and cross-fiber stretch ratios for both implicit and explicitly integrated solutions [43] (Fig. 4). The computational predictions varied less than 3% from the analytical solution [43] (Fig. 4), which indicated that the constitutive relation was properly coded and therefore its implementation was verified.

Figure 4.

Theoretical and material point method (MPM) predictions for fiber stress vs. strain during uniaxial extension for a transversely isotropic hyperelastic material representation. Separate simulations were carried out with the fiber orientation aligned with (along) the direction of extension and transverse (cross) to the direction of extension. There was less than a 3% difference between analytical and computational results using both explicit and implicit integration. Reprinted from [43] with permission.

4.4.2 Example of Mesh Convergence

Mesh convergence studies are fairly prevalent in the biomechanics literature [1,28–30,34]. For example, Ellis et al. performed a mesh convergence study on a FE model of the inferior glenohumeral shoulder ligament [28]. Fringe plots of 1st principal Green-Lagrange strains for the refined and un-refined meshes demonstrated considerable differences for the same loading conditions (Fig. 5). This finding was especially prevalent in areas of buckling where it was first necessary to establish a minimum mesh refinement to capture the general buckling behavior of the ligament (Fig. 5, top) and then increase the mesh resolution until computational strain predictions asymptoted (Fig. 5, bottom).

Figure 5.

Top panel - fringe plot of 1st principal Green-Lagrange strains for a course mesh of the inferior glenohumeral ligament complex (1650 shell elements). Model deformation is correct, but mesh induced ‘hot-spots’ are prevalent. Bottom panel - refined mesh of the inferior glenohumeral ligament complex (6600 shell elements) showing considerable differences in strains when compared to the coarse mesh, especially in areas of ligament buckling. Average strains from this final mesh were less than one percent different than a mesh with twice as many elements. Reprinted from [28] with permission.

5. Validation

Validation is the process of determining the predictive capability of computational models by comparison to experimental data (Fig. 6). The primary difference between model validation and verification is that mathematical errors (e.g. due to code implementation, discretization error, machine round-off error etc.) are not assessed during validation. Validation must always follow verification so that validation error can be isolated from verification error. The fundamental strategy of validation is the identification and quantification of error and uncertainty in the conceptual and computational model. The process can be subdivided into: 1) validation experiments, 2) validation metrics, and 3) accuracy assessment.

Figure 6.

Flow chart of the validation procedure. During model validation computational predictions are quantitatively compared to experimental data that is organized in order of increasing complexity. Adapted from [5] with permission.

Most computational biomechanics models are ‘validated’ by comparing model predictions to experimental data from the literature. This practice may be appropriate in instances where the integrity of these data can be ensured (e.g. cases where raw data are available, specific details regarding the loading and boundary conditions are given, and assessments of experimental error are reported). However, difficulties can often arise when using data from the literature for model validation, including: 1) reliance on another’s ability to gather quality experimental data, 2) difficulty in extrapolating experimental uncertainty error, and 3) gross differences in the test specimen, loading and boundary conditions. Nevertheless, use of data from the literature can be useful as a secondary means for validation.

5.1 Validation Experiments

Validation experiments are performed to generate data for assessing accuracy of the computational model [4,5,10,11,22,25]. These experiments differ from engineering tests or physical discovery experiments in that often their sole purpose is to produce data for comparison to model predictions rather than to address specific scientific hypotheses. In this regard, validation experiments may appear remedial by comparison to complex phenomenological studies. However, execution of a well-defined validation experiment will ultimately strengthen the entire validation process. If the modeling and validation experiments are conducted in a collaborating laboratory, the experimental design should be a collaborative effort between the analysts and experimentalists. This will ensure consistency between computational and experimental loading and boundary conditions.

A validation experiment should be designed to capture the essential physics of interest, including all relevant physical modeling data and boundary conditions. Investigators should consider how experimental random and systematic errors will be determined and how the accuracy of the model will be assessed (choice of validation metric). Whenever possible, measurement methods should be chosen to capture data that complements the intended use of the model. For example, if strain is the parameter of interest, then experimental strain measurements (e.g., from a video system or strain gauges) should be used. However, in some instances it may be very difficult to obtain the desired level of accuracy or resolution in experimental measurements. For example, measuring localized tissue strains in a small sample may be impossible but clamp to clamp strain may be easily obtained. The lower-order data may still be useful as part of the model validation process. In either case, additional experimental data such as measurements of global tissue displacement can and should be used to supplement model validation data to establish a higher level of model credibility. Random and systematic errors should be reported in terms of experimental data means and standard deviations [4,5,10,11].

Whenever possible, validation experiments should be designed in a hierarchical fashion. Ideally, the validation protocol will consist of unit problems, benchmark cases, subsystem cases, and finally complete systems [4,5]. Although it may be tempting to develop an experiment that captures the complex mechanics of the entire system, it will be difficult to determine which subsystem or particular aspect of the model is contributing to model error without using a hierarchical approach [4].

5.2 Validation Metrics

A validation metric is a mathematical measure of the difference between computational predictions and experimental results [4,10]. An ideal validation metric reflects all modeling assumptions and incorporates estimates of systematic and random errors in the experimental data [10].

Qualitative validation metrics provide a simple means to assess agreement between computational and experimental results. Fringed color contour plots of stresses and strains are examples of qualitative metrics. These comparisons should provide a general sense of model agreement. However, they rely on visual intuition and do not yield information regarding experimental and computational uncertainty.

Quantitative metrics are most appropriately described by way of increasing complexity and type (experimental or numerical) [10]. Deterministic metrics use graphical comparisons to show correspondence between computational and experimental results [10]. Data can be represented using bar graphs, line graphs, or scatter plots. Validation using this metric can be problematic since comparison between results still relies on a qualitative assessment. Regression analyses of scatter plot data can partially circumvent this issue. However, uncertainty and error are still not considered at the deterministic level. Experimental uncertainty metrics include an assessment of the accuracy of the input sensing device (e.g., video system or linkage to measure kinematics) and response sensing device (e.g., strain gauge or load cell) in the experimental data [10]. Sensor accuracy could be based upon manufactured stated tolerances, which would allow an analyst to conclude whether or not the computational predictions fell within the tolerance [10]. However, it is best to quantify sensor accuracy independently. The accuracy could then be reported as the mean response ± 2 standard deviations [4,10]. Numerical error metrics include an estimation of the computational error over the system response. This error can be quantified by varying the model solution methodology (e.g., explicit or implicit time integration) or individual solution parameters (e.g., penalty values for contact, convergence criteria, etc.). Non-deterministic metrics are the most comprehensive measure of agreement between computational predictions and experimental results [10]. In addition to including all aforementioned errors into the metric, computations are made using experimentally estimated probability distributions for all input quantities of interest, including material properties and experimental input parameters (e.g. range of forces measured by load cell). The computational data points are represented as a mean value ± 2 standard deviations over both the system response and the system input. This metric allows for model validation to be based on truly quantitative comparisons between experimental and computational results, accounting for both experimental and computational uncertainties and errors [10].

5.3 Accuracy Assessment

Careful development of the validation plan will assist in interpreting the validity of the model predictions since the required degree of accuracy for the intended use of the model will be specified as part of the process [4]. Statistical tests should be used to assess significance between results so that the appropriate hypotheses can be tested and conclusions can be made regarding the validity of the model. One should accept the fact that model predictions may not fall within pre-determined tolerances and that the model may not accurately predict experimental measurements, and thus may not be appropriate for its intended use [4]. In this case the analyst should re-assess the appropriateness of the modeling assumptions. Alternatively, it may be acceptable to modify the validation plan to account for such discrepancies as long as the intended use of the model is changed accordingly.

5.4 Examples of Validation in Computational Biomechanics

There are many studies in the area of computational biomechanics that have made fruitful efforts to validate computational models, especially in the area of finite element modeling. For example, FE models of hard and soft tissues have been validated using experimental joint kinematics (e.g. [44,45]), tissue strains (e.g. [32,34,41,46]), and contact pressure measurements (e.g. [47,48]).

5.4.1 Validation using Joint Kinematics

Joint kinematics can be used to construct a convenient and simple metric for validating computational models of biological joints. For example, Beiser et al. [44] obtained patient-specific knee MRI images in an unloaded (patient supine) and loaded (open MRI scan with patient in squatting position) configuration. A FE model was constructed in the unloaded configuration, passively transformed to the loaded configuration, and loaded using patient-specific muscle forces that were calculated using inverse dynamics. The model was validated by comparing the predicted location of the patella after loading in the FE model to the location obtained by segmentation of the MRI images. Further, contact area predicted by the FE model was compared with the contact area measured from the MRI images. The location of the patella was within 2.1 mm and the predicted contact area was within 2.3% of the MRI determined values, which illustrated fair agreement using joint kinematics as the basis for validation [44].

5.4.1 Validation using Experimentally Measured Tissue Strains

Studies have validated FE models of bone mechanics by comparing predicted strains to experimental measurements [34,46,49]. For example, Dalstra et al. reported the development and validation of a three-dimensional FE model of the pelvis using subject-specific geometry and material properties [49]. The FE model was validated using experimental measures of strain in the peri-acetabular region of a cadaveric pelvis. However, validation by direct comparison with subject-specific experimental measurements was not performed. Different cadaveric specimens were used for FE mesh generation and experimental tests, which limits the validity of the model predictions. In a similar study, Anderson et al. developed a subject-specific FE model of the human pelvis using CT image data and compared computationally predicted strains to those obtained from the same specimen whose cortical bone surface was instrumented with 10 tri-axial strain gauges and loaded experimentally [34]. Regression analysis of the computationally estimated vs. experimentally measured principal strains demonstrated strong correlation (r²=0.776) with a best fit line (y = 0.933x - 0.298) that was nearly identical to the line y = x (computational predictions = experimental results), which indicated excellent model agreement overall (Fig. 7, top) [34].

Figure 7.

FE predicted vs. experimental cortical bone principal strains. Top panel - subject-specific, middle panel - constant trabecular modulus, bottom panel - constant cortical thickness. For the subject-specific model there was strong correlation between predicted and experimentally measured strains, with a best-fit line that did not differ significantly from the line y=x (Experimental strains=FE predicted strains). Predicted cortical bone strains were more sensitive to cortical bone thickness than trabecular modulus. Reprinted from [34] with permission.

Studies have also validated FE models using experimentally measured soft tissue strains. For example, Gardiner and Weiss developed and validated 8 subject-specific FE knee models of the medial collateral ligament (MCL) [32]. Each knee was subjected to a varus-valgus torque at flexion angles of 0, 30, and 60 degrees. A video based strain measurement technique was used to record MCL strains at each configuration. In-situ strains were determined by transecting the ligament free from the femur and tibia following testing. Subject-specific material properties were determined for each ligament. FE predicted strains were compared qualitatively with experimental measures using fringe plots of strain and quantitatively using scatter plot data for all knees. Good agreement was noted between the models and experimental data. It was also concluded that use of subject-specific material properties did not improve computational predictions when compared to use of average ligament material properties. However, predictions that used average in-situ strains resulted in relatively poor correlations with subject-specific, experimentally measured strains. Using similar techniques, Ellis et al. investigated the effects of anterior cruciate ligament (ACL) deficiency on MCL mechanics using a combined experimental and computational approach [41]. Again, FE predictions were compared with experimental results and good agreement was noted after interpretation of the regression analysis data. It was concluded that ACL deficiency resulted in increased MCL strains during anterior-posterior loading but not during varus-valgus loading.

6. Sensitivity Studies

Regardless of whether model inputs are measured experimentally or obtained from the literature, they cannot be assumed to be free of error [22]. This is especially true when analyzing subject-specific models in the field of computational solid biomechanics since model inputs can vary substantially with donor or patient parameters such as sex, age and pathology. Sensitivity studies involve altering model inputs in an effort to gain a better understanding of their influence on model predictions [11,22].

There is some discrepancy in the literature regarding when sensitivity studies should be performed. Roache states that they should be performed only after model validation [11]. In contrast, the ASME guidelines suggest that sensitivity studies may be performed prior to model validation to elucidate the model characteristics that will be important to monitor during experimental testing, but should be revisited following model validation [4]. If model predictions were highly sensitive to a particular parameter, the appropriate validation experiment could be designed to exert more control on this input a-priori, saving a considerable amount of time and effort. We believe that sensitivity studies should be used prior to experimental testing for the reason mentioned previously, but should also be considered as an integral component of the entire validation process rather than a separate entity performed after validation. Sensitivity studies are essentially included during model validation if a non-deterministic validation metric is chosen, since inputs to these analyses are based on experimentally estimated probability distributions [10] (see section 5.2).

Besides complementing model validation, sensitivity studies may assist in identifying structure–function relationships in living systems (i.e. which biological parameters influence tissue mechanics the most) and may be used to conduct virtual experiments or parameter optimizations without having to assemble a large experimental sample. However, both of these applications assume that the model will be working within the same limits that were used during validation. Results from sensitivity studies also allow the analyst to understand how error is propagated in models that cannot be validated (i.e. patient-specific models). For example, if computational predictions are not sensitive to a given material property (over a range of reasonable values) then slight over or underestimation of this parameter as input into a patient-specific model should not result in a substantial amount of computational error. However, the model boundary and loading conditions must be similar to those applied to the validated model.

Finally, it is important to distinguish model sensitivity studies from model calibration. Calibration of a model is a process by which model inputs are adjusted (preferentially) until computational results align with experimental data. Although calibration of a model may demonstrate the ability of a model to describe data from validation experiments, it does not demonstrate its overall predictive capability [4]. Thus, calibration is not validation [4]. Sensitivity studies, on the other hand, use model inputs based on experimentally measured distributions, without preferential treatment.

6.1 Sensitivity Studies in Computational Biomechanics

Sensitivity studies are commonplace in the computational biomechanics literature (e.g. [28,32,34,36,38,47,48,50–57]. Such analyses have been particularly useful for determining how alterations in material coefficients affect model predictions. For example, the pelvis FE modeling study by Anderson et al. [34] (see section 5.4.2) assessed the influence of several experimental parameters such as trabecular and cortical bone modulus, cortical bone thickness, and bone Poisson’s ratios on FE predicted cortical bone strains. Coefficients from linear regression analysis of data for each model were statistically compared with one another to determine if altering the material parameter of interest resulted in significant changes. FE predicted cortical bone strains were highly sensitive to changes in cortical bone thickness (Fig. 7, bottom) and cortical bone modulus, but were relatively insensitive to changes in the trabecular bone modulus (Fig. 7, middle) and bone Poisson’s ratios [34]. This information clarified the structure-function relationship of the pelvis (loads were predominately carried by the cortex for the boundary conditions examined) and also provided valuable guidelines for future patient-modeling efforts (models should include position dependent cortical thickness to obtain greater accuracy).

Sensitivity studies can also be used to determine the influence of other important model inputs besides material properties. For example, Bernakiewicz and Viceconti investigated the influence of computational contact parameters such as contact stiffness, convergence norm and tolerance, and over-relaxing factors on the accuracy of FE models accounting for bone-implant frictional contact [57]. Contact stiffness and convergence tolerance were found to play a crucial role in establishing the accuracy of the FE results and it was recommended that future contact studies investigate the influence of these parameters via sensitivity studies prior to publishing results.

7. Conclusions

This paper reviewed verification, validation, and sensitivity studies as they pertain to studies in computational biomechanics. Proper model verification and validation often require a coupling of computational and experimental studies. Verification and validation are separate activities, and verification should always precede validation to ensure that errors due to model implementation can be separated from errors due to inadequate representation of the physics. Assessments of uncertainty and error should be performed for simulation and experimental outcomes to be meaningful. What is considered ‘good enough’ must be based on engineering judgment, the intended use of the model, and peer review [4,5,10,11,22,23,25].

Although commercial software developers are expected to bear the brunt of the code verification burden for studies that use commercial software packages, further model verification activities should be performed by the analyst. At bare minimum, analysts must ensure that model predictions have converged by performing a mesh convergence study. When custom codes are developed, the analysts and code developers are responsible for code verification.

In addition to careful planning and design of validation experiments, model validation requires the estimation of experimental uncertainties that are present in validation experiments. Ideally, the investigator would conduct sensitivity studies using parameter values representing either experimentally measured probability distributions or based on a range of values reported in the literature. Besides providing multiple comparisons for model validation, sensitivity studies can be used as the basis of parameter optimization studies and may provide insight to the mechanics of biological systems.

Computational models in biomechanics are sometimes developed to simulate phenomenon that cannot be measured experimentally and require model inputs that are unknown or may vary by several orders of magnitude. Interpretation of predictions from these modeling studies may appear to contradict the above-described validation process since measurements and predictions cannot be compared directly. However, if a careful and thorough verification is performed, and sensitivity studies are used to interpret the mechanical response of the model to assumed and known inputs, the model may provide valuable (albeit qualitative) insight into the mechanical behavior of a complex biological system. The limitations of any study that incorporates computational modeling must be assessed relative to the degree of model V&V to ensure that the model results are interpreted appropriately and that conclusions are reasonable [23].

Investment of time and effort in V&V will take various forms, and the cost associated with experimental validation studies may often be the greatest [4]. Although one could argue that the cost of generating experimental data for validation exceeds the value added to the computational modeling study, these added costs must be weighed against the costs of incorrect or inappropriate conclusions based on computational predictions [10].

Computational models at individual physical scales (e.g., tissue, cell, molecule) are already being extended to multi-scale analyses via sophisticated algorithms for bridging the scales (e.g., [58–62]). While these investigations may present additional and unforeseen challenges with regard to validation, one should not assume that such studies would be exempt from incorporating a plan for V&V. Although this review was tailored to V&V in computational biomechanics, the V&V procedures discussed herein apply to a wide range of studies in computational bioengineering.

Proper V&V and assessment of model sensitivity will establish computational modeling as a valid tool for investigations in the field of computational biomechanics, thereby increasing peer acceptance and effectively reducing the gaps between computational engineering, experimental biology and clinical medicine. It is hoped that this review will initiate an increased awareness of V&V procedures in the field of computational biomechanics, thereby encouraging continued growth and acceptance by the peer community of this rapidly expanding field. In addition to an understanding and appreciation by computational scientists and engineers, the editorial boards and reviewers for journals in the engineering, life science and medical fields must understand the procedures for V&V. This may require formal journal policy statements and/or detailed guidelines for reviewers.