Constrained Fourth Order Latent Differential Equation Reduces Parameter Estimation Bias for Damped Linear Oscillator Models

Abstract

Second order linear differential equations can be used as models for regulation since under a range of parameter values they can account for return to equilibrium as well as potential oscillations in regulated variables. One method that can estimate parameters of these equations from intensive time series data is the method of Latent Differential Equations (LDE). However, the LDE method can exhibit bias in its parameters if the dimension of the time delay embedding and thus the width of the convolution kernel is not chosen wisely. This article presents a simulation study showing that a constrained fourth order Latent Differential Equation (FOLDE) model for the second order system almost completely eliminates bias as long as the width of the convolution kernel is less than two thirds the period of oscillations in the data. The FOLDE model adds two degrees of freedom over the standard LDE model but significantly improves model fit.

Introduction

One of the fundamental principals of life is that it undergoes change. Mathematically, change as an outcome has long been represented by differential equations (see, e.g., Butcher, 2016; Hotelling, 1927; Hubbard & West, 1991; Laplace, 1779). Dynamical systems are a useful way to think about differential equations as applied to many types of biological (Glass & Mackey, 1988), psychological (Nesselroade, 2002; Boker, 2002), social (Laurenceau, Barrett, & Pietromonaco, 1998), emotional (Butler, 2017), cognitive (Port & vanGelder, 1995), and developmental (Smith & Thelen, 1993) phenomena. In the past two decades there has been rapidly increasing interest in modeling psychological systems using differential equations. A few of the many examples in recent literature are developmental change (Gerstorf, Ram, Röcke, Lindenberger, & Smith, 2008), fluctuations and self-regulation of mood (Russell et al., 2011), resilience in the face of adversity (Bisconti, Bergeman, & Boker, 2004; Montpetit, Bergeman, Deboeck, Tiberio, & Boker, 2010; Ong, Bergeman, Bisconti, & Wallace, 2006), day to day changes in hunger and appetite (Klump, Keel, Burt, et al., 2013; Racine et al., 2013), monthly change in hormones (Klump, Keel, Kashy, et al., 2013), age-related change in brain functioning and cognitive performance (Gerstorf et al., 2015; Raz et al., 2005; Salthouse & Davis, 2006), and dyadic-regulation of romantic couples (Boker & Laurenceau, 2007; Laurenceau et al., 1998). Fitting differential equation models to data using Structural Equation Modeling (SEM) has become increasingly popular (e.g, Benson, Ram, & Stifter, 2018; Bisconti, Bergeman, & Boker, 2006; Boker, Neale, & Klump, 2014; Driver & Voelkle, 2018; Hurley, Fink, & Janata, 2018; Montpetit et al., 2010; Morales et al., 2017; Oud & Jansen, 2000).

When one considers individual differences from the perspective of dynamical systems models, it is not the measurements themselves that comprise important differences between people. We like to think of this in the context of the questions “what”, “where”, “why”, and “how”. Traditional cross-sectional individual differences designs ask the question, what is the difference in the observed scores? Dynamical systems theory encourages us to think about the process of change within each individual. Then individual differences can be separated into three main categories (Boker, 2013; Boker, Staples, & Hu, 2016). First, we must consider individual differences in equilibria, that is to say, where is the natural homeostatic condition of each individual and how do people differ in the state to which they tend in the absence of contextual pressure (or perturbation in the language of dynamical systems)? So the first category of dynamical systems question asks, where do an individual’s observations tend to return to when perturbed away from equilibrium? Second, we must find a model for the return to equilibrium, in other words, why does a person return to equilibrium? There might be individual differences in the model by which the dynamic return to equilibrium occurs. Third, we are interested in the question, how does an individual return to equilibrium? The answer to this question is manifest in the parameters of the differential equation model. Even if the same model can account for everyone in the population, individuals may still differ in the parameters of that model.

In order to test a dynamical systems theory for regulation within individuals we need time-intensive measurements of each individual. Each person in the sample must be measured a sufficient number of times in order to have power to be able to reject a hypothesis that everyone is the same. In addition, when the models are fit to the data, if some individuals’ dynamic return to equilibrium happens relatively rapidly and for others the return is slower, we must have methods that do not induce bias towards a value that is an artifact of the design of the experiment. Unfortunately, when fitting traditional discrete time auto- and cross-regressive SEM models the measurement interval is nonlinearly confounded with the estimated parameters (e.g., Montfort, Oud, & Voelkle, 2018; Oud, 2007). Bias in parameter estimates will be a function of their values and thus individual differences in parameter estimates may either be inflated or reduced depending on the data, the measurement interval, and the model. Continuous time models (i.e., differential equations models) are used so as to avoid the trap of having the measurement interval confounded with the estimated parameter.

Differential equations express dynamics as predictive relationships between variables of continuous time change (see Hubbard & West, 1991; Kaplan & Glass, 1995, for useful introductions). Suppose that an individual has been measured on a variable, $X$, on $p$ occasions each separated by an interval of time, $\Delta t$, so that $X=\{x_0\mathrm{,}{x}_{\Delta t}\mathrm{,}{x}_{2\Delta t}\mathrm{,}{x}_{3\Delta t}\mathrm{,}\dots \mathrm{,}{x}_{(p-1)\Delta t}\}$. Now consider an individual’s score at a chosen time $t$ (denoted $x_t$) and its derivatives, e.g., how fast is the score changing at time $t$ (denoted as $dx(t)/dt$ or $\dot{x}(t)$), and how fast is this change accelerating or decelerating at the same moment of time $t$ (denoted as $d^{2}x(t)/d{t}^2$ or $\ddot{x}(t)$). If there are predictable covariance relationships between the score and its instantaneous derivatives then we can define a differential equation composed of predictive relations between the score and its derivatives and this differential equation defines the continuous time dynamic of the system.

The field of dynamical systems analysis is large and growing and it is beyond the scope of this article to provide a substantial introduction to the field. There are many sources (e.g., Montfort et al., 2018; Smith & Thelen, 1993; Thompson & Stewart, 1986) that may help the interested reader to become more familiar with the terms and uses of dynamical systems models. The current article will focus on second order linear differential equations that can provide an appealing model for self-regulating psychological processes.

Second order linear differential equations

Second order linear differential equations are those in which a target variable and its first and second derivatives are linearly related. This may be expressed by a regression equation where the second derivative is an outcome variable predicted by the variable and the first derivative such that,

$$d^{2}x(t)/d{t}^2=\eta x(t)+\zeta dx(t)/dt+e(t)$$ (1)

where $\eta$ is often termed the frequency parameter and $\zeta$ the damping parameter. The residual term, $e(t)$, can be interpreted as the degree to which the second derivative is not predicted by the displacement from equilibrium and first derivative. But this residual term may include stochastic (non-deterministic) variance as well as deterministic variance that does not conform to this linear model. We will return to this problem when we describe the constrained fourth order LDE.

A stable damped linear oscillator (DLO) occurs when $\eta <0$ and $\eta +{\zeta }^2/4<0$. In this case, oscillation will occur with a period $\lambda =\frac{2\pi }{\sqrt{-(\eta +{\zeta }^2/4)}}$. If $\zeta <0$ then if the system is perturbed away from its equilibrium, the system will over time tend towards its point equilibrium value along a negative exponential curve. If $\zeta >0$ then the system is unstable and any perturbation away from equilibrium will induce exponential increase in displacement from equilibrium. Also, if $\eta >0$ then the system is unstable and will exponentially diverge from equilibrium. Finally, if $\eta =0$ or $\eta +{\zeta }^2/4>0$ then the system behaves similarly to a first order system with exponential damping or divergence depending on the sign of $\zeta$. The simulation presented here will only consider parameter values that result in a stable DLO, the conditions most often of interest in self-regulating systems.

The flexibility in the types of curves that the second order linear differential equation can approximate is one reason that the use of this equation has become popular: Individual differences in trajectories can be modeled with a single equation even when some individuals exhibit simple damping towards equilibrium and others exhibit oscillating trajectories. Another reason is that the second order differential equation model has appealing and intuitive interpretations for its parameters. If one is interested in variability analysis, the second order differential equation model separately estimates the variance of a variable’s value (amplitude of fluctuations) and the variance of its first derivative (intuitively the “roughness” of the fluctuation trajectories) which can be independent qualities of variability in the behavior of a system (Deboeck, Monpetit, Bergeman, & Boker, 2009). If one is interested in modeling self-regulating systems, then the $\eta$ parameter can give an estimate of the average cycle time inherent in an individual’s self-regulation. The $\zeta$ parameter can give an estimate of overall return to equilibrium. The two parameters together can be interpreted as being related to resilience to adversity (Boker, Montpetit, Hunter, & Bergeman, 2010; Montpetit et al., 2010; Tarter & Vanyukov, 2002). There is increasing interest in using the second order linear (i.e., DLO) model for many types of human systems (e.g., Butler, 2017; Hurley et al., 2018; Morales et al., 2017).

Parameters of second order differential equations models can be estimated from data using two main approaches. The first approach is to analytically integrate the differential equation model and then use the resulting specific integral to predict a score at a known interval of time in the future based on scores in the immediate past using state space methods (see, e.g., Chow, Hamaker, Fujita, & Boker, 2009; Ho, Shumway, & Ombao, 2005; Petris & Petrone, 2011). The exact and approximate discrete methods (Bergstrom, 1966, 1984; Singer, 1993; Oud & Jansen, 2000; Oud, 2017) and latent Kalman filtering methods (Kalman, 1960; Chow, Ferrer, & Nesselroade, 2007; So, Ott, & Dayawansa, 1994) are examples of this approach. The second approach uses convolution filtering methods such as Latent Differential Equations (LDE, Boker, Neale, & Rausch, 2004; Boker et al., 2016; McKee, Rappaport, Boker, Moskowitz, & Neale, 2018) or Generalized Local Linear Approximation (GLLA Boker, Deboeck, Edler, & Keel, 2010) to estimate the parameters of the differential equation without integrating the differential equation. The current article will focus on an improvement to the LDE method of estimation of parameters for the second order linear differential equation.

LDE Advantages and Disadvantages

The LDE method of fitting a differential equation model to data is a variant of a class of operations known as convolution filtering (Levinson, 1946, 1947; Wiener, 1949), a technique that multiplies a filter with windowed samples of a time series in order to obtain a result with desired characteristics. This technique has been widely used in image processing (e.g., Gonzalez & Wintz, 1977; Ozaktas, Barshan, Mendlovic, & Onural, 1994), audio processing (e.g., Reilly & McGrath, 1995), and physics (e.g., Byerly, 1965). In particular, this technique has been applied to time series data to estimate derivatives (Savitzky & Golay, 1964). Time delay embedding is a closely associated technique that was developed for nonlinear dynamical systems attractor reconstruction (Ruelle & Takens, 1971; Sauer, Yorke, & Casdagli, 1991) and was proven to be able to capture the dynamical properties of a time series (Takens, 1985) even when that series was nonlinear and had no analytic solution. Efficient calculation of a convolution filter with a time series can be produced by a matrix multiplication of the filter with a time delay embedding of a time series as is used in the LDE method (see Boker et al., 2016, for an extended discussion).

There are advantages and disadvantages to the use of the LDE method for specifying and fitting differential equations to time series data. The first advantage is that the LDE method uses time delay embedding as its data representation, which in the context of second order linear differential equations has been shown to increase precision of parameter estimates (Oertzen & Boker, 2010) and has surprising robustness to violations of time interval homogeneity (Boker, Tiberio, & Moulder, 2018).

The second advantage of using an LDE model is that the model is easily specified in commonly understood latent variable terms. This means that it becomes easier to consider many alternative models for the same system, specify parameter constraints, create coupled systems, and even specify nonlinear systems that can lead to bifurcation. The second order linear system of differential equations is flexible and useful, but one cannot reasonably assert that no better model exists for human systems. The use of an easily specified and applied modeling method helps encourage exploration of model alternatives and model comparisons.

A third advantage to the convolution filtering in LDE becomes apparent when nonlinear systems of more than one variable are considered. In many physical systems there exist dynamics that both diverge and converge in a balanced way. When a system is converging to a known equilibrium, errors in measurement of the true signal at time $t$ imply smaller prediction errors at time $t+\Delta t$, giving an advantage to integration forward predictive methods. But when a system is diverging, errors in measurement of the true signal at time $t$ imply larger prediction errors at time $t+\Delta t$. If a system is only diverging and one has the full time series in hand it becomes easy to circumvent this problem by running time backwards, i.e., predicting the value at time $t$ from the value at time $t+\Delta t$. So, in this case, a forward prediction method can be shown to not be at a disadvantage to a filtering method. But what happens when a system with more than one variable is both diverging and converging, i.e, when one Lyapunov exponent is positive and another is negative? The time reversal trick no longer works for prediction, and one variable is always at a built-in prediction disadvantage to another variable. Convolution filtering circumvents this problem by maintaining symmetry of the filter and thus does not advantage one variable in the system over any other variable no matter the sign of the associated Lyapunov exponents.

These advantages of the use of time delay embedding are offset by two main disadvantages of the method. First, if the time series are very short, less than about 20 observations per person, the advantages of the overlapping samples of time delay embedding disappear and one would be advised to use another method (Oud, 2017). For longer time series, time delay embedding and convolution filtering remain viable and appealing methods for estimation.

The second disadvantage is that choosing the number of columns in the time delay embedding matrix (known as the embedding dimension) has consequences for the estimation of the parameters. Parameter bias increases markedly when the embedding dimension is poorly chosen. Reduction of the dependence between the embedding dimension and parameter bias would be a significant improvement in the performance of the LDE method. The current article derives a constrained fourth order system of equations and demonstrates through simulation that implementing this system of equations as an LDE reduces parameter bias to near zero over a wide range of choices of embedding dimension.

Methods

Constrained Fourth Order LDE

A linear second order differential equation can be transformed into a system of two first order equations. Consider the equation

$$d^{2}x(t)/d{t}^2=\eta x(t)+\zeta dx(t)/dt+e(t)\mathrm{.}$$ (2)

If

$$dx(t)/dt=y(t)\mathrm{,}$$ (3)

then

$$d{x}^2(t)/dt=dy(t)/dt\mathrm{,}$$ (4)

and now we can substitute Equations 3 and 4 into Equation 2 so that the following system of linear first order equations

$$dx(t)/dt=y(t)$$ (5)

$$dy(t)/dt=\eta x(t)+\zeta y(t)+e(t)$$

becomes equivalent to the second order linear differential equation in Equation 2. By this well-known method, a higher order linear differential equation can be reduced to a system of first order differential equations.

Using the same logic, we can start this substitution process with higher order derivatives and end up with the second order linear differential equation. Consider that if

$$d^{3}x(t)/d{t}^3=\eta dx(t)/dt+\zeta d^{2}x(t)/d{t}^2+e(t)$$ (6)

and

$$dx(t)/dt=z(t)\mathrm{,}$$ (7)

then by substituting Equation 7 into Equation 6 we find that

$$d^{2}z(t)/d{t}^2=\eta z(t)+\zeta dz(t)/dt+e(t)\mathrm{,}$$ (8)

the familiar second order differential equation for the variable $z$. By the same logic, two substitutions

$$dx(t)/dt=z(t)$$ (9)

$$dz(t)/dt=w(t)$$ (10)

will transform

$$d^{4}w(t)/d{t}^4=\eta d^{2}w(t)/d{t}^2+\zeta d^{3}w(t)/d{t}^3+e(t)$$ (11)

into Equation 2.

From this logic, we propose a constrained system of equations such that the second, third, and fourth order forms hold simultaneously:

$$\begin{gathered} d^{4}x(t)/d{t}^4=\eta d^{2}x(t)/d{t}^2+\zeta d^{3}x(t)/d{t}^3+e_4(t) \\ {d}^{3}x(t)/d{t}^3=\eta dx(t)/dt+\zeta d^{2}x(t)/d{t}^2+e_3(t) \\ {d}^{2}x(t)/d{t}^2=\eta x(t)+\zeta dx(t)/dt+e_2(t)\mathrm{.} \end{gathered}$$ (12)

If these were entirely deterministic equations, in other words if the residual terms were zero, $e_4(t)=e_3(t)=e_2(t)=0$, then if any one of the equations held, then all of the equations would hold. In this case, there would be nothing to be gained by estimating these equations simultaneously. However, if $e_4(t)\ne e_3(t)\ne e_2(t)\ne 0$, then each regression estimation has the opportunity to improve on the others. These two extra degrees of freedom (three residual terms versus one residual term) are, as presented in Equation 13, still mixtures of deterministic and stochastic residuals. By specifying the derivatives as latent variables, the deterministic and stochastic (non-deterministic) residual variances can be separately estimated. When a traditional second order LDE is estimated, parameter bias can be introduced either when stochastic variance is mistaken for deterministic variance or vice versa. By allowing higher order latent derivatives, the separation between stochastic and deterministic variance is improved and thus parameter bias is reduced.

A path diagram of an LDE model of the second order linear differential equation in Equation 2 applied to a time delay embedding of dimension 5 is shown in Figure 1–a. This LDE model is a simplified form that does not estimate the equilibrium or have any individual differences in parameters. Alternative forms of this second order model that estimate DLO when there are individuals differences in parameters, individual differences in equilibrium value, and within person change in equilibrium are available elsewhere as downloadable scripts (Boker et al., 2016). Figure 1–b presents the fourth order constrained approximation to the second order model that is simulated and tested in the current article against the model shown in Figure 1–a.

Figure 1.

Second and fourth order LDE models to fit a second order linear differential equation.

Model A, as shown in Figure 1–a was specified and fit in OpenMx (Neale et al., 2016) using the script in Appendix A. In the simulation condition where the data are time delay embedded with an embedding dimension of 5, a matrix ${\mathbf{X}}^{(5)}$ is the data to which the model is fit (see, e.g., Boker et al., 2016, for an explanation of this matrix). Then the LDE model in Figure 1–a would be specified as

$${\mathbf{X}}^{(5)}=\hspace{0.17em}GL\text{'}+\hspace{0.17em}U\mathrm{,}$$ (13)

where $\hspace{0.17em}G$ is a matrix of unobserved latent derivative scores, $\hspace{0.17em}U$ is a matrix of unobserved unique scores and $\hspace{0.17em}L$ is a fixed matrix that specifies the convolution filter kernel and is in this case defined as

$$\hspace{0.17em}L=\left[\begin{array}{rrr}\hfill 1& \hfill -2\Delta t& \hfill {(-2\Delta t)}^2/2\\ \hfill 1& \hfill -1\Delta t& \hfill {(-1\Delta t)}^2/2\\ \hfill 1& \hfill 0& \hfill 0\\ \hfill 1& \hfill 1\Delta t& \hfill {(1\Delta t)}^2/2\\ \hfill 1& \hfill 2\Delta t& \hfill {(2\Delta t)}^2/2\\ \hfill & \hfill & \hfill \end{array}\right]\mathrm{,}$$ (14)

where $\Delta t$ is the elapsed time between adjacent lagged columns in the time–delay embedded matrix ${\mathbf{X}}^{(5)}$.

Using the Reticular Action Model (RAM) algebra (McArdle & McDonald, 1984) and path diagram conventions (McArdle & Boker, 1990), the expected covariance of the time delay embedded matrix, $\epsilon ( \text{Cov} ( {\mathbf{X}}^{(5)}))$ is

$$\epsilon \left(\text{Cov} ({\mathbf{X}}^{(5)})\right)= \mathbf{L}\text{Cov}(G)\mathbf{L}\text{'}+\text{Cov}(U)\mathrm{,}$$ (15)

where $\text{Cov} (U )$ is a diagonal matrix in which all elements on the diagonal are constrained to be equal to one another. The covariance of the latent variables, Cov(G), can now be specified as

$$\text{Cov}(G)={\left(I -A\right)}^{-1}S{\left(I-A\right)}^{-1^{\prime }}\mathrm{.}$$ (16)

where $I$ is the identity matrix,

$$A =\left[\begin{array}{rrr}\hfill 0& \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill 0\\ \hfill \eta & \hfill \zeta & \hfill 0\end{array}\right] \mathrm{,}\hspace{0.17em}\text{and}$$ (17)

$$\hspace{0.17em}S =\left[\begin{array}{rrr}\hfill V_g& \hfill C_{g\mathrm{,}dg/dt}& \hfill 0\\ \hfill C_{g\mathrm{,}dg/dt}& \hfill V_{dg/dt}& \hfill 0\\ \hfill 0& \hfill 0& \hfill V_{d^{2}g/d{t}^2}\end{array}\right]\mathrm{.}$$ (18)

The parameters $\eta$ and $\zeta$ are the frequency and damping parameters, $V_g$ and $V_{dg/dt}$ are the variances of the latent variables $g$ and $dg/dt$, $V_{d^{2}g/d{t}^2}$ is the residual variance for $d^{2}g/d{t}^2$, and $C_{g\mathrm{,}dg/dt}$ is the covariance between $g$ and $dg/dt$.

Model B, the constrained fourth order model in Figure 1–b is specified and fit in OpenMx as shown in the script in Appendix B. The same 5 dimensional time delay embedded matrix ${\mathbf{X}}^{(5)}$ was again predicted by Equation 13 but now there are 5 columns in the matrix of latent scores $G$. The convolution kernel matrix, $L$, for Model B is now a fourth order geometric approximation that can be defined as

$$\hspace{0.17em}L =\left[\begin{array}{rrrrr}\hfill 1& \hfill -2\Delta t& \hfill {(-2\Delta t)}^2/2& \hfill {(-2\Delta t)}^3/6& \hfill {(-2\Delta t)}^4/24\\ \hfill 1& \hfill -1\Delta t& \hfill {(-1\Delta t)}^2/2& \hfill {(-1\Delta t)}^3/6& \hfill {(-1\Delta t)}^4/24\\ \hfill 1& \hfill 0& \hfill 0& \hfill 0& \hfill 0\\ \hfill 1& \hfill 1\Delta t& \hfill {(1\Delta t)}^2/2& \hfill {(1\Delta t)}^3/6& \hfill {(1\Delta t)}^4/24\\ \hfill 1& \hfill 2\Delta t& \hfill {(2\Delta t)}^2/2& \hfill {(2\Delta t)}^3/6& \hfill {(2\Delta t)}^4/24\\ \hfill & \hfill & \hfill & \hfill & \hfill \end{array}\right]\mathrm{,}$$ (19)

where $\Delta t$ is the elapsed time between adjacent lagged columns in the time–delay embedded matrix ${\mathbf{X}}^{(5)}$. Note that the divisors of the columns of $L$ are due to the differentiation of the geometric series approximation to the derivatives used in the respective columns. If a time interval is cubed, $\Delta t^3$, then in order for it to be an appropriate weight for its corresponding third derivative, $d^{3}g/dt$, one must consider that $d(\Delta t^3)/dt=3\Delta t^2$ and in the same way $d(3\Delta t^2)/dt=6\Delta t$. Thus, $\Delta t^3/6$, provides the same timescale contribution of the weights in the third column as in the other columns.

The expected covariance of ${\mathbf{X}}^{(5)}$ is again defined as in Equation 15 except that now

$$\hspace{0.17em}A =\left[\begin{array}{rrrrr}\hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill 0\\ \hfill \eta & \hfill \zeta & \hfill 0& \hfill 0& \hfill 0\\ \hfill 0& \hfill \eta & \hfill \zeta & \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill \eta & \hfill \zeta & \hfill 0\end{array}\right] \mathrm{,}\hspace{0.17em}\text{and}$$ (20)

$$\hspace{0.17em}S =\left[\begin{array}{rrrrr}\hfill V_g& \hfill C_{g\mathrm{,}dg/dt}& \hfill 0& \hfill 0& \hfill 0\\ \hfill C_{g\mathrm{,}dg/dt}& \hfill V_{dg/dt}& \hfill 0& \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill V_{d^{2}g/d{t}^2}& \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill 0& \hfill V_{d^{3}g/d{t}^3}& \hfill 0\\ \hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill V_{d^{4}g/d{t}^4}\end{array}\right]\mathrm{.}$$ (21)

Note that since $L$ has no free parameters and every time $\eta$ (or $\zeta$) appears in $A$ it is constrained to be equal to the other $\eta$s (or $\zeta$s), there are only 2 more degrees of freedom in Model B than in Model A. Also note that if we were to start with Model B and constrain to zero the variance terms for the third and fourth derivatives as well as their corresponding $\eta$ and $\zeta$ regression predictors, we would have constructed a model identical to Model A and nested within Model B. Thus the difference between the fit of Models A and B is chi square distributed with 2 degrees of freedom and the maximum likelihood ratio test can be applied to test for the difference in fit between Models A and B.

Simulation Conditions

A simulation was run in which the traditional LDE model (Model A from Figure 1–a) and the fourth order constrained approximation LDE model (Model B from Figure 1–b) were each run on the same simulated data and the results compared. The simulation was designed to test the boundaries of the applicability of each model and so we used a univariate version of the LDE, which has been shown to perform substantially worse than the multivariate version (Boker et al., 2004). Our results thus show both Model A and Model B in a particularly poor light and highlight the differences between the models and their points of failure.

Since bias in the LDE model of the DLO has been shown to be a function of embedding dimension (Hu, Boker, Neale, & Klump, 2014), the simulation has the greatest number of conditions applied to embedding dimension. In addition, LDE parameter bias in the DLO has been shown to be a function of noise in the data and so we also varied the signal to noise ratio (SNR) in the simulated data and the decay time of the DLO so that, in some conditions, the signal is only present in the part of the target data prior to when the DLO has returned to close to its equilibrium. When the true simulated signal is close to its equilibrium, it is swamped with noise. In the case of a quickly damped system, the SNR is varying markedly over the course of the time series.

In the last condition, we varied the number of observations in the time series. Since often a researcher will have some prior hypothesis about the cycle time of the system when an experimental design is constructed, we kept the total elapsed time and frequency of the simulated data constant and varied the number of observations within that total elapsed time. Note that frequency is defined as cycles per interval of time. Thus this condition varied the number of samples per cycle. A critical feature of continuous time estimation is that parameter values should not change based on the interval of time between samples. Parameters of the second order differential equation include an estimate of the frequency of oscillations. Therefore, parameters from continuous time estimation cannot depend on how many samples are collected within one cycle. The simulation conditions were constructed so that two critical conditions could be independently varied: the number of samples per cycle and the embedding dimension used to construct the time delay embedding matrix.

The simulation conditions were chosen to be representative of the range of parameters reported in articles presented in the Introduction. Reported frequency and damping parameters were converted into a common metric and number of observations were determined so that commonly reported frequencies would be contained in the conditions. In all cases, the number of observations between columns in the time delay embedding matrix (commonly referred to as the $\tau$ parameter) were set to be equal to $\tau =1$. In practice, we have found that setting $tau=1$ and only adjusting the embedding dimension produces more precise estimates than setting $tau$ to any other value.

With the previous paragraphs in mind, the frequency parameter was fixed, $\eta =-0.50$, for all cells of the simulation. There were two conditions for the damping parameter, $\zeta =\left\{-\mathrm{0.10,}-0.30\right\}$ corresponding to long and short decay times. The signal to noise ratio had three conditions SNR = {1, 2, 4} corresponding to poor, medium, and good SNRs. While these do not have exact correspondence to reliability, these SNR conditions are roughly equivalent to reliabilities of .5, .65, and .8 when defined by true score over total score. There were three conditions in the length of the series, $\text{N}=\left\{\mathrm{50,100,300}\right\}$ corresponding to a short, medium, and long time series for a single individual. Different conditions for time intervals between observations were used in order to spread the total time interval variation across one full cycle of the simulated DLO. Convolution kernels spanning a total time interval greater than the cycle length were not considered since they violate the sampling theorem (the so-called Nyquist Limit) and thus produce invalid estimates a priori. In the $N=50$ condition, the embedding dimensions were set to $D=\left\{\mathrm{4,5,6,7,8,9,10,11,12,13,14,15}\right\}$, for a total of 12 conditions when the length of the simulation was short. In the $N=100$ condition, the embedding dimensions were set to $D=\left\{\mathrm{5,7,9,11,13,15,17,19,21,23,25,27,29}\right\}$, for a total of 13 conditions. In the condition when N = 300, the embedding dimension conditions were $D=\left\{\mathrm{5,9,13,17,21,25,29,33,37,41,45,49,53,57,61}\right\}$, for a total of 15 conditions. In this way, each condition of $N$ had similar spread of embedding dimension conditions across a full simulated cycle interval. The full simulated data design included $2\times 3\times 3\times (12+13+15)=360$ conditions in all.

In each of the 360 condition cells 1,000 time series were simulated conforming to the parameters of the cell while for each time series randomizing the initial conditions of the DLO and the normally distributed measurement error conforming to the SNR condition. Both Model A and Model B were fit to each data set and the parameters were saved. Thus overall, the simulation ran 720,000 LDE models. Total wall-clock time for the simulation was less than 16 hours when run on a recently manufactured laptop. When strict model convergence criteria were not obtained, the reasons for non-convergence were saved, but the parameters of the non-converging models did not contribute to the analysis of the results. No attempt was made to restart non-converging models and so the non-convergence statistics can be considered to be worst-case figures: a lower bound on convergence percentages.

Results

The mean and standard deviation of each parameter was calculated for the 1000 replications in each cell. This provides the mean point estimate of each parameter and the variability of the parameter over the replications. The mean squared error (MSE) was calculated as the mean of the squared difference between each parameter point estimate and its associated simulated value. The percent bias was calculated as the mean of the difference between each point estimate and its simulated value divided by the simulated value and multiplied by 100. This measure of bias allows comparisons in bias performance independent of the chosen simulated parameter value. The percent of convergence was calculated as the percent of 1000 replications that had no convergence errors. Overall, Model A converged 79.1% and Model B converged 89.5% over all the cells in the simulation. Convergence errors that occurred were either the failure of a boundary walk by the optimizer (the SLSQP option in OpenMx) or that the optimizer exceeded the maximum allotted iterations (the default “Auto” option for SLSQP). The above calculated values are shown in Table 1 for all simulated conditions when the width of the convolution kernel covered half of one cycle of the simulated DLO (i.e., the embedding dimension was one half the number of samples in one cycle of the simulated DLO).

Table 1.

Mean ($\mu$) and standard deviation ($\sigma$) of point estimates, mean squared error (MSE), the percent bias (Pct Bias) and the percent convergence (Pct Cvrg) for $\eta$ and $\zeta$ for both 2nd and 4th order LDEs over all simulated conditions where the convolution kernel covered 1/2 a cycle of the simulated DLO (i.e., the embedding dimension was 1/2 the number of samples per cycle).

Simulation				MODEL A				MODEL B
				Est		Pct	Pct	Est		Pct	Pct
	N	SNR	Sim	$\mu$($\sigma$)	MSE	Bias	Cvrg	$\mu$($\sigma$)	MSE	Bias	Cvrg
$\eta$	50	1	−0.5	−0.425(0.095)	0.015	−14.9	96.6	−0.495(0.107)	0.012	−1.0	95.2
$\zeta$			−0.1	−0.088(0.070)	0.005	−12.3		−0.089(0.080)	0.007	−11.0
$\eta$	100	1	−0.5	−0.428(0.015)	0.005	−14.3	99.6	−0.507(0.023)	0.001	1.4	98.0
$\zeta$			−0.1	−0.093(0.031)	0.001	−7.1		−0.102(0.035)	0.001	2.2
$\eta$	300	1	−0.5	−0.427(0.009)	0.005	−14.7	99.1	−0.504(0.014)	0.000	0.9	96.9
$\zeta$			−0.1	−0.093(0.016)	0.000	−7.2		−0.102(0.018)	0.000	2.4
$\eta$	50	2	−0.5	−0.421(0.042)	0.008	−15.7	95.5	−0.484(0.052)	0.003	−3.1	96.0
$\zeta$			−0.1	−0.092(0.033)	0.001	−8.0		−0.092(0.040)	0.002	−7.8
$\eta$	100	2	−0.5	−0.427(0.008)	0.005	−14.6	95.6	−0.503(0.011)	0.000	0.6	92.4
$\zeta$			−0.1	−0.092(0.015)	0.000	−8.0		−0.101(0.017)	0.000	1.5
$\eta$	300	2	−0.5	−0.425(0.004)	0.006	−14.9	75.8	−0.502(0.006)	0.000	0.4	93.3
$\zeta$			−0.1	−0.091(0.008)	0.000	−8.6		−0.101(0.009)	0.000	1.4
$\eta$	50	4	−0.5	−0.416(0.021)	0.008	−16.9	42.2	−0.480(0.027)	0.001	−4.1	94.6
$\zeta$			−0.1	−0.089(0.017)	0.000	−10.6		−0.092(0.020)	0.000	−7.7
$\eta$	100	4	−0.5	−0.426(0.004)	0.005	−14.8	51.0	−0.502(0.006)	0.000	0.5	85.3
$\zeta$			−0.1	−0.092(0.008)	0.000	−7.9		−0.102(0.008)	0.000	2.0
$\eta$	300	4	−0.5	−0.427(0.009)	0.005	−14.5	98.8	−0.505(0.014)	0.000	1.1	96.4
$\zeta$			−0.1	−0.093(0.016)	0.000	−7.0		−0.103(0.018)	0.000	2.8
$\eta$	50	1	−0.5	−0.418(0.085)	0.014	−16.5	98.0	−0.493(0.105)	0.011	−1.3	95.6
$\zeta$			−0.3	−0.276(0.082)	0.007	−8.1		−0.297(0.104)	0.011	−1.0
$\eta$	100	1	−0.5	−0.424(0.021)	0.006	−15.2	99.5	−0.508(0.030)	0.001	1.5	97.8
$\zeta$			−0.3	−0.273(0.031)	0.002	−8.9		−0.303(0.036)	0.001	0.9
$\eta$	300	1	−0.5	−0.424(0.020)	0.006	−15.1	99.7	−0.508(0.029)	0.001	1.6	97.6
$\zeta$			−0.3	−0.273(0.030)	0.002	−9.0		−0.303(0.034)	0.001	1.0
$\eta$	50	2	−0.5	−0.414(0.040)	0.009	−17.1	94.1	−0.485(0.051)	0.003	−2.9	96.5
$\zeta$			−0.3	−0.273(0.039)	0.002	−9.0		−0.294(0.050)	0.003	−2.1
$\eta$	100	2	−0.5	−0.422(0.011)	0.006	−15.5	96.0	−0.503(0.015)	0.000	0.6	94.1
$\zeta$			−0.3	−0.273(0.016)	0.001	−8.9		−0.304(0.019)	0.000	1.3
$\eta$	300	2	−0.5	−0.423(0.014)	0.006	−15.3	66.9	−0.506(0.020)	0.000	1.3	72.1
$\zeta$			−0.3	−0.274(0.021)	0.001	−8.7		−0.304(0.024)	0.001	1.5
$\eta$	50	4	−0.5	−0.410(0.019)	0.008	−18.0	24.5	−0.484(0.025)	0.001	−3.2	92.5
$\zeta$			−0.3	−0.271(0.019)	0.001	−9.6		−0.294(0.025)	0.001	−1.9
$\eta$	100	4	−0.5	−0.421(0.005)	0.006	−15.8	63.4	−0.501(0.008)	0.000	0.1	86.3
$\zeta$			−0.3	−0.274(0.008)	0.001	−8.6		−0.305(0.009)	0.000	1.7
$\eta$	300	4	−0.5	−0.421(0.005)	0.006	−15.9	27.3	−0.502(0.007)	0.000	0.4	31.1
$\zeta$			−0.3	−0.274(0.007)	0.001	−8.8		−0.305(0.009)	0.000	1.6

It is evident from Table 1 that in every condition and for both estimated parameters, the bias as a percent of the parameter value was lower for Model B than it was for Model A. Model B’s bias was in all but three conditions between −5% and +5% and the mean across all conditions was −0.5%, leading to the conclusion that, overall, Model B was performing as an unbiased estimator. On the other hand, Model A’s bias ranged from −7% to −18% and the mean bias was −12.1%. Overall, Model A always estimated oscillator parameters that indicated slower cycle times and slower decay than was simulated. Thus, if percentage bias is the sole consideration, Model B is always preferred, and to a large degree.

However, in every condition the standard deviations of Model B parameter estimates were larger than those of Model A estimates and so if bias is not taken into account, Model A would be preferred. This would be a poor criterion in practice, since Model A provides estimates that are more precisely inaccurate. One way to balance precision and bias is to use a criterion such as MSE, which is a combined measure of bias and estimation variability. Here we see that in 27 out of 36 cases the MSEs of Model B are smaller than those of Model A, indicating that Model B is to be preferred 75% of the time by the criterion of MSE. The results for percentage convergence do not prefer one model over the other: Out of 18 model conditions, Model A converged more often in 9 conditions and Model B converged more often in 9 conditions.

While the results presented in Table 1 indicate that Model B gives an unbiased estimate of both the $\eta$ and $\zeta$ parameters, this is under the condition that the convolution kernel is approximately one half of the period of the DLO. Since previous work has shown that bias in $\eta$ is sensitive to the proportion of the period of the DLO covered by the convolution kernel, Figure 2 plots the results of estimating $\eta$ and $\zeta$ with Models A and B for 12 different embedding dimensions under the simulation condition of the medium length time series ($N=100$), medium signal to noise ratio (SNR=2:1), and longer decay ($\zeta =-0.1$).

Figure 2.

Results of simulation of second order differential equations with signal to noise ratio of 2:1, $N=100$, $\eta =-.5$, and $\zeta =-.1$ resulting in 30 observations per cycle as fit with second and fourth order models for 12 different embedding dimensions. (a) Second order approximation model estimated $\eta$ parameter for each selected embedding dimension. Simulated $\eta =-.5$ is shown as horizontal line. Mean and standard deviation of the estimated parameters are plotted for all models that converged. (c) Second order approximation results for $\zeta =-.1$. (e) Percent of models that converged (green), did not converge (red), and stopped due to exceeding the maximum iterations. (b,d,f) The $\eta$, $\zeta$, and convergence percentages for the fourth order model.

The relationship between the bias in estimation of $\eta$ and the choice of embedding dimension for Model A can be clearly seen in Figure 2–a and its suppression by Model B can be seen in Figure 2–b. Model B is unbiased for all embedding dimensions up to 22: in other words, when the convolution kernel covers less than about 2/3 of a single 30 observation cycle of the DLO. For embedding dimensions greater than 20, the bias for Model B increases rapidly until at 30 observations the Nyquist limit is reached. However, for all embedding dimensions, the bias for $\eta$ in Model B is smaller than the bias in Model A.

Model B has similar relationships between the bias in $\eta$ or $\zeta$ and the associated embedding dimension. That is to say, up to embedding dimension 22, Model B provides unbiased estimates of $\zeta$. However, for embedding dimensions between 24 and 30, the bias increases markedly. Model A performs better than Model B in this regard, since Model A’s bias in $\zeta$ is small over the whole range of embedding dimensions between 5 and 30.

With respect to convergence, it can be seen that both Model A and Model B have much more trouble converging when the convolution kernel covers more than 2/3 of the period of the DLO. It should be noted that this means that the parameter estimate results plotted in Figures 2–a, b, c, and d are based on substantially reduced sample cell sizes when the embedding dimension is greater than 22.

In order to understand the effects of length of the time series, two figures were created where, as in Figure 2, the SNR was set to its medium value (SNR=2:1) and simulated $\zeta =-0.1$ but the length of the time series differed. Figures 3 and 4 plot the results for the shorter ($N=50$) and longer ($N=300$) time series length respectively. In Figure 3, it is unsurprising that fewer observations in the time series lead to more variable estimates. The Model A results plotted in Figures 3–a and c show the previous pattern of results, but with larger standard deviations. However, Figures 3–b and d are somewhat surprising since parameter bias begins to be different from zero as the coverage of the convolution kernel is increased to be greater than 8, which is only 1/2 the period of the DLO in this simulation condition.

Figure 3.

Results of simulation of second order differential equations with signal to noise ratio of 2:1, $N=50$, $\eta =-.5$, and $\zeta =-.1$ resulting in 30 observations per cycle as fit with second and constrained fourth order LDEs for 12 different embedding dimensions. (a) Second order approximation model estimated $\eta$ parameter for each selected embedding dimension. Simulated $\eta =-.5$ is shown as horizontal line. Mean and standard deviation of the estimated parameters are plotted for all models that converged. (c) Second order approximation results for $\zeta =-.1$. (e) Percent of models that converged (green), did not converge (red), and stopped due to exceeding the maximum iterations. (b,d,f) The $\eta$, $\zeta$, and convergence percentages for the constrained fourth order model.

Figure 4.

Results of simulation of second order differential equations with signal to noise ratio of 2:1, $N=300$, $\eta =-.5$, and $\zeta =-.1$ resulting in 30 observations per cycle as fit with second order and constrained fourth order models for 12 different embedding dimensions. (a) Second order model estimated $\eta$ parameter for each selected embedding dimension. Simulated $\eta =-.5$ is shown as horizontal line. Mean and standard deviation of the estimated parameters are plotted for all models that converged. (c) Second order approximation results for $\zeta =-.1$. (e) Percent of models that converged (green), did not converge (red), and stopped due to exceeding the maximum iterations. (b,d,f) The $\eta$, $\zeta$, and convergence percentages for the constrained fourth order model.

If we now look at the results of estimating the parameters of the longer simulated time series ($N=300$) plotted in Figure 4, it is again unsurprising that all parameter estimates have smaller standard deviations at all embedding dimensions. Model B now performs well over all but the smallest choice of embedding dimensions whereas the bias in $\eta$ for Model A is improved, but does not go away. For the longer time series, Model B is preferred in every way, giving unbiased performance with good efficiency and good convergence rates.

Since Model A and B are nested models, the difference between their respective optimized minus two log likelihood function values is distributed as chi-square with two degrees of freedom—a likelihood ratio difference test. Table 2 displays the results of calculating the difference between Model A and Model B function value for each replication in which both models successfully converged when fit using the same embedding dimension as used for Table 1. The mean, $\mu$, and standard deviation, $\sigma$, of this difference are displayed. Since Model B has two more degrees of freedom than Model A, Model B is guaranteed to always fit at least as well as Model A. However, as can be seen from the column of means of the difference in fit values, the fit for Model B is almost always much better than would be expected by chance for two degrees of freedom. The column labeled $p<0.05$ displays the percentage of replications for which this difference is larger than the chi-square two degree of freedom critical value of 5.991. Only when the time series are short and noisy is this percentage not greater than 95%.

Table 2.

Mean ($\mu$) and standard deviation ($\sigma$) of the difference between Model A and Model B minus two log likelihood function values over all simulated conditions where the convolution kernel covered 1/2 a cycle of the simulated DLO (i.e., the embedding dimension was 1/2 the number of samples per cycle). ($p<0.05$ indicates the percent of comparisons where this difference exceeded a two degree of freedom chi-square critical value of 5.991 and Cnvrg indicates the number of times both models converged out of 1,000 replications so that the likelihood difference could be calculated)

Simulation					(−2LL MODEL A) – (−2LL MODEL B)
N	SNR	$\eta$	$\zeta$	$\mu$	$\sigma$	$p<0.05$	Cnvrg
50	1	−0.5	−0.1	12.53	6.97	83.9%	937
100	1	−0.5	−0.1	30.85	12.60	99.4%	980
300	1	−0.5	−0.1	154.51	35.75	100.0%	967
50	2	−0.5	−0.1	44.04	14.48	100.0%	937
100	2	−0.5	−0.1	118.31	27.13	100.0%	902
300	2	−0.5	−0.1	599.27	83.53	100.0%	724
50	4	−0.5	−0.1	135.48	25.93	100.0%	418
100	4	−0.5	−0.1	408.41	60.36	100.0%	492
300	4	−0.5	−0.1	155.25	36.48	100.0%	963
50	1	−0.5	−0.3	15.07	7.91	91.6%	953
100	1	−0.5	−0.3	29.77	12.33	98.5%	978
300	1	−0.5	−0.3	95.54	34.85	100.0%	976
50	2	−0.5	−0.3	52.92	15.76	100.0%	929
100	2	−0.5	−0.3	111.76	25.83	100.0%	919
300	2	−0.5	−0.3	195.25	53.13	100.0%	607
50	4	−0.5	−0.3	155.30	26.31	100.0%	245
100	4	−0.5	−0.3	377.83	56.21	100.0%	598
300	4	−0.5	−0.3	1420.39	181.39	100.0%	141

Discussion

The current article presents the results of simulated second order linear differential equations that have parameters which result in damped linear oscillators, DLOs. The results of the simulation are quite clear with respect to bias. When a time delay embedding is chosen such that the elapsed time between the first and last column of the time delay embedded matrix was less than 1/2 of a cycle, a constrained fourth order LDE (Model B) had less bias in estimating the frequency parameter, $\eta$, in every tested condition than did the traditional second order latent differential equation (Model A). In estimating $\zeta$, the damping parameter, Model B also exhibited low bias similar to that of Model A in every condition except when the time series were short and noisy. In that case, Model A performed better than Model B in estimating $\zeta$, although worse than Model B in estimating $\eta$. When time series included 100 or more observations per person, Model B was unbiased when the elapsed time between the first and last column of the time delay embedded matrix was less than 2/3 of a cycle.

The improvement in estimation can also be observed when comparing Model A and Model B using a likelihood ratio difference test. In all but two conditions, Model B was significantly better fitting than chance at an alpha level of $p=0.05$ more than 95% of the time. In all but four conditions, Model B was significantly better fitting than Model A in 100% of the replications where both Model A and Model B converged.

The choice of embedding dimension has been of concern to many researchers interested in using the LDE method to estimate parameters of models for fluctuating data. The current results suggest that when there are at least 8 observations per cycle and at least 100 total observations per person, a constrained fourth order LDE can perform as an unbiased estimator of both the frequency and damping parameter of a second order linear differential equation model. Fewer observations per person will require a experimental or observational design in which there are more observations per cycle in order for the method to perform optimally.

Limitations

While the constrained fourth order LDE substantially reduces bias relative to a second order LDE in most of the simulated conditions, it did not do so in all conditions. In particular, when the embedding dimension created a convolution kernel of width more than 2/3 the cycle interval, bias was no longer attenuated. We believe that this bias may be able to be corrected up to a full cycle interval, but at present do not have a method to do so. This is a potential area of further exploration.

The simulation conditions were extensive (720 total conditions), but certainly were not comprehensive. The constrained fourth order LDE needs to be tested in a variety of other commonly used conditions. The current article only simulated stable DLOs—unstable systems and non-oscillating systems need to be tested. The current article only explored univariate LDE, but could be easily extended to multivariate LDE. We chose to explore the univariate case because univariate LDE is more prone to bias than multivariate LDE and so the simulation provided a worst-case scenario for the constrained fourth order LDE test. Future work needs to test whether multivariate LDE can be improved in the same manner.

The latent differential equation that was tested was a second order linear system with one latent variable. While this is one of the common use cases for the LDE estimation method, there are many possible differential equation models that are of use to the psychological, sociological, educational, and life sciences. One common instance is the case of coupled damped linear oscillators. In this case, the dynamics of two latent variables are linearly coupled. Estimation bias for this type of model seems to be likely to be reduced by constrained fourth order LDE, simulation work needs to be done to test this hypothesis. Reduction of bias in the coupled oscillator model is particularly important when the two latent variables are not oscillating at the same rate. In this case, the choice of embedding dimension in second order LDE becomes difficult, and the two latent variables end up being biased to appear to be more similar than they actually are. If FOLDE exhibits similar bias suppression in coupled oscillator models, it will be particularly helpful for modeling these types of systems.

We did not test the case of estimation of time varying equilibria (e.g., Boker et al., 2016), but our intuition is that this will prove to not affect the results presented here. We also did not test the case of individual differences in parameters or equilibria, but again, the results of the current simulation are expected to apply just as well to these two cases. As for the coupled oscillator case, when there are individual differences in parameters of a DLO, the choice of embedding dimension is difficult and every choice will tend to attenuate individual differences in parameters when traditional LDE is used. Future work on these two cases would be especially useful to those interested in predicting individual differences in dynamics.

We did not test the case of time series phase resets. This is when a sudden jump occurs in a time series and then the dynamic continues from the new initial condition. This case produces bias particularly in the estimation of the damping parameter in LDEs (McKee et al., 2018). In the use of interpolation splines, fourth order splines reduce what is called “overshoot” by the spline (e.g., El Tarazi & Sallam, 1987). It seems reasonable that the fourth order LDE would similarly reduce bias in the damping parameter when the time series is interrupted by phase resets. Future simulation work is needed to explore this possibility.

We did not test the case of missing data and/or time interval misspecification. While these are important problems to consider, time delay embedding has proved to be surprisingly robust to this type of misspecification. (Boker et al., 2018) report four methods for accounting for time interval misspecification when using time delay embedding and find that imputation methods are outperformed by accounting for the time misspecification in each row using OpenMx’s definition variable feature. Simply ignoring the time misspecification produced estimates that were correct for the mean time interval and outperformed the two imputation methods. Future work needs to be done to test whether time interval misspecification and/or missing data interfere with the positive attributes of FOLDE.

Finally, the simulation was limited in that it did not include any nonlinearity or nonstationarity in the differential equation. The recently proposed Adaptive Equilibrium Regulation (AER) models for conditioning and withdrawl observed in addictive substance use (Boker, 2015; McKee et al., 2018) address nonstationarity as a dynamic in and of itself. Future research is needed to understand how the FOLDE technique might be adapted to AER models.

Given so many recommendations for future work appear here and in the interest of reproducibility, the authors include as supplemental materials the full set of simulation scripts used in the current article in hopes of encouraging others in making rapid progress in testing additional hypotheses.

Conclusions

The use of constrained fourth order LDE reduces parameter estimation bias relative to second order LDE in almost all of the conditions simulated here. When a second order model is fit to data with at least 100 observations per person and at least 8 observations per cycle, FOLDE reduces bias and significantly improves model fit with no apparent downsides, and thus is recommended as an improvement over the use of a standard LDE on these data. When time series are short and noisy, i.e., 50 observations per person and a 1:1 signal to noise ratio, there are tradeoffs to the use of FOLDE such that its recommendation is not so clear. Given short and noisy time series, if the estimation of the damping parameter, $\zeta$, is most critical, we recommend LDE over FOLDE. However, if the frequency parameter, $\eta$, is most critical, FOLDE is recommended even for short, noisy time series. For panel data, i.e., very short time series with 4 to 20 observations per person, the advantages of time delay embedding are much reduced and other continuous time methods such as the exact discrete (Singer, 1993), approximate discrete (Oud & Jansen, 2000) or unscented Kalman filter (Chow et al., 2007) methods are may produce better parameter estimates than convolution filtering methods such as LDE and FOLDE (Oud, 2017).

One of the surprising findings of the simulation is how quickly the large (720,000 model fits) simulation ran. The average time per model fit was less than 12 seconds over all the conditions and replications when run with OpenMx on a modern laptop. Thus there does not appear to be a computational complexity downside to FOLDE. Given that, a safe recommendation is to fit both LDE and FOLDE to ones data. In all simulated conditions when FOLDE resulted in a large improvement in fit over LDE, the parameter estimates of FOLDE were substantially less biased than LDE.

Appendix A – R script for MODEL A specification

# ———————————————————————————- 
# Previously created the simulation data and time delay embedded it 
# to create the data “tEmbedded” that are fit by this model. 
# Please see the full scripts in the supplemental material. 

embedD <- 8 
theTau <- 1 
deltaT <- .1 

# —————————————————————————- 
# Create the fixed 2nd order LDE loading matrix. 
L1 <- rep(1,embedD) 
L2 <- c(1:embedD)*theTau*deltaT-mean(c(1:embedD)*theTau*deltaT) 
L3 <- (L2^2)/2 
LMatrix <- cbind(L1,L2,L3) 

# —————————————————————————- 
# Create 2nd order LDE model. 

manifestVars <- dimnames(tEmbedded)[[2]] 

ldeModel1 <- mxModel(“LDE_Model_1”, 
mxMatrix(“Full”, 
values=LMatrix, 
free=FALSE, 
name=“L”, 
byrow=TRUE 
),
mxMatrix(“Full”, 3, 3, 
values=c( 0, 0, 0,
0, 0, 0, 
−.2,−.2, 0), 
labels=c( NA, NA, NA, 
NA, NA, NA, 
“eta”,”zeta”, NA), 
free=c(FALSE,FALSE,FALSE, 
FALSE,FALSE,FALSE, 
TRUE, TRUE,FALSE), 
name=“A”, 
byrow=TRUE 
),
mxMatrix(“Symm”, 3, 3,
values=c( .8, 
.1, .8, 
0, 0, .8), 
free=c( TRUE, 
TRUE, TRUE, 
FALSE, FALSE, TRUE), 
labels=c(“Vx”, 
“Cxdx”, “Vdx”, 
NA, NA, “Vd2x”), 
name=“S”, 
byrow=TRUE, 
lbound=c(0.00000001, 
NA, 0.00000001, 
NA, NA, 0.00000001) 
), 
mxMatrix(“Diag”, embedD, embedD, 
values=.8, 
free=TRUE, 
labels=“u1”, 
name=“U”, 
lbound=0.000001 
),
mxMatrix(“Iden”, 3, name=“I”), 
mxAlgebra(L %*% solve(I-A) %*% S %*% t(solve(I-A)) %*% t(L) + U,
name=“R”, 
dimnames = list(manifestVars, manifestVars) 
), 
mxExpectationNormal(covariance=“R”), 
mxFitFunctionML(), 
mxData(cov(tEmbedded), 
type=“cov”, 
numObs=dim(tEmbedded)[1] 
) 
) 
# ———————————- 
# Fit the LDE model and examine the summary results. 
ldeModel1Fit <- mxRun(ldeModel1) 
summary(ldeModel1Fit) 

Appendix B – R script for MODEL B specification

# ———————————- 
# Previously created the simulation data and time delay embedded it 
# to create the data “tEmbedded” that are fit by this model. 
# Please see the full scripts in the supplemental material. 
embedD <- 8 
theTau <- 1 
deltaT <- .1
################################################### 
# Fourth Order model 
# ———————————- 
# Create the fixed 4th order LDE loading matrix. 
L1 <- rep(1,embedD) 
L2 <- c(1:embedD)*theTau*deltaT-mean(c(1:embedD)*theTau*deltaT) 
L3 <- (L2^2)/2 
L4 <- (L2^3)/6 
L5 <- (L2^4)/24 
LMatrix <- cbind(L1,L2,L3,L4,L5) 
# ———————————- 
# Create 4th order LDE model. 
manifestVars <- dimnames(tEmbedded)[[2]] 
ldeModel1 <- mxModel(“LDE_Model_1”, 
mxMatrix(“Full”, 
values=LMatrix, 
free=FALSE, 
name=“L”, 
byrow=TRUE 
), 
mxMatrix(“Full”, 5, 5,
values=c( 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 
−.2,−.2, 0, 0, 
0,0,−.2,−.2, 0, 0, 
0, 0,−.2,−.2, 0), 
labels=c( NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, 
“eta”,”zeta”, NA, NA, NA, 
NA, “eta”,”zeta”, NA, NA, 
NA, NA, “eta”,”zeta”, NA), 
free=c(FALSE,FALSE,FALSE,FALSE,FALSE, 
FALSE,FALSE,FALSE,FALSE,FALSE, 
TRUE, TRUE,FALSE,FALSE,FALSE, 
FALSE, TRUE, TRUE,FALSE,FALSE, 
FALSE,FALSE, TRUE, TRUE,FALSE), 
name=“A”, 
byrow=TRUE 
),
mxMatrix(“Symm”, 5, 5, 
values=c( .8,
.1, .8, 
0, 0, .8, 
0, 0, 0, .8, 
0, 0, 0, 0, .8), 
free=c( TRUE, 
TRUE, TRUE, 
FALSE, FALSE, TRUE, 
FALSE, FALSE, FALSE, TRUE, 
FALSE, FALSE, FALSE, FALSE, TRUE), 
labels=c(“Vx”, 
“Cxdx”, “Vdx”, 
NA, NA, “Vd2x”, 
“Cxd3x”, NA, NA, “Vd3x”, 
“Cxd4x”, “Cdxd4x”, NA, NA, “Vd4x”), 
name=“S”, 
byrow=TRUE, 
lbound=c(0.00000001, 
NA, 0.00000001, 
NA, NA, 0.00000001, 
NA, NA, NA, 0.00000001, 
NA, NA, NA, NA, 0.00000001) 
), 
mxMatrix(“Diag”, embedD, embedD, 
values=.8, 
free=TRUE, 
labels=“u1”, 
name=“U”, 
lbound=0.000001 
), 
mxMatrix(“Iden”, 5, name=“I”), 
mxAlgebra(L %*% solve(I-A) %*% S %*% t(solve(I-A)) %*% t(L) + U,
name=“R”, 
dimnames = list(manifestVars, manifestVars) 
), 
mxExpectationNormal(covariance=“R”), 
mxFitFunctionML(), 
mxData(cov(tEmbedded), 
type=“cov”, 
numObs=dim(tEmbedded)[1] 
) 
) 
# ————————————————- 
# Fit the 4th order LDE model and save the summary results. 
ldeModel1Fit <- mxRun(ldeModel1) 
summary(ldeModel1Fit)