Three Novel Methods for Determining Motor Threshold with Transcranial Magnetic Stimulation Outperform Conventional Procedures

Abstract

Objective.

Thresholding of neural responses is central to many applications of transcranial magnetic stimulation (TMS), but the stochastic aspect of neuronal activity and motor evoked potentials (MEPs) challenges thresholding methods. We analyzed existing methods for obtaining TMS motor threshold and their variations, introduced new methods from other fields, and compared their accuracy and speed.

Approach.

In addition to existing relative-frequency methods, such as the five-out-of-ten method, we examined adaptive methods based on a probabilistic motor threshold model using maximum-likelihood (ML) or maximum a-posteriori (MAP) estimation. To improve the performance of these adaptive estimation methods, we explored variations in the estimation procedure and inclusion of population-level prior information. We adapted a Bayesian estimation method which iteratively incorporated information of the TMS responses into the probability density function. A family of non-parametric stochastic root-finding methods with different convergence criteria and stepping rules were explored as well. The performance of the thresholding methods was evaluated with an independent stochastic MEP model.

Main Results.

The conventional relative-frequency methods required a large number of stimuli, were inherently biased on the population level, and had wide error distributions for individual subjects. The parametric estimation methods obtained the thresholds much faster and their accuracy depended on the estimation method, with performance significantly improved when population-level prior information was included. Stochastic root-finding methods were comparable to adaptive estimation methods but were much simpler to implement and did not rely on a potentially inaccurate underlying estimation model.

Significance.

Two-parameter MAP estimation, Bayesian estimation, and stochastic root-finding methods have better error convergence compared to conventional single-parameter ML estimation, and all these methods require significantly fewer TMS pulses for accurate estimation than conventional relative-frequency methods. Stochastic root-finding appears particularly attractive due to the low computational requirements, simplicity of the algorithmic implementation, and independence from potential model flaws in the parametric estimators.

1. Introduction

1.1. Significance and challenges of thresholding in TMS

Transcranial magnetic stimulation (TMS) is a noninvasive technique to activate neurons in the brain [1], [2]. Certain pulse rhythms can further modulate neural circuits, i.e., change how they process endogenous signals [3]. A key parameter of the stimulation pulses is their strength, which influences not only how many neurons but also which neuron types are stimulated [4]-[6]. As the TMS pulse strength affects the activation of neuron types with different functions, it also influences the efficacy of neuromodulation protocols and can even reverse the direction of the effect [7]-[11]. Finally, the stimulation strength is a key parameter not only for ensuring an effective and reproducible procedure but also for safety, and strength limitations for various modulatory protocols are recommended to avoid seizures [12]-[14].

Typically, the TMS pulse strength is individualized to a specific subject’s or patient’s anatomy and physiology based on a measured response threshold. A commonly used threshold is the motor threshold based on activation of finger muscles to stimulation of their representation in the primary motor cortex, but other types of thresholds could also be used [15]-[22]. For the motor threshold, the muscle movement following stimulation in the primary motor cortex is observed visually in some cases, typically in clinical practice, whereas in other cases, typically in research, the size of motor evoked potentials (MEPs) is measured with electromyography (EMG) at the target muscle [23]-[25]. The motor threshold is routinely also used as a reference for other brain targets and its detection is recommended before any further neuromodulatory intervention, even for recurring subjects or patients [26]-[28]. Furthermore, the threshold and its changes can serve as a quantitative metric, e.g., for the diagnosis of lesions [29]. The centrality of the motor threshold to TMS procedures underscores the need for well-defined, rapid, accurate, and unbiased thresholding procedures.

However, the MEPs in response to stimulation in the primary motor cortex are highly stochastic due to the dynamic variability in cortical neurons and the corticospinal tract — a pulse administered twice with identical strength can lead to a maximum MEP response close to saturation once and no response at all in the other case [30]-[32]. Moreover, the variability is far from Gaussian and appears to involve several variability sources [32]-[35]. Consequently, accuracy of the estimation procedure is hard to guarantee and achieving it with a suboptimal estimation algorithm can be time consuming.

1.2. Existing motor threshold estimation methods

The TMS motor threshold, $x_{\text{th}}$, is usually defined as the lowest stimulation pulse amplitude, $x$, given as a percentage of the maximum stimulator output (MSO), leading to a peripheral muscle response, $r$, with a 50% probability [27], [28], [36],

$$x_{\text{th }}=\underset{x}{\text{argmin}}\left(P\left(r=1\mid x\right)=50\text{\%}\right).$$ (1)

The binary response $r$ is defined as

$$r=\{\begin{array}{ll}1\hfill & \text{ movement observed, or }{V}_{\text{pp}}\ge 50\mu \text{V}\hfill \\ 0\hfill & \text{ movement absent, or }{V}_{\text{pp}}<50\mu \text{V}\hfill \end{array},$$ (2)

where $V_{\text{pp}}$ is the peak-to-peak MEP amplitude [37]. Historically, the value of $50\mu \text{V}$ was chosen relatively arbitrarily as a low level that is not far from the noise limit of most EMG detection systems [38]-[40]. The most common practice for motor threshold finding, or “threshold hunting”, is estimating the relative frequency of positive and negative responses by appropriate stepping of the stimulus amplitude to find the 50% probability level. Therefore, at each test pulse amplitude, a number of stimuli—usually up to ten—have to be administered, hence the name “five-out-of-ten method”. Such an approximation of the probability by a relative frequency within just ten stimuli can be rather imprecise, and a larger number of trials, e.g., twenty or more, per stimulation amplitude is required to obtain more reproducible results for clinical and research purposes [28], [41]. Given its simplicity, the relative-frequency method is widely used and often serves as a benchmark for other methods, despite concerns regarding its mathematical validity or usefulness [41]-[43].

Another well-described method based on the relative response frequency was proposed by Mills and Nithi [39]. Instead of detecting the threshold directly, it uses one point above threshold (upper threshold) and one below (lower threshold), and the threshold estimate itself is defined as the mean of these two points. Mathematically, this relates to a local linearization of the sigmoidal probability distribution of the binary neuronal response as a function of the stimulation amplitude. Similarly, both upper and lower thresholds are defined statistically—all ten responses suprathreshold and subthreshold, respectively. Both the relative-frequency and Mills–Nithi methods require a high number of stimuli [40], [44] and consequently a long duration to complete due to the necessary pause between two stimuli for preventing inter-pulse correlation [45]-[47]. Furthermore, a large number of stimuli also bears a risk of unintended neuromodulation and side effects [14], and therefore thresholding methods should ideally use fewer pulses.

Instead of determining relative frequencies, an alternative thresholding method performs a maximum-likelihood (ML) test for a predefined model describing the response probability distribution—most commonly of a cumulative Gaussian function with two parameters, the midpoint/threshold and slope/spread¹. This maximum likelihood estimation (MLE) method was introduced by Awiszus [48] and was adapted from psychophysics where responses have strong stochastic characteristics as well [49], [50]. The result of such parameter estimation by sequential testing (PEST) provides the most likely parameters, which include the threshold. An essential condition is a good selection of the tested stimulus amplitudes [51]. In contrast to frequency-based methods, each amplitude is usually applied just once. Mishory et al. [44] reported that the number of required stimuli and time for determining threshold was markedly lower for ML-PEST compared to the Mills–Nithi two-threshold method, whereas both methods have similar accuracy.

1.3. Limitations of existing methods and alternative approaches

Existing motor threshold estimation methods have some limitations that can be addressed to improve the accuracy and speed of the procedure. A key feature that can notably influence performance is the stepping rule [51], which chooses the next stimulation amplitude adaptively by taking into account the information from the previous observations. For the next stimulation amplitude, the ML-PEST [48] chooses the MLE of the threshold, which is obtained from a reduced single-parameter model of the response and uses a fixed nonstochastic relationship for the spread of the probability function. This relationship is assigned based on averages from representative measurements [48], and therefore does not reflect individual variability. A limitation of this approach is that by not allowing any deviation from a fixed spread–threshold relationship, the method may place all stimuli systematically too high or too low with respect to the optimal stepping for an individual subject, which can lead to slow stepping and large threshold misestimations if the initial stimulation amplitude is poorly chosen [52], [53].

The threshold estimation can be improved by a full two-parameter MLE in which both the threshold and spread are free to vary. Incorporating additional statistical information from prior or typical results of the thresholding procedure can further improve the estimation [52], [54], which is a method adapted from the field of psychophysics [51]. Although Qi et al. called it a Bayesian method [54], this name is not adopted here since the approach incorporates a-posteriori information into a classical Fisher MLE using Bayes’ theorem. This has been known in mathematics [55] and applied fields [56] as maximum a-posteriori (MAP), a term we adopt in this paper.

While Bayes’ theorem has been used for MAP estimation to improve the TMS threshold estimate [52], [54], both ML and MAP methods are point estimators. In contrast to these point estimators, strictly Bayesian methods that are consequently and completely premised on Bayesian statistics [57]-[59] have not been used in TMS. In this text, the Bayesian style particularly refers to the updating rule for the probability function after new observations, which should therefore follow the Bayesian learning paradigm. We adapted the Bayesian estimation method from Kontsevich and Tyler [57] for TMS motor threshold estimation, using the same model as in the point estimators.

Both the relative-frequency methods and the parameter estimation methods adapted from psychophysics reduce the TMS motor responses to binary events (2). The use of EMG for measuring muscle activation by TMS provides a continuous response strength and thus richer information. The EMG motor threshold is not a binary transition between no response and response but a point on the dose–response curve close to the lowest stimulation strength that elicits an excitatory response detectable with conventional EMG recording and analysis methods [60]-[62]. Further, the motor threshold is likely not the lowest stimulation strength to evoke detectable responses, as recent studies demonstrate that the $50\mu \text{V}$ threshold is likely closer to the middle of the log-transformed dose–response curve of MEPs when it is separated from the baseline noise [63], [64]. The smoother transition of the expected value of the responses in dependence of the stimulation strength may allow the use of the MEP amplitude to find the threshold faster. To consider this additional analog information of the response amplitude appropriately, stochastic root-finding methods [65], [66] (also known as stochastic approximation) are adapted and evaluated here.

1.4. Organization of this paper

We first describe existing TMS thresholding methods as well as the aforementioned improvements and introduce the Bayesian estimation and stochastics root-find methods, covering a total of three classes and 22 variations of methods. To evaluate and compare the thresholding methods, an independent testbed incorporating a detailed stochastic MEP model [61] is used, with 25,000 virtual subjects. The performance of the thresholding methods is analyzed based on their accuracy of the threshold estimate and speed to acquire threshold (relative-frequency methods) or convergence rate of the threshold error (other methods). The best-performing methods of each class are identified and the trade-off between different methods is discussed. The results show that the three alternative approaches—two-parameter MAP estimation, Bayesian estimation, and stochastic root-finding—outperform single-parameter ML estimator, and all these methods require significantly fewer TMS pulses for accurate motor threshold estimation than the conventional relative-frequency methods.

2. Methods

2.1. Overview of thresholding methods

The estimation of a neuronal property such as the excitation threshold can be performed in two fundamentally different ways. The model-based parameter estimators (MLE, MAP, and Bayesian methods) require a probabilistic description of the input–output black box being analyzed in dependence of a certain parameter set. An inaccurate model will very likely deteriorate the quality of the estimate. In contrast, the early approaches, namely the five-out-of-ten or ten-out-of-twenty relative-frequency method based on recommendations of the International Federation of Clinical Neurophysiology (IFCN) [27], [28] or the Mills–Nithi two-threshold method [39], do not need such a predefined model description as they refer directly to the definition of the threshold. The stochastic root-finding methods can also be counted among this class of non-parametric systems. In the following, the existing and several new or adapted algorithms (Table 1) are shortly described and compared for the application in TMS. The thresholding methods were simulated with Matlab (version 2021a, The Mathworks, Natick, MA, USA) on the Duke Compute Cluster (code available online at [67]) and data analysis and visualization were performed on a desktop computer (MATLAB, version 2023a).

Table 1.

Summary of thresholding methods

Class	Methods	Variation, threshold estimate ${\mathit{x}}_{\mathbf{t}\mathbf{h},\mathit{i}}$ and stepping ${\mathit{x}}_{\mathit{i}+1}$, or convergence of ${\mathit{a}}_{\mathit{i}}$
Relative-frequency based	IFCN relative-frequencya	Five-out-of-ten (5/10)
		Ten-out-of-twenty (10/20)
	Mills-Nithia	Original
		Modified
Parametric estimation	MLE-1b,c, MLE-2b	$x_{\text{th},i}=x_{i+1}=t_{\text{MLE},i}$
	MAP-1d, MAP-2	$x_{\text{th},i}=x_{i+1}=t_{\text{MAP},i}$
	Bayesian, Bayesian-NPe	$x_{\text{th},i}=E(t);x_{i+1}=\text{argmin }E\left(H_i(x,r)\right)$
	Bayesian-TS	$x_{\text{th},i}=x_{i+1}=E(t)$
Stochastic root-finding	ACS-H, DCS-H	Harmonic ($i^{-1}$), non-adaptive
	ACS-HA, DCS-HA	Harmonic ($i^{-1}$), adaptive
	ACS-GA, DCS-GA	Geometric ($2^{-i/2}$), adaptive
	ACS-GHA	Geometric ($2^{-i/2}$) switchingf to harmonic ($i^{-1}$), adaptive

^aEach variation implemented with both 1% and 2% MSO step size.

^bImplemented using both direct/implicit maximization and explicit evaluation of the log-likelihood function.

^cAlso known as adaptive threshold hunting/estimation [48], best-PEST [48], [54], MLE [68], or ML-PEST [53]. Implemented in MTAT [69] and ATH-tool [70].

^dAlso known as Bayesian-PEST [54].

^eImplemented with no prior population information.

^fTransition when the maximum change of $x_i$ was expected to be less than 1.5% MSO.

2.2. Nonparametric methods

2.2.1. IFCN relative-frequency method

The procedure described by the IFCN [27], [28], [71] leaves freedom of interpretation of some details. For the analysis in this paper, the relative-frequency method was defined as follows to realistically represent the implementation in many laboratories:

1. The starting amplitude is known from the hotspot search (see section 2.4.2 Simulation routine for further details on the starting amplitude), in which the corresponding neuron population being responsible for activating the observed muscle group (e.g., the first dorsal interosseus or the abductor pollicis brevis) is located with test stimuli on the primary motor cortex. The required amplitude adjustments in this step lead to an amplitude not too far above the desired threshold as spatial focality would be compromised with stronger stimuli. Starting at an amplitude too high above threshold further risks discomfort and other side effects, especially for subjects with high sensitivity and/or low threshold.

2. At a given stimulation amplitude, a number of stimuli are applied up to a maximum of ten or twenty until either the count of positive responses reaches half the number (five or ten, respectively) or that of negative responses exceed half the number (six or eleven, respectively).

3. Being suprathreshold by definition in the situation of positive responses accounting for at least half the number of stimuli, the stimulation amplitude is stepped downward, typically by 1% or 2% MSO [28], [71], and Step b) is repeated.

4. The latter outcome in Step b) precludes 50% positive responses within the number of stimuli and the amplitude is subthreshold. The threshold itself is defined as the last level before this termination point.

Versions with the number of stimuli of ten and twenty and step size of 1% and 2% MSO were performed to show the effect of the number of stimuli per step and the step size on the accuracy and the number of stimuli required for the methods.

2.2.2. Mills–Nithi two-threshold method

The original Mills–Nithi method [39] was carried out as follows:

1. The subject is stimulated with pulses starting at an amplitude of 20% MSO and increasing in 10% MSO steps until the first positive response is identified.

2. Applying up to 10 pulses per stimulation amplitude, the stimulation is then decreased in 1% MSO steps when a positive response is observed. The intensity at which all 10 stimuli produce no response is designated as the lower threshold.

3. Starting from the lowest amplitude that has not resulted in a “no response”, i.e., the last amplitude that has one stimulus with positive response during Step b), and increasing the stimulation in 1% MSO steps, the upper threshold is designated as the minimum intensity at which 10 stimuli all produce positive responses.

4. The threshold is calculated as the mean of the lower and upper thresholds.

In our study, a modified version of the Mills–Nithi method was performed using the same starting amplitude as the IFCN relative-frequency method. The first negative response was identified by decreasing the amplitude in 10% MSO steps and the upper and lower thresholds were then similarly obtained as in the original methods but in reversed order. Step sizes of 1% and 2% MSO were tested.

2.2.3. Stochastic root-finding methods

The IFCN relative-frequency and Mills–Nithi methods described above both require binary information reduction through quantization of the response (above versus below $50\mu \text{V}$). In contrast to such quantization, robust numeric root-finding algorithms that are still relatively unknown in TMS can directly act on analog data, e.g., the voltage amplitude of the MEP response. Preserving valuable continuous information in the response, such root-finding approaches could enable an improvement of convergence. The sequential formulation proposed here was derived from the method of Robbins and Monro [65], [66], with the neuronal variability playing the role of a noise source. The requirements for this approach are rather weak and comprise constant monotonicity on average [65].

The next stimulation amplitude, $x_{i+1}$, was defined relative to the current one using an iterative analog control sequence (ACS) that was intended to converge towards the threshold:

$$x_{i+1}=x_i-a_i\cdot \Delta y_i=x_i-a_i\cdot \left(y_i-y_{\text{th}}\right),$$ (3)

where $i\ge 1$ was the step number, $a_i$ was the step size of the control sequence, $x_i$ denoted the stimulation amplitude and the current threshold estimate $\left(x_{\text{th},i}=x_i\right),\Delta y_i=y_i-y_{\text{th}}$, and $y_i$ and $y_{\text{th}}$ were logarithms of the response $\left(y_i=\text{lg}\left(V_i/\left[1\text{ V}\right]\right)\right)$ observed at step $i$ and at the MEP threshold $V_{\text{th }}(50\mu \text{V}$ peak-to-peak by definition), respectively.

The stochastic root-finding sequence converged towards a stimulation amplitude generating an average/expected $50\mu \text{V}$ MEP, which should be close to the threshold amplitude generating a median $50\mu \text{V}$ MEP. The log-transformation accounted for the first-approximation log-normal distribution of the MEP response amplitude following an inductive stimulus in the motor cortex [46]. The resulting distribution of $y_i$ had reduced skewness and should be closer to Gaussian and therefore statistically well-behaved, e.g., with mean close to median (Figure 1A_i). Performing the whole procedure in the logarithmic space also granted the requirement of constant monotonicity. The iteration using the MEP responses thus became

$$x_{i+1}=x_i-a_i\text{lg}\left(V_i/V_{\text{th}}\right).$$ (4)

Figure 1: TMS response model.

A. An example of the TMS sigmoidal recruiting behavior that is mimicked by the MEP model. i. In the EMG space, this virtual stimulator provides analog values, which some of the algorithms can utilize. The model provides uncorrelated MEP responses (gray dots), which are independent from the tested black-box models in the parameter estimators to avoid model-determined advantages. The median MEP amplitude (black line) crosses $50\mu \text{V}$ (black dashed line) at the stimulation threshold (blue line), whereas the mean (log-transformed) MEP amplitude (red dashed line) may not. ii. In the probability space, the threshold is determined from the relative frequency of responses by linear regression (blue line) to find the 50% probability amplitude. The starting amplitude was determined as the minimum amplitude at and above which the response probability was no less than 95%. A Gaussian response model (red line) can also be obtained by applying MLE to the stimulation-response pairs, which fitted well to the relative frequencies for the majority of subject. B. The ground truth thresholds and the slope/spread at the threshold are obtained by linear regression for the 25,000 virtual subjects. i. Joint distribution of threshold and spread. ii.-iv. Distributions of threshold, spread, and spread-to-threshold ratio. The threshold and spread distributions are close to normal or log-normal, with the fitted distributions shown by the red dashed lines. The nominal 0.07-ratio is shown by black dashed lines.

We implemented several versions of the sequence of step sizes $a_i$ to compare the convergence properties. In the basic version ACS-H, the step size followed a harmonic series

$$a_i=a_0/i,\text{ for }i\ge 1,$$ (5)

with almost sure (a.s.) convergence² for the stimulation amplitude towards its target.

Following the style of Delyon and Juditsky [72], adaptive methods were utilized to accelerate convergence by adjusting the control sequence at step $i(i>1)$ only when the response changed from subthreshold to suprathreshold or vice versa from the previous step $(i-1)$, i.e., following a sign change in $\Delta y_i:\text{sgn}\left(\Delta y_i\right)\ne \text{sgn}\left(\Delta y_{i-1}\right)$. For harmonic convergence, the corresponding adaptive control sequence ACS-HA was

$$a_i=a_0/\left(c_i+1\right),\text{ for }i\ge 1,$$ (6)

where $c_i$ was the number of sign changes of $\Delta y_i$ up to step $i$,

$$c_i=\{\begin{array}{ll}{c}_{i-1}\hfill & \text{sgn}\left(\Delta y_i\right)=\text{sgn}\left(\Delta y_{i-1}\right)\hfill \\ c_{i-1}+1\hfill & \text{sgn}\left(\Delta y_i\right)\ne \text{sgn}\left(\Delta y_{i-1}\right)\hfill \end{array},\text{ for }i>1,$$ (7)

with $c_1\triangleq 0$. Unlike classical mathematical problems, the TMS sequence has only a very limited number of steps. Thus, quick convergence seems more attractive here rather than any statement for the case of infinite duration. To test faster geometric convergence of $2^{-i/2}$, ACS-GA used an adaptive control sequence of

$$a_i=a_0\cdot 2^{-c_i/2}\text{. }$$ (8)

Further variations of these methods were designed to examine the convergence rate and performance. To take advantage of both fast convergence and the a.s. convergence properties, a hybrid method ACS-GHA was designed to switch the control sequence $a_i$ from geometric to harmonic convergence when the threshold estimate was close to the true threshold and the expected change in $x_i\left(\left|\Delta x_i\right|=\left|a_i\cdot \Delta y_i\right|\right)$ was small—less than 1.5% MSO. Quantized versions, i.e., digital control sequences (DCS), were implemented for comparison as well. The analog value $\Delta y_i$ was reduced using a modified sign function to $\delta y_i=\text{sgn}\left(\Delta y_i\right)\in \{-1,+1\}$ for the two different cases of below threshold and above (or equal) threshold. DCS allowed the stochastic root-finding method to be used also without EMG recordings, just “response” and “no response” inputs, in a manner compatible with the other estimation methods. We also explored three second-order analog variations of root-finding methods, which did not improve the performance [73] and were not reported here.

The definition of appropriate initial step size $a_0$ was assisted by the iteration formulation (3), which resembled a slope [74] after rearrangement

$$a_i^{-1}=\frac{y_{\text{th}}-y_i}{x_{i+1}-x_i}.$$ (9)

For reaching the threshold quickly, $a_0$ should accordingly be not too far from the cotangent of the line connecting the present stimulation–response $\left(x_i,y_i\right)$ towards the threshold $\left(x_{\text{th}},y_{\text{th}}\right)$. Larger values increase the speed of approach of $x_i$ towards $x_{\text{th}}$, but may cause the stepping to overshoot and oscillate around the target, whereas smaller numbers promote lengthy one-sided convergence. Statistical information about typical and maximum slopes can inform the choice of $a_0$. For the simulations in this paper, the default initial step size was chosen such that it approximated typical cotangents around the inflection point of the recruitment curve (see section 2.4.1 Response model)

$$a_0=\frac{0.2}{\text{lg}\left({10}^{-2}\right)-\text{lg}\left({10}^{-5}\right)}\approx 0.067.$$ (10)

Here lg⁡(10⁻⁵) and lg⁡(10⁻²) (i.e., $10\mu \text{V}$ and 10 mV, respectively) were approximate response amplitudes near the EMG noise floor and saturation level at the lower and higher ends of the input–output curve, respectively, and 0.2(20% MSO) was a sufficient change in stimulus strength to cover the transition of the typical sigmoidal response curve. Initial step size from one tenth to ten times the default values evenly sampled on a logarithmic scale were also tested.

2.3. Parametric methods

2.3.1. Basic maximum-likelihood estimation method

The implementation of the basic MLE method (ML-PEST) followed substantially the suggestions and definitions by Awiszus [48]. The response probability was described by a cumulative Gaussian model with two degrees of freedom—the threshold $t=x_{\text{th}}$ and the spread $s$:

$$p\left(r=1\mid x,t,s\right)=\frac{1}{s\sqrt{2\pi }}{\int }_{-\infty }^x\text{exp}\left(-\frac{{(x-t)}^2}{2s^2}\right)\text{d}x.$$ (11)

Here, $r$ denoted the binary response (2) of a single stimulus $x$, and the slope at the threshold, $(s\sqrt{2\pi }{)}^{-1}$, was inversely related to the spread.

The likelihood function was defined for independent events

$$p\left(r\mid x,t,s\right)=\prod _{\left\{i\mid r_i=1\right\}}p\left(r_i=1\mid x_i,t,s\right)\prod _{\left\{i\mid r_i=0\right\}}p\left(r_i=0\mid x_i,t,s\right),$$ (12)

where $\mathbf{x}$ and $\mathbf{r}$ were the lists of stimuli and corresponding responses, respectively, and the probability for failing to evoke a response was $p(r=0\mid x,t,s)=1-p(r=1\mid x,t,s)$. According to the classical Fisher approach, the maximum of this expression for the arguments $t$ and $s,\left(t_{\text{MLE}},s_{\text{MLE}}\right)={\text{argmax}}_{(t,s)}p(\mathbf{r}\mid \mathbf{x},t,s)$, provided the most likely parameter set, i.e., the MLE. Due to the strict monotonicity of the logarithm, the product can be split into an equivalent sum, which formed the log-likelihood and was also numerically more robust. The MLE of $t$ at step $i$ provided both the threshold estimate at the current step $x_{\text{th},i}$ and stimulation amplitude at the next step $x_{i+1}$, i.e., $x_{i+1}=x_{\text{th},i}=t_{\text{MLE},i}$.

Previous implementations of MLE (12) (e.g., MTAT and ATH-tool, [48], [69], [70]) were simplified by using only the threshold parameter $t$, whereas the spread parameter was fixed to the value $s=0.07\cdot t$. We replicated this method (MLE-1) and also performed a full MLE with both parameters (MLE-2). Two approaches were implemented and compared to find the threshold $t$ that maximize the likelihood function. The first approach directly maximized the likelihood function over $t$ (implicitly using MATLAB function fmincon, applied on the negative of the log-likelihood). This approach, however, could be unstable during the initial steps when there were few amplitude and response samples $(\mathbf{r},\mathbf{x})$ and the likelihood function had a plateau, i.e., a range of $t$ or $(t,s)$ combinations resulted in the same maximum. The second approach explicitly evaluated the likelihood function for a dense sampling of the parameters $t$ and $s$ and $t_{\text{MLE}}$ was calculated as the mean of the argument $t$ over the range of the maximum plateau of the likelihood function [48]. The parameter space was discretized with $\Delta t=0.002$ for $t\in \left(\mathrm{0,1.3}\right]$; for MLE-2, 100 logarithmical-distributed samples $s$ were chosen from [0.005, 0.5] and the mean of $t$ was calculated for all $(t,s)$ combinations for which the likelihood function was at its maximum (i.e., if for different $s$ samples, the likelihood function reached maximum for the same $t$, these $t$ samples were repeated in the average). The upper bound of the threshold parameter exceeded one, which is explained in Section 2.4.1 Response model.

2.3.2. Maximum-a-posterior estimation method

Modified estimation methods (MAP-1 and MAP-2) were designed based on a MAP-like approach [52], [54]. To improve the performance for common cases, the likelihood function (12) was modified to incorporate statistical information of the parameters using Bayes’ theorem per

$$p\left(\theta \mid x,r\right)=\frac{p\left(\theta \right)\cdot p\left(r\mid x,\theta \right)}{p\left(r\mid x\right)},$$ (13)

in which the vector $\mathit{\theta }=(t,s)$ denoted the parameter set. The additional information about the distribution of typical parameters $p\left(\mathit{\theta }\right)$ had to be provided in advance (see section 2.4.1 Response model). When such information is not available, the MAP algorithm can be designed to collect the information itself, i.e., to estimate and progressively refine the parameter distribution from subjects on which it already performed threshold estimation. In that regard, MLE can be considered a special case of MAP estimation, where the parameters have a wide uniform distribution.

For maximizing the a-posterior likelihood (13) with respect to $\mathit{\theta }$, the last factor $p(\mathbf{r}\mid \mathbf{x})$ was a constant and therefore omitted. Similar to MLE, the calculation was performed on the logarithm of the likelihood, and the parameter $t_{\text{MAP}}$ at step $i$ provided both the threshold estimate at the current step and stimulation amplitude at the next step, i.e., $x_{i+1}=x_{\text{th},i}=t_{\text{MAP},i}$.

2.3.3. Kontsevich–Tyler Bayesian estimation method

The Bayesian methods here were adapted from Kontsevich and Tyler [57]. All central elements were taken as described, using the Gaussian black box (11) as in the other ML and MAP estimators for consistency. Based on this, new information was embedded into the probability density function $p(r\mid x,\mathit{\theta })$ at every step $i$ using Bayes’ theorem to form the posterior probability

$$p_i\left(\theta \mid x,r\right)=\frac{p_i\left(\theta \right)p\left(r\mid x,\theta \right)}{{\int }_{{ℝ}_{+}^n}{p}_i\left(\theta \right)p\left(r\mid x,\theta \right){\text{d}}^n\theta },$$ (14)

where $p_i\left(\mathit{\theta }\right)$ was the probability distribution of the parameters at step $i$.

The next stimulation amplitude $x_{i+1}$ was determined using an information-theoretical approach which minimized the expected entropy at the next step [57]

$$x_{i+1}=\underset{x}{\text{argmin }}E\left(H_i\left(x,r\right)\right)\text{. }$$ (15)

Here, the entropy $H_i(x,r)$ was calculated from the posterior probability as

$$H_i\left(x,r\right)=-{\int }_{{ℝ}_{+}^n}{p}_i\left(\theta \mid x,r\right)\text{ log}\left(p_i\left(\theta \mid x,r\right)\right){\text{d}}^n\theta ,$$ (16)

and the expectation $\mathbf{E}$ was a weighted sum of the entropy $H_i(x,r)$

$$E\left(H_i\left(x,r\right)\right)=\sum _{r=0,1}{p}_i\left(r\mid x\right)H_i\left(x,r\right).$$ (17)

The weights were the probability $p_i(r\mid x)$ of sub- and supra-threshold responses at step $i$, calculated from the current parameter distribution and the Gaussian model (11)

$$p_i\left(r\mid x\right)={\int }_{{ℝ}_{+}^n}p\left(r\mid x,\theta \right)\cdot p_i\left(\theta \right){\text{d}}^n\theta ,r\in \left\{0,1\right\}.$$ (18)

Each stimulus–response pair narrowed down the posterior probability density distribution to

$$p_{i+1}\left(\theta \right)=p_i\left(\theta \mid x_{i+1},r_{i+1}\right),$$ (19)

and the expected value provided the estimates for the parameters at each step per

$${\theta }_i^{\left(l\right)}=E\left({\theta }^{\left(l\right)}\right)={\int }_{{ℝ}_{+}^n}{\theta }^{\left(l\right)}{p}_i\left(\theta \right){\text{d}}^n\theta ,$$ (20)

where $l$ was the parameter index (i.e., $l=\mathrm{1,2}$ for $\mathit{\theta }=(t,s)$). Therefore, the threshold estimate was $x_{\text{th},i}=t_i=\mathbf{E}\left(t\right)$. In contrast to the point estimator methods, the probably density functions (11) and (14) were explicitly computed and stored for the whole parameter space: $r\in \left\{\mathrm{0,1}\right\},x\in \left(\mathrm{0,1.3}\right],t\in \left(\mathrm{0,1.3}\right]$, and $s\in \left[\mathrm{0.005,0.5}\right]$. For computational purposes, the parameter space was discretized with $\Delta x=0.01$ and $\Delta t=0.002$, and 100 logarithmical-distributed samples were chosen for $s$, corresponding to 8.5 million points for $p(r=1\mid x,\mathit{\theta })$. Such sampling density turned out sufficient as doubling the number of samples did not improve performance in earlier exploration (results not shown). The initial distribution $p_0\left(\mathit{\theta }\right)$ was chosen as the population distribution $p\left(\mathit{\theta }\right)$ (see section 2.4.1 Response model).

Two variations of the Bayesian method were implemented. The first one had no prior (NP) population information—using a uniform distribution for $p_0\left(\mathit{\theta }\right)$—to evaluate the contribution of the prior information to convergence. To compare the entropy minimization stepping rule against the threshold stepping (TS) of the MLE and MAP methods, the second variation skipped the entropy calculation ((15)–(18)) and used the latter’s choice of the threshold estimate at step $i$ (20) for the stimulation amplitude for the next step: $x_{i+1}=x_{\text{th},i}=\mathbf{E}\left(t\right)$.

2.4. Testbed

2.4.1. Response model

We implemented a realistic MEP model [61] which included variability patterns such as amplifier noise, saturation, and neural variability behavior [30], [34], [35]. The model’s increased complexity compared to other numerical studies concerning threshold-estimation methods [48], [75] was intended not only to reproduce more faithfully MEP behavior but also to avoid using the very same models underlying the parameter estimation methods for the test procedure, which would risk numerical favoritism for those particular methods. The response model provided analog MEP amplitude values based on a stochastic recruitment curve (Figure 1A_i), with parameters mimicking real measurements [76]. The analog amplitude response, i.e., the peak-to-peak MEP amplitude $V_{\text{PP}}$ in units of volt, was defined as

$$V_{\text{PP}}\left(x\right)={ϵ}_{\text{yadd},ij}\left(p_{1,j},p_{8,j},p_{9,j}\right)+{10}^{{ϵ}_{\text{ymult},ij}\left(p_{6,j}\right)}\cdot {10}^{p_{2,j}/\left(1+\frac{p_{3,j}}{{\left(x-p_{5,j}+{ϵ}_{\text{xadd},ij}\left(p_{7,j}\right)\right)}^{p_{4,j}}}\right)},$$ (21)

where $p_{1,j}$ to $p_{9,j}$ were the individual response parameters for subject $j$ obtained from an experimental population [34], and intra-individual variability sources ${ϵ}_{\text{xadd},ij}$ and ${ϵ}_{\text{ymult},ij}$ and the additive noise ${ϵ}_{\text{yadd},ij}$ varied between subjects $j$ and for pulses $i$ along distributions with subject-dependent parameters. The distributions of recruitment parameters, intra-individual variability, and noise are given in the Appendix. While nominally the stimulation amplitude $x$ should be in the range [0, 1], we extended the range of possible stimulation amplitudes up to 130% MSO in this study, which was 30% above the maximum threshold within the virtual population. This extension was performed to focus on the performance of the thresholding algorithms themselves and eliminate influence from the device and choice of stimulation target. Specifically, the MEP response model was fitted to experimental data obtained with a controllable pulse parameter TMS device using $60\mu \text{s}$ pulse width with the maximum capacitor voltage chosen to match stimulation strength to conventional TMS devices for a specific muscle target [34], [76], [77], and stimulation amplitudes higher than the model’s nominal 100% MSO could be achieved by increasing the capacitor voltage or with a different TMS device.

Model parameters were generated for a total of 25,000 virtual subjects. Each subject’s input–output curve was characterized using 100,000 stimulation–response pairs, with a higher number of pulses applied near the amplitudes that generated a $50\mu \text{V}$ response in the corresponding noiseless model (see Appendix). The ground truth threshold for each subject was determined by linear regression of the relative frequency of suprathreshold responses to find the amplitude with 50% response probability (Figure 1, panels A_ii, B_i, and B_ii). The regression also provided the slope at the threshold, which was converted to the spread for the corresponding Gaussian model (Figure 1, panels B_i and B_iii). The ratio between the spread and threshold (Figure 1B_iv, median, mean, and SD of 0.076, 0.077, and 0.012, respectively) was close to but on average ~10% larger than the fixed nominal value of 0.07 from a previous study [48] and also had a wide distribution. The joint probability density distribution of threshold and spread $p(t,s)$ can be fitted with normal and lognormal distributions, respectively, with details given in the Appendix. The fitted distribution $p(t,s)$ served as prior information $p\left(\mathit{\theta }\right)$ or $p_0\left(\mathit{\theta }\right)$ for the MAP and Bayesian methods, respectively. By applying MLE to each subject, the parameters $t$ and $s$ for the Gaussian model were obtained and analyzed (Figure S1A). For approximately 90% of subjects, the response probability distributions estimated with Gaussian MLE matched their ground truth well (Figure S1B). For the remaining 10% of subjects, however, MLE resulted in large errors (especially for the spread parameter), showing that the Gaussian model was not a universal fit.

2.4.2. Simulation routine

The testbed comprised all threshold detection algorithms defined previously and the response model. In order to meet the real conditions of a threshold search, each algorithm was given information from the spatial hotspot search, i.e., a starting amplitude which was known to be above the threshold [52], [78], [79]. To avoid ambiguity and circular dependence between the definitions of threshold amplitude and hotspot location [80], the amplitude for the hotspot search was chosen as the amplitude that for at least 95% of the stimuli (and thus “consistently” [28]) generated a response (Figure S2). The starting amplitude was on average 8.7% MSO higher than the ground truth threshold (1.13 times threshold). For the algorithms that use specific integer amplitudes (i.e., IFCN relative-frequency and Mills–Nithi methods), the starting amplitude was rounded to 1% MSO.

The responses for subsequent stimuli were provided to the algorithms by the full MEP model. Whereas the analog versions of the stochastic root-finding methods directly processed this continuous information, all other methods quantized the response into the binary representation of above and below threshold first. To examine both their short- and long-term behaviors, the parametric estimation algorithms and stochastic root-finding methods were tested with a maximum of 200 steps each. The absolute and relative errors of each threshold estimate $x_{\text{th,est}}$ versus the ground truth $x_{\text{th,GT }}$ were analyzed with

$$\begin{gathered} \Delta x_{\text{th}}=x_{\text{th},\text{est}}-x_{\text{th},\text{GT},} \\ \delta x_{\text{th}}=\frac{\Delta x_{\text{th}}}{x_{\text{th},\text{GT}}}. \end{gathered}$$ (22)

For comparison with previous studies in the literature, the absolute values of the errors $\left|\Delta x_{\text{th}}\right|$ and $\left|\delta x_{\text{th}}\right|$ were also analyzed, of which $\left|\delta x_{\text{th}}\right|$ was previously referred to as the “relative error” [48], [54]. The results mostly present the relative errors $\delta x_{\text{th}}$, as they provide normalized distance of the threshold estimate from the true value expressed as a percentage of the latter. The absolute error $\Delta x_{\text{th}}$ overall had similar distributions and are shown in the supplement.

3. Results

3.1. IFCN relative-frequency method and Mills–Nithi two-threshold method

As both methods have been previously established in the field, only the high-level performance is described here, with more details and full statistics provided in the supplement (Tables S1–S7).

For the IFCN relative-frequency methods (Figures 2 and S3–S5), the number of stimuli until the method terminated and reported a threshold value for a given subject had a wide distribution. On the lower end, the methods stopped after reducing the stimulation amplitude by two or three steps below the starting amplitude, which could be as few as 15 to 20 stimuli for the five-out-of-ten methods or between 40 and 60 for the ten-out-of-twenty methods. On the other hand, some subjects required a larger number of stimuli, reaching over 100 or even 200 depending on the method. Estimates obtained with fewer stimuli overestimated the threshold due to false early fulfillment of the termination rule, whereas those with large numbers of stimuli leaned towards underestimation due to false continuation at an amplitude with response probability less than 50%. The errors were linearly dependent on the number of stimuli with coefficients of determination $\left(r^2\right)$ more than 0.7, and overall had a positive bias (Figure S5, panels B_i and B_ii; e.g., median, mean, and SD of 0.91%, 1.00%, and 1.75%, respectively, for $\delta x_{\text{th}}$ of the ten-out-of-twenty method with 1% MSO stepping size). Such linear relationship and bias were inherently determined by the binomial probability distribution to either terminate or continue (Figure S4; see method for calculating the probability in the Supplement), consistent with a previous argument against relative-frequency methods [43]. The number of stimuli drastically decreased by approximately 40% and 50% when the stepping size was increased from 1% to 2% MSO and the coarser five-out-of-ten rule was used instead of the ten-out-of-twenty rule for the relative frequency approximation, respectively, although the shape of the distribution was not affected qualitatively. These changes overall decreased the threshold accuracy (i.e., increased median, mean, and standard deviation of errors) as intuitively expected. However, the errors had similar ranges for both ten-out-of-twenty or five-out-of-ten methods with either step sizes. Therefore, it appears ineffective to use smaller step size and/or a larger number of stimuli at a given amplitude to improve the accuracy, as the threshold error of individual subjects does not necessarily decrease but the required number of pulses increases significantly.

Figure 2: Relative threshold errors of the IFNC relative-frequency and Mills–Nithi methods.

The relative errors $\delta x_{\text{th}}$ are plotted against the number of stimuli required for all the virtual subjects as dots. Several statistics are shown, with median as white circle, interquartile range between first and third quartiles (Q₁ and Q₃) as thick error bars, and ranges extending to the lower and upper adjacent values—which are 1.5 times the interquartile range below Q₁ and above Q₃ and cover approximately ± 2.7 standard deviations for normally-distributed data—as thin error bars. A probability density estimate based on a normal kernel function is also shown with the contour lines. Linear regressions are shown for the IFCN relative-frequency methods as red lines. A. IFNC ten-out-of-twenty method. i. 1% MSO step size. ii. 2% MSO step size. B. IFNC five-out-of-ten method. i. 1% MSO step size. ii. 2% MSO step size. C. Original Mills–Nithi methods. i. 1% MSO step size. ii. 2% MSO step size. D. Modified Mills–Nithi methods. i. 1% MSO step size. ii. 2% MSO step size. Note different horizontal scale in panel A_i versus the others.

The Mills–Nithi methods (Figures 2, S3, and S5) had narrow distributions for the number of stimuli, requiring at least 20 stimuli but rarely exceeding 100. The numbers of stimuli for all four variations were on average similar to or smaller than the IFCN five-out-of-ten method with 1% MSO step size but larger than that with 2% MSO step size. In comparison, the IFCN relative-frequency method could be much faster for any individual subject due to its false early termination and/or if the starting point was already close to the threshold (i.e., steep slope near the threshold). The modified Mills–Nithi methods exploited the additional information obtained from the hotspot search but only reduced the number of stimuli by three or four on average compared to the original method using the same step size. Nevertheless, none of the Mills–Nithi methods resulted in extremely large numbers of stimuli. Due to the averaging of the upper and lower thresholds, the errors of the Mills–Nithi methods were independent of the number of stimuli $\left(r^2<0.01\right)$ and, compared to the IFCN relative-frequency methods, had tighter distributions and were centered closer to zero. Compared to the modified methods, the original Mills–Nithi methods had wider error distributions and many outliers with large errors. Increasing the step size to 2% MSO for either the original or modified method reduced the number of stimuli on average by 14 or 15 steps and narrowed its distribution. Increasing the step size also widened the error distribution slightly.

3.2. Parametric estimation

Unlike the relative-frequency methods, the parametric estimation methods generated a threshold estimate and thus an error at every step (relative error $\delta x_{\text{th }}$ in Figure 3A and absolute error $\Delta x_{\text{th}}$ in Figure S6A). The median $\left|\delta x_{\text{th}}\right|$ (Figure 3B) and the interquartile range (IQR) of $\delta x_{\text{th}}$ characterized how close the errors approach zero and how wide the errors were distributed. The IQR of $\delta x_{\text{th}}$, and therefore the range between the lower and upper adjacent values (which were 1.5×IQR below and above the first and third quartile) as well, correlated well with the median $\left|\delta x_{\text{th }}\right|$ : the ratio between these two statistics after a few steps was approximately 2 and was very stable (Figure S7A). Therefore, the median $\left|\delta x_{\text{th}}\right|$ captured the distribution of most errors, with the remaining outliers summarized as a percentage of the total number of subjects (Figure S7B). Besides the step-by-step distributions of errors, a few metrics were used to benchmark and compare the performance (Table 2). Estimation speed was quantified by the number of steps for a method to reach the same median $\left|\delta x_{\text{th}}\right|$ as the IFCN five-out-of-ten method with 1% MSO step size (median of 64 steps and median $\left|\delta x_{\text{th}}\right|$ of 1.6%). Whereas recent IFCN guidelines [28] recognized that the relative-frequency methods require ten-out-of-twenty trials to generate reliable results, the five-out-of-ten method is still widely used [81]-[84] and therefore chosen as the reference. For accuracy, the median $\left|\delta x_{\text{th }}\right|$ was compared at 20 and 30 steps, which are respectively considered to produce a sufficiently accurate estimate for all subjects using existing ML-PEST (MLE-1) [41], [48] and commonly used as the upper limit to terminate adaptive thresholding procedures [53], [69], [70].

Figure 3: Relative threshold errors of the parametric estimation methods.

For MLE methods, results from implicit (im.) maximization of the likelihood function are shown. A. Statistics are shown up to the seventieth steps as boxes-and-whiskers at the first step and every ten steps starting from the tenth step, with median as marker, interquartile range as thick vertical line, upper and lower adjacent values as whiskers (thin vertical lines), and outliers as dots. The lines show the median at every step. The boxes-and-whiskers for different methods are slightly offset for clarity. The log-lin-log mixed scale is used to better visualize the errors around zero, with linear scaling in the range of [−1%, 1%] and logarithmic scaling of amplitudes outside this range. The median, interquartile range, and range of adjacent values of the IFCN five-out-of-ten method with 2% MSO step size are shown as a circle and black error bars as in Figure 2 B_ii for reference. B. The median $\left|\delta x_{\text{th}}\right|$ on logarithmic scale, with the inset showing the details in the first 30 steps (the gray area in the main panel).

Table 2.

Error convergence of parametric estimation and root-finding methods

Method	Step reaching median $\left|\delta x_{\text{th}}\right|$ of 1.6%	Median $\left|\delta x_{\text{th}}\right|$ at step 20	Median $\left|\delta x_{\text{th}}\right|$ at step 30	Optimal $a_0$ (ratio to default value)
MLE-1a	20 (21)	1.57% (1.62%)	1.25% (1.26%)	–
MAP-1	27	1.96%	1.48%
MLE-2a	40 (39)	2.26% (2.27%)	1.85% (1.83%)
MAP-2	18	1.51%	1.21%
Bayesian	18	1.48%	1.198%	–
Bayesian-NP	37	2.37%	1.81%
Bayesian-TS	18	1.50%	1.196%
ACS-Hb	24 (18)	1.69% (1.51%)	1.46% (1.31%)	×1.6
DCS-Hc	17 (16)	1.45% (1.45%)	1.17% (1.18%)	×0.79
ACS-HAc	18 (18)	1.52% (1.51%)	1.280% (1.30%)	×0.79
DCS-HAb	22 (19)	1.67% (1.54%)	1.37% (1.24%)	×0.63
ACS-GAc	20 (20)	1.59% (1.58%)	1.45% (1.47%)	×0.79
DCS-GAc	22 (21)	1.68% (1.62%)	1.43% (1.44%)	×0.63
ACS-GHAb	19 (18)	1.53% (1.514%)	1.283% (1.29%)	×0.63

^aResults obtained with direct maximization of the likelihood followed by those using explicit evaluation in parentheses.

^bResults obtained with default $a_0$ followed by those with optimal $a_0$ for lowest median $\left|\delta x_{\text{th}}\right|$ at both step 20 and 30 in parentheses.

^cResults obtained with default $a_0$ followed by those with optimal $a_0$ for lowest median $\left|\delta x_{\text{th}}\right|$ at step 20 in parentheses, with default $a_0$ being optimal for lowest median $\left|\delta x_{\text{th}}\right|$ at step 30.

The basic MLE method (MLE-1, Figure 3, dark blue lines with circle markers), which used a single-parameter MLE for threshold estimation and stepping [48], reached the same error level as the reference IFCN five-out-of-ten method with only 20 steps (Table 2), and the error further decreased by about 20% at 30 steps. Evaluating the likelihood function explicitly had little effect on the threshold estimation performance (Table 2, results in parentheses, and Figure S8) and computational runtime (Figure S9) compared to implicit maximization. Introducing prior information of the parameters of the population (MAP-1, Figure 3, light blue lines with square markers) did not improve the performance for the single-parameter estimation. On the contrary, with the empirical relationship $s=0.07\cdot t$ from another population imposed on the parameter distribution of the virtual population in this study, MAP-1 could not fully utilize the prior distribution. Its performance was considerably worse than MLE-1 as the parameters of some subjects in our model were far from such a 0.07 relationship (Figure 1 B_iv).

Compared to MLE-1, the two-parameter MLE method (MLE-2, Figure 3, dark red lines with circle markers) had a slower convergence and reached the median error level of the IFCN reference with more (40) pulses. The reduced speed of convergence reflected the exploration of the full parameter space: the likelihood function initially had a large plateau region, which reduced in size as more stimulation–response pairs were recorded. Only after a sufficient number of steps did the likelihood plateau narrow to generate a clear optimal estimate. Similar to MLE-1, the method for maximizing the likelihood function had little effect on performance for MLE-2 (Table 2, results in parentheses, and Figure S8). However, the explicit evaluation with two parameters resulted in a much higher computational runtime per step that was two order-of-magnitudes slower (Figure S9). Using a two-parameter MAP estimator for the threshold (MAP-2, Figure 3, orange lines with square markers) significantly improved the performance of MLE-2, reaching the error level of the IFCN five-out-of-ten method with only 18 steps, and had lower median $\left|\delta x_{\text{th}}\right|$ compared to either MLE-1 or MAP-1. As the spread parameter was no longer fixed to the threshold, MAP-2 could match the wide range of behaviors of the subjects in our model and improved the threshold estimate for all subjects. The prior information significantly influenced the likelihood landscape, rendering one parameter pair $(t,s)$ in the otherwise flat maximum plateau unique. The computational runtime of MAP-1 and MAP-2, which used implicit evaluation, were slightly faster their implicit MLE counterparts (Figure S9).

The original Kontsevich-Tyler method (Bayesian, Figure 3, yellow lines with diamond markers) had the least bias in error distribution (Figure 3A), reached the IFCN five-out-of-ten method’s median $\left|\delta x_{\text{th}}\right|$ quickly at step 18 and had faster convergence for small number (< 20) of stimuli (Figure 3B). The Bayesian method achieved more accurate estimation early on by considering the entire function space of the probability density functions. Whereas ML and MAP estimators evaluated the underlying likelihood function in every step anew based on the same key equation, the Bayesian approach added information into the explicitly evaluated and stored distributions from the last iteration. However, such advantages disappeared with more pulses as the point estimator methods gathered sufficient information. Removing the prior population information (Bayesian-NP, Figure 3, purple lines with diamond markers) resulted in slower convergence and performance comparable to that of MLE-2 with a similar slow down comparing MAP-2 to MLE-2, showing that the prior population information narrowed the distribution of parameters in the probability space alike for the point and distribution estimators. Utilizing the threshold estimate for stepping (Bayesian-TS, Figure 3, green lines with diamond markers) had similar performance compared to the original method of minimizing the expected entropy, demonstrating that the threshold was at or very close to the stimulation amplitude that provided the maximum information gain. This provided further validation for choosing the threshold estimate as the next stimulation amplitude for the MLE and MAP methods, despite the difference of how the estimate was obtained—for the Bayesian methods, the threshold was estimated as the expected value based on its probability distribution (20), whereas for MLE and MAP methods the estimation came from maximizing the likelihood functions (12)–(13). The Bayesian methods were the most computationally costly and slowest (Figure S9), with runtime around four seconds per step and more than two order-of-magnitudes slower than the implicitly evaluated MLE/MAP methods. This was mostly due to updating of the entire probability distribution (14), as the runtime without the entropy calculation in Bayesian-TS was only slightly reduced and still on the same order-of-magnitude. The Bayesian methods were also highly sensitive to rounding-error propagation and sampling of the parameter space, which was a disadvantage compared to MLE and MAP methods.

3.3. Stochastic root-finding methods

The parameter-free stochastic root-finding methods were based on the threshold definition only and therefore had the important advantage over the parametric estimation approaches, which depended on a potentially biased or inaccurate model. Their simple stepping rule was also much faster to compute, with runtime per step on the millisecond scale. Like the parametric methods, the root-finding methods also generated a threshold estimate and an error at every step (relative error $\delta x_{\text{th}}$ in Figures 4 and S13A, and absolute error $\Delta x_{\text{th}}$ in Figures S10 and S13B). The ratio between the IQR of $\delta x_{\text{th}}$ and the median $\left|\delta x_{\text{th}}\right|$ (Figures S11A and S14A) was also centered on 2 after a few steps, but showed larger variations compared to those of parametric estimation methods.

Figure 4: Relative threshold errors of stochastic root-finding methods with default initial step size $a_0$.

Similar to Figure 3.

The error distributions of all analog root-finding methods (Figure 4, lines with circle markers) became negatively biased after a certain number of steps, i.e., the median was less than zero, the absolute values of the first quartile and lower adjacent value were larger than the third quartile and upper adjacent value, respectively, and there were more negative than positive outliers (Figures 4A and S10A). The median error of ACS-H took about 15 to become negative and did not reach a steady state at step 200. The process for the adaptive methods was much quicker and their median errors reached within ten steps the same steady state (−0.34% and −0.23% MSO for relative and absolute errors, respectively), which the non-adaptive ACS-H should also converge to eventually. The bias originated from the asymmetry of the log-transformed MEP distribution at the threshold [61, Fig. 2]. The $50\mu \text{V}$ MEP threshold (which is the median of the MEP distribution according to the probability definition of stimulation threshold) was smaller than the average of the right-skewed distribution due to the nearby EMG noise floor (around $10\mu \text{V}$). Therefore, subthreshold responses generated a smaller $\Delta y$ to increase a subthreshold stimulation amplitude towards the threshold. In contrast, suprathreshold MEPs can be orders-of-magnitudes larger than the threshold, resulting in large $\Delta y$ that decreased the stimulation amplitude much lower than the threshold, especially during the early steps. The analog methods converged on the stimulation amplitude where the mean (log-transformed) MEP response equaled $50\mu \text{V}$, which typically occurred below the threshold (Figure 1A_i, statistics in Figure S15) The performance of analog methods could be improved using techniques for MEP detection below the EMG noise floor [63], [64] by recovering the full MEP distribution which should be more symmetric around $50\mu \text{V}$. Three of the analog methods (ACS-HA, ACS-GA, and ACS-GHA) with default parameters reached the accuracy level of the IFCN five-out-of-ten method with 1% MSO step size with only 20 steps or less (Figures 4B and Table 2), whereas ACS-H needed 24 steps.

The non-adaptive ACS-H (Figure 4, green lines) had a monotonically declining median $\left|\delta x_{\text{th}}\right|$ for the entire range of steps. The percentage of outliers (Figures S11B and S14B), however, increased with the steps, showing that the error did not improve for some subjects, which became a larger proportion of the total population as the errors of the other subjects in the population became more tightly distributed. The adaptive ACS-HA (Figure 4, orange lines) outperformed its non-adaptive counterpart by reaching a lower median $\left|\delta x_{\text{th}}\right|$ faster, but with distributions of $\delta x_{\text{th}}$ being more negatively biased and having larger proportion of outliers. The ACS-GA method (Figure 4, blue lines) with higher rate of convergence had a rapid exponential decay of the control sequence and did not further head for the threshold as soon as $a_i$ was too small for a notable influence. Its median $\left|\delta x_{\text{th}}\right|$ reached steady state at around 40 steps, albeit with a smaller portion of outliers as its distribution did not continue to tighten. In comparison, ACS-GHA (Figure 4, purple lines) switched from fast geometric convergence that provided the rough estimate to a. s. harmonic convergence and performed very similarly compared to ACS-HA in terms of median $\left|\delta x_{\text{th}}\right|$ and percentage of outliers.

Besides the convergence of the control sequence $a_i$, the root-finding methods’ performance was also determined by the initial value $a_0$ of the sequence. The default value of $a_0$ (10) was chosen to approximate the typical slope of the recruitment curve but its definition was inevitably empirical and potentially suboptimal. Examining the performance as a function of the initial step size indeed revealed that this was the case for some of the root-finding methods. The change from overshoot and oscillatory behavior for large $a_0$ towards lengthy one-sided convergence for smaller numbers were clearly visible (Figure S12, panels A_i–G_i). Some methods did not have one single optimal $a_0$, however, as the minimum of the median $\left|\delta x_{\text{th}}\right|$ at different steps (Figure S12, panels A_ii–G_ii, black dashed line and dots) may not be obtained with the same $a_0$. Instead of evaluating asymptotic behavior, we chose the optimal $a_0$ that minimized the error at step 20 for which the default was not optimal and the performance indeed improved (Table 2, results in parentheses, and Figure S13). The optimal $a_0$ for most methods was close to the default value, showing that using the slope of typical input–output curves (10) indeed provided an appropriate $a_0$. For ACS-H, the optimal $a_0$ also resulted in a negative bias of error distribution in just a few steps, similar to the other analog versions.

When the default $a_0$ was used, the digital versions of the root-finding methods (Figure 4, darker lines with triangle markers) could asymptotically perform either better than (DCS-H and DCS-HA) or very similar to (DCS-GA) their analog counterpart in terms of median $\left|\delta x_{\text{th}}\right|$ (Figure 4B). As the root-finding approached the target, the difference between the actual and target responses became very small $\left(\left|a_i\Delta y_i\right|\ll 1\right)$. The step towards the target $\left|\Delta x_i\right|$ was therefore smaller for the analog version $\left(\left|a_i\Delta y_i\right|\right)$ compared to the digital version $\left(\left|a_i\delta y_i\right|=a_i\right)$ and thus approached the target slower than the latter. It should be further noticed that all digital versions had mostly positive bias for $\delta x_{\text{th}}$ that decayed to zero, with the decay being especially quick for the two adaptive methods. The lack of asymptotic bias in contrast to the analog versions was due to the sign function of the digital versions equalizing the stepping towards threshold in both directions, regardless of the asymmetry of the MEP distribution. With optimal $a_0$, the digital versions had a slower decay of the positive bias of the error distribution compared to those for default $a_0$ (Figure S13A_i) but had improved convergence of $\left|\delta x_{\text{th}}\right|$ (Figure S13A_ii).

4. Discussion

We examined a variety of existing and new TMS thresholding methods and compared their performance in a virtual population of 25,000 subjects. The methods included eight based on relative frequencies using either a single threshold (IFCN relative-frequency methods) or two thresholds (Mills–Nithi methods) and either 1% MSO or 2% MSO step size, seven parametric estimation methods, of which four used different combinations of single- or two-parameter ML or MAP estimation and three were based on Bayesian estimation that updated the probability density function itself, and seven stochastic root-finding methods utilizing control sequences with different convergence and stepping rules.

Although there were some trade-offs between different methods, several threshold estimation methods can be considered to have overall better performance within their class or even across all methods (Figures 5 and S16). All relative-frequency methods had similar error performance and should be avoided due to the large number of pulses required and the presence of inherent error bias or large error outliers (see Figure S5). The two-parameter MAP estimation demonstrated the best performance among the point estimators considering the speed of convergence, median error, and computational complexity. The Kontsevich–Tyler Bayesian estimation performed even better regarding the first two aspects, but with increased computational cost, which might mostly matter if implemented on embedded microcontrollers. The best stochastic root-finding methods using analog control sequences was ACS-HA, whereas DCS-H had the strongest performance among both analog and digital versions. Unsurprisingly, the best-performing novel methods as well as the previous single-parameter MLE-1 all outperformed the reference IFCN five-out-of-ten method (median 64 steps and 1.6% median $\left|\delta x_{\text{th}}\right|$), requiring much fewer steps (within 20 steps, with the novel methods a few steps faster than MLE-1) to reach the same accuracy (Figure 5B, top). Compared to MLE-1, the best-performing novel methods also demonstrated better accuracy at 20 and 30 steps (Figures 5A and S16), which are typical numbers used to terminate the single-parameter MLE methods [41], [48], for example in ATH-tool [70].

Figure 5: Summary of best performing methods.

Selected statistics of the best performing methods, with full statistics shown in Figure S16. The best parametric methods are two-parameter MAP estimation (MAP-2) and Bayesian estimation, with the single-parameter MLE method (MLE-1) given for comparison. The best stochastic root-finding methods is DCS-H, with both default and optimal (for 20 steps) $a_0$ shown. The MAP-2 and Bayesian methods utilize prior information about the threshold distribution. A. Median $\left|\delta x_{\text{th}}\right|$ at steps 20 and 30. The dashed and dotted lines shows for reference the IFCN five-out-of-ten and ten-out-of-twenty methods with 1% MSO step size, respectively. B. The number of steps needed for a given method to outperform the reference IFCN five-out-of-ten method with 1% MSO step size (top) and the single-parameter MLE-1 at 20 steps (bottom).

Despite their wide range of behavior and difference in performance, all methods performed remarkably accurate in terms of threshold error on the population level. Either $\delta x_{\text{th }}$ and $\Delta x_{\text{th }}$ (or $\left|\delta x_{\text{th}}\right|$ and $\left.\left|\Delta x_{\text{th}}\right|\right)$ had mean or median on the level of 1% and 1% MSO, respectively. In comparison, the pulse amplitude variability of TMS machines themselves have exactness (deviation from nominal value) and precision (variation between pulses) in comparable ranges (1% to 3%) depending on the device [85]. The errors of individual subjects, however, had a large spread and may not fall within the ±5% criteria for safety [12], [42] for either relative-frequency methods (Figure S5) or more advanced estimation methods (Figures 3A, 4A, and S13A_i). For some methods, the estimation may not improve with increasing number of TMS pulses for certain subjects (e.g., ACS-H) or the entire population (e.g., ACS-GA) after a certain number of steps.

We analyzed both the errors and their absolute values, and provided detailed visualization and descriptive statistics of the entire distribution. In contrast, many previous studies quantified the absolute values of the relative error ($\left|\delta x_{\text{th}}\right|$, simply referring to it as the relative error) and/or presented mean and standard deviation only [48], [54]. The use of absolute values for errors provided the convenience to compare a positive metric that decreased with stepping and allowed the use of logarithmic scale for visualization. Whereas the median of the absolute values can capture the spread of the errors very well—our results showed that the ratio between the IQR of the errors and the median of the errors’ absolute value was quite consistently around two for the methods examined—the absolute value of errors, however, could not capture the bias of the estimated thresholds or its symmetry around zero. Similarly, the mean and standard deviation did not capture outliers and skewness of the distributions, which were all important characteristics of the thresholding methods. For example, the asymmetric and positively-biased error distribution of the IFCN relative-frequency methods, particularly the five-out-of-ten method that is widely used in clinical practice, may lead to significant variability of the stimulation amplitude and a tendency to larger amplitudes in treatment. This exacerbates the risk of exceeding safety limits due to the overestimation of thresholds using movement observation instead of EMG measurement [25], as many clinics do. Similarly, the original Mills–Nithi methods had a large number of outliers with negative error, contradicting assertions about its reliability [42].

Our implementation of the various methods all started from a suprathreshold simulation amplitude with a minimum 95% response probability. This, however, is often not the case in practice. A lower starting amplitude could be used to accelerate the threshold search for some methods, especially for subjects with shallow input–output curves, whereas a default stimulator output is sometimes used as starting point for all subjects. The stimulation amplitudes of stochastic root-finding methods are constrained by previous pulses, and different starting points change the error convergence, require a different optimal initial step size $a_0$, and favor analog methods and/or adaptive stepping over the non-adaptive DCS-H (results not shown). In contrast, the starting point does not affect the performance of parametric estimation methods on the population level, e.g., using a fixed 50% MSO amplitude (results not shown), as their stimulation amplitudes determined by the likelihood maximization are not constrained and can freely “jump” arbitrary distances from pulse to pulse. However, both of these situations risk convergence errors when a positive response is observed for a stimulator output well below the true threshold, especially early during the procedure [52], [53].

Existing estimation methods previously were tested using virtual subjects that had the same response model underlying the parameter estimation methods [48]. Such tests of estimators against their own estimation model can be misleading due to statistical bias. We therefore used an independently derived MEP model from the literature, which was previously trained to the statistics of experimental data. The response model used here, although more sophisticated than most other approaches, may still have limitations in representing all features and statistical properties of MEP responses due to the limited data on which it was based [32], [86]-[88]. In addition, complex temporal effects, such as inter-pulse correlation caused by stimulation-induced short-term plasticity and also stimulation-independent excitability fluctuations [45] may become important, particularly for short inter-pulse intervals of only a few seconds used in some labs. Accurate knowledge of the physiologic system, however, is essential for the thresholding problem. This does not only influence the design of a realistic testbed, but also directly determines, for example, how well the Gaussian model used in all parameter estimators suits the requirements or if another model, such as logistic distribution [40], [50], is needed in order to avoid bias.

The performance of some estimation methods demonstrated substantial improvement over existing methods. For example, the two-parameter MAP and Kontsevich–Tyler Bayesian estimation methods all outperformed the single-parameter MLE method. The improvement in accuracy for the Bayesian method, however, comes at a cost of increased runtime per step and larger memory requirement. While these costs may still be challenging for many microcontrollers depending on the specific application and potential implementation as an embedded version on a TMS device, they are readily manageable on personal computers which many recent TMS machines anyway include. The Bayesian method could become more appealing with improved implementation efficiency, for example by reducing the range and/or resolution of parameter space. All parametric estimators, however, had the potential weakness that the underlying model could be incorrect. Some of the parametric methods further relied on accurate prior information of the model parameters to achieve the presented accuracy and improvement over other methods. Such prior information on the population level could require significant effort to collect or may be completely unavailable.

The newly introduced family of non-parametric stochastic root-finding methods overall had similar or even lower errors compared to the parametric estimation methods. Comparing the best performing methods between the two classes, DCS-H was superior in terms of both speed and accuracy compared to the two-parameter MAP estimation and Kontsevich–Tyler Bayesian methods, which both required prior information. Furthermore, stochastic root-finding methods did not rely on potentially incorrect models and thus could work for subjects whose MEP response curves did not follow the Gaussian model. The rule for choosing and updating the step size $a_i$ was also simple enough to be performed by hand or a calculator and would not necessarily rely on installed software (e.g., [48], [70]). Hence, the root-finding methods are a promising addition to the available threshold methods that motivate further studies, improvements, and potential generalization to non-motor TMS thresholds where a response model may be unavailable. To enable the TMS community to use this method, we implemented DCS-H as an easy-to-use web-based tool SAMT (Stochastic Approximator of Motor Threshold [89]) using HTML/JavaScript, which is accessible via web browsers on desktop computers and mobile devices.

5. Conclusion

This work analyzed three classes and a total of 22 variations of TMS thresholding methods and compared their performance, including their accuracy and speed. As pointed out in previous critiques, the relative-frequency methods, such as five-out-of-ten, were far from optimal due to their large number of required stimuli, wide distribution of errors, and inherent error bias. Their accuracy was outperformed by parametric estimators and stochastic root-finding methods with a fraction of the stimuli. As the response model underlying parametric estimation methods did not always fit well with the virtual subject data, some parametric methods performed poorly for a small subset of subjects in the population. Including all model parameters in the estimation and incorporating prior information boosted performance. Stochastic root-finding methods were comparable or better in performance than parametric estimation methods, were much simpler to implement, and did not rely on an underlying model.

List of key abbreviations and mathematical symbols and notations

Abbreviations

TMS

Transcranial magnetic stimulation

MEP, EMG

Motor evoked potential, electromyography

MSO

Maximum stimulator output

IFCN

International Federation of Clinical Neurophysiology

PEST

Parameter estimation by sequential testing

ML, MLE

Maximum likelihood, maximum likelihood estimation

MAP

Maximum a-posterior (estimation)

ACS, DCS

Analog control sequence, digital control sequence (of stochastic root-finding)

a.s.

Almost sure (convergence)

Mathematical symbols and notations

$x,\mathbf{x}$

Stimulation amplitude, list of stimuli

$r,\mathbf{r}$

Response (binary), list of responses

$i$

Step number of parametric estimation and stochastic root-finding methods

$V_{\text{PP}},V_{\text{th}},V_i$

Voltage of MEP: peak-to-peak value, value for determining threshold, at step $i$

$t,s,\mathbf{\theta }$

Threshold, spread, and parameter set (of response probability model)

$t_{\text{MLE}},t_{\text{MAP}}$

Estimate of $t$ parameter using MLE and MAP

$a_0,a_i$

Step size of control sequence (stochastic root-finding methods): initial value, at step $i$

$y_i$

Logarithm (base 10) of MEP voltage $V_i$ (in units of Volts)

$\Delta y_i,\delta y_i$

Analog and digital control sequence

$x_{\text{th}},x_{\text{th},\text{GT}}$

Threshold amplitude: general, ground truth of model

$x_{\text{th},\text{est}},x_{\text{th},i}$

Threshold estimate: general, at step $i$ (parametric estimation and stochastic root-finding methods

$\Delta x_{\text{th}},\delta x_{\text{th}}$

Absolute and relative threshold error

Appendix

Response model parameters

The individual recruitment parameters $p_{1,j}$ to $p_{5,j}$ followed normal and lognormal distributions [61]:

$$p_{1,j}~𝒩\left(\mu =-0.5818,{\sigma }^2=3.5705\cdot {10}^{-3}\right),$$

$$p_{2,j}~𝒩\left(\mu =4.5323,{\sigma }^2=0.12619\right),$$

$$\text{lg}\left(\begin{array}{l}{p}_{3,j}\hfill \\ p_{4,j}\hfill \\ p_{5,j}\hfill \end{array}\right)~𝒩\left(\mu =\left[\begin{array}{c}3.6466\\ 0.42639\\ 1.6665\end{array}\right],\Sigma =\left[\begin{array}{ccc}7.2027& 0.56974& -0.19746\\ 0.56974& 4.7426\cdot {10}^{-2}& -1.5113\cdot {10}^{-2}\\ -0.19746& -1.5113\cdot {10}^{-2}& 7.1771\cdot {10}^{-2}\end{array}\right]\right).$$

The intra-individual variability sources ${ϵ}_{\text{xadd},ij}$ and ${ϵ}_{\text{ymult},ij}$ followed normal distributions with subject-specific standard deviation at each step :

$${ϵ}_{\text{xadd },ij}~𝒩\left(\mu =0,{\sigma }^2=p_{6,j}^2\right),$$

$${ϵ}_{\text{ymult },ij}~𝒩\left(\mu =-7,{\sigma }^2=p_{7,j}^2\right).$$

in which the standard deviations for subjects $j$ followed lognormal distributions:

$$\text{lg}\left(p_{6,j}\right)~𝒩\left(\mu =-0.96453,{\sigma }^2=2.2759\cdot {10}^{-2}\right),$$

$$\text{lg}\left(p_{7,j}\right)~𝒩\left(\mu =0.68827,{\sigma }^2=2.3671\cdot {10}^{-2}\right).$$

The additive noise ${ϵ}_{\text{yadd,}ij}$ followed a generalized extreme-value (GEV) distribution, with subject-specific parameters

$${ϵ}_{\text{yadd },ij}~\text{GEV}\left(\mu ={10}^{p_{1,j}},\sigma =1.55\cdot {10}^{-6}\cdot {1.5}^{p_{8,j}},k=0.14\cdot 2^{p_{9,j}}\right),$$

where $p_{8,j}$ and $p_{9,j}$ were independent normal distributions $𝒩\left(\mathrm{0,1}\right)$, and the GEV distribution had a probability density function of

$$f_{\text{GEV}}=\frac{1}{\sigma }{\left(1+\frac{k\left(x-\mu \right)}{\sigma }\right)}^{-\left(1+\frac{1}{k}\right)}\cdot \text{exp}\left(-{\left(1+\frac{k\left(x-\mu \right)}{\sigma }\right)}^{-\frac{1}{k}}\right)$$

The noiseless model had noise parameters ${ϵ}_{\text{xadd},ij}$ and ${ϵ}_{\text{ymult},ij}$ set to zero and ${ϵ}_{\text{yadd},ij}={10}^{p_{1,j}}$, which approached the mean of the GEV for the parameters $p_{1,j},p_{8,j}$, and $p_{9,j}$.

Distribution of threshold and spread

The threshold $t$ and spread $s$ (Figure 1B) was fitted by a joint distribution

$$\left(\begin{array}{c}t\\ \text{lg}\left(s\right)\end{array}\right)~𝒩\left(\mu =\left[\begin{array}{c}0.65847\\ -1.2989\end{array}\right],\Sigma =\left[\begin{array}{cc}{\sigma }_t^2& \rho {\sigma }_t{\sigma }_{\text{lg}\left(s\right)}\\ \rho {\sigma }_t{\sigma }_{\text{lg}\left(s\right)}& {\sigma }_{\text{lg}\left(s\right)}^2\end{array}\right]\right),$$

with ${\sigma }_t=8.9703\cdot {10}^{-2},{\sigma }_{\text{lg}\left(s\right)}=2.7576\cdot {10}^{-2}$, and the correlation coefficient $\rho =0.130$. This distribution served as $p\left(\mathit{\theta }\right)$ for the MAP methods and $p_0\left(\mathit{\theta }\right)$ for the Bayesian methods.