A TRIAL OF PROFICIENCY OF NERVE CONDUCTION: GREATER STANDARDIZATION STILL NEEDED

Abstract

Introduction

The aim of this study was to test the proficiency (accuracy among evaluators) of measured attributes of nerve conduction (NC).

Methods

Expert clinical neurophysiologists, without instruction or consensus development, from 4 different medical centers, independently assessed 8 attributes of NC in 24 patients with diabetes mellitus (DM) on consecutive days.

Results

No significant intraobserver differences between days 1 and 2 were found, but significant interobserver differences were seen. Use of standard reference values did not correct for these observed differences.

Conclusions

Interobserver variability was attributed to differences in performance of NC. It was of sufficient magnitude that it is of concern for the conduct of therapeutic trials. To deal with interrater variability in therapeutic trials, the same electromyographers should perform all NC assessments of individual patients or, preferably, NC procedures should be more standardized. A further trial is needed to test whether such standardization would eliminate interobserver variability.

Assessment of signs, symptoms, functional impairments, and nerve tests are the the main clinical measures used for diagnosing and characterizing peripheral nerve disease. Among tests, nerve conduction (NC) has increasingly been advocated for objective electrodiagnosis and characterization of focal, multifocal, or generalized polyneuropathies.^1–4 NC is also being used for conducting epidemiological surveys^5–7 and therapeutic trials.^8,9 For the latter purposes, NC has gained consensus approval,^10–13 because its results are considered sensitive, objective, quantitative, and reproducible indications of nerve dysfunction that correlate with clinical signs and symptoms and with neurophysiological and neuropathological abnormalities.^14,15

In knowing that assessment of attributes of NC can provide sensitive and quantitative diagnostic and characterizing information about diabetic sensorimotor polyneuropathy (DSPN) for use in medical practice and therapeutic trials, it would also be helpful to learn how accurately and reproducibly within and among clinical neurophysiologists this assessment is done (proficiency). There is already considerable information about test–retest reproducibility of individual attributes of NC focusing mainly on which attributes show the greatest reproducibility and, with this criterion, are suitable for therapeutic trials (albeit test–retest reproducibility is only 1 criterion for such selection). Reproducibility studies have not addressed the issue of proficiency—that is, the variability of NC results among different clinical neurophysiologists without retraining, consensus development, or quality control of their evaluations.

Two previous studies have assessed intra- and interobserver variability of measured attributes of NC. In the first study, electromyographers from 1 medical center studied healthy subjects,¹⁶ and in a second they studied patients with diabetes mellitus (DM).¹⁷ They found no significant intraobserver differences, but significant interobserver differences. They concluded that, for therapeutic trials, it may be advisable to have the same electromyographers perform the sequential NCs over time on patients in the trial.

The present trial, Clinical vs. Neurophysiology Trial 3 (Cl vs. NPhys Trial 3) had each group of clinical neurophysiologists and their associate technologist perform NC assessment of 8 attributes of NC of the leg of 24 masked patients with DM without and with DSPN on consecutive days. The expert clinical neurophysiologists were asked to make independent measurements of the attributes without any clinical information, record whether their values were normal or abnormal, and finally make a judgment of whether the abnormalities were diagnostic of DSPN. Thus, these studies specifically addressed 3 questions: (1) Did expert neurophysiologists obtain the same measured values of the 8 attributes of NC without intra- or interobserver differences? (2) Assuming that they did, did they judge abnormality of attributes without intraor interobserver differences? (3) Did they judge DSPN as present or not without intra- or interobserver differences? It was recognized prior to the study that if the first question could not be answered affirmatively, it may not be possible to adequately assess questions (2) and (3). This trial also addresses the broader question of whether further standardization of NC assessment is needed and possible, or whether it is really necessary to have the same electromyographer perform serial evaluations of individual patients.

This trial is part of a series of studies organized by Cl vs. NPhys Trial investigators to assess the proficiency of assessment of signs, symptoms, and clinical diagnosis,^18,19 attributes of NC (the present Trial 3) and other nerve tests for the diagnosis and characterization of polyneuropathies, especially DSPN. The overall goal is to improve the quality of these evaluations of measures of polyneuropathy in medical practice, medical education, and the conduct of therapeutic trials.

METHODS

Study Setting and General Description

The trial was performed in a small ballroom of the Kahler Grand Hotel in Rochester, Minnesota, on December 8–9, 2011, after the study protocol and the recruiting and consenting process had been approved by the institutional review board of the Mayo Clinic.

The original recruitment of subjects for Cl vs. NPhys Trial 1 had focused on obtaining an even number of patients without and with DSPN by NC criteria. For the present trial, an attempt was made to recruit the same group of patients, but 3 were unable to participate, necessitating substitution of other patients whose clinical and NC status (normal/abnormal) was not known until after completion of the study. Clinical neurophysiologists were given information on age, gender, height, and weight of the patients and were told that they had DM, some without and others with DSPN. Otherwise, their identity, disease status, and signs and symptoms were withheld, and supervisory attendants ensured complete blinding of patients’ identity and polyneuropathy status. Furthermore, evaluators were told that all limbs to be examined had been pre-warmed. Patients were asked not to provide disease information, and examiners were asked not to request such information. An honorarium was paid to the research subjects to offset their time away from other activities.

To carry out the examinations, we enlisted 4 expert clinical neurophysiologists and their associate technologist from Queen’s University, Kingston, Ontario, Canada (C.B. and N.W.); the University of Maryland, Baltimore, Maryland (J.R. and K.T.); the University of Michigan, Ann Arbor, Michigan (J.A. and J.W.); and the Mayo Clinic, Rochester, Minnesota (C.K. and A.Z.). The clinical neurophysiologist and technologist both participated in nerve conduction assessment. There was no training or consensus development before the trial. The evaluators independently performed NC tests using different electromyographic (EMG) instruments, test peripherals, and their own procedures. They were asked not to confer with each other about test procedures or the normal values to be used. However, these clinical neurophysiologists were all asked to evaluate the same 8 attributes of NC of the left leg of the same masked 24 patients on 2 consecutive days. The 4 groups of evaluators were stationed in 4 curtained cubicles at the periphery of the hotel ballroom so that lower limbs from patients lying on examining beds could be introduced for their NC assessment without examiners seeing the patients’ upper bodies (Fig. 1). This physical arrangement also allowed study personnel to readily rotate (every 15–20 minutes) patients among the different clinical neurophysiologists, and it prevented recognition of patients by their physical appearance. Moreover, random assignment of patients to test cubicles was used to prevent recognition of patients by order of their evaluation. Each testing cubicle was equipped with separate (and, in some cases, different) manufacturers’ EMG instruments, along with individual stimulation and recording electrodes, tape to hold electrodes in place, grease pencils to mark test sites, and measuring tapes. Study coordinators had warmed the left leg of patients by immersion in hot water prior to the onset of testing; heat lamps were used to maintain limb temperature; and, between tests, limbs were kept warm (31°–34°C) by use of applied insulating leg casts. After completion of NC testing and before transfer of the patients to a different cubicle, all indicators of electrode placement or sites of stimulation were removed by supervisory personnel.

FIGURE 1.

The physical arrangement used to assess attributes of nerve conduction in the Cl vs. NPhys Trial 3, allowing masked examination of 24 patients with diabetes on 2 separate days. Curtained examination cubicles for NC assessment were at the periphery of the ballroom, which permitted lower limbs of the patients to be introduced for examination without the examiners being able to identify patients by their physical appearance. This setup allowed examiners to explain what was being done to each patient. It also allowed patients to report on excessive symptoms or reactions to the tests being given while not revealing their identity or disease condition.

After performing the NC assessments, the clinical neurophysiologists calculated NC values and completed forms indicating whether measured individual attributes were normal or abnormal and then made a judgment of whether patients had electrodiagnostic evidence of DSPN. Examiners provided measurements of fibular nerve amplitude [compound muscle action potential (CMAP)]; motor nerve conduction velocity (MNCV), and motor nerve distal latency (MNDL); tibial CMAP, MNCV, and MNDL; and sural sensory nerve action amplitude (SNAP) and sensory nerve distal latency (SNDL). Immediately after the study of each patient, the data forms and the machine tracings were collected by study coordinators.

Analysis

Standard descriptive statistical tests were used to assess intra- and interrater agreement of measured attributes of NC, judgment of attribute abnormality, and diagnosis of DSPN. The analyses and statistical tests used are described from point of use in the Results section. To assess whether use of standard reference values could improve proficiency, we also converted measured NC attribute values to percentiles using standard reference values obtained in a previously studied healthy subject cohort of 330 subjects (from an initially evaluated group of 430 Olmsted County, Minnesota, USA residents), without metabolic or neurological disease and who were normal by neurological examination and laboratory tests, such as fasting plasma glucose, plasma creatinine, and percent plasma hemoglobin A_1c, and the Rochester Diabetic Neuropathy Study healthy subject cohort (RDNS-HS).^20,21 These percentile reference values of measured attributes of NC were corrected as applicable for the influence of age, gender, height, weight, body mass index, and body surface area. Abnormality of an attribute of NC was set at ≤2.5th or ≥97.5th percentile. For a comparative standard and referenced criterion for DSPN, we used Σ5 NC nds (standard normal deviates of fibular CMAP, MNCV, and MNDL; tibial MNDL; and sural SNAP).^22,23 To derive this composite score, the average value of measurable attributes was multiplied by 5 (with all attributes placed into the lower tail of the normal distribution). The 2.5th percentile line of this composite score was set using the RDNS-HS data just described.

RESULTS

Raw Values and Intra- and Interrater Agreement of Measured Attributes of NC

The variability of raw values of attributes of NC as measured among the evaluators is shown in Table S1 (see online Supplementary Material). The median and range values overlapped in all cases, but differences of some magnitude occurred in some instances. Using coefficients of variation, the differences of measured values on days 1 and 2, respectively, were as follows: fibular CMAP, 90.2% and 92.9%; fibular MNCV, 16.2% and 14.2%; fibular MNDL, 18.4% and 20.2%; tibial CMAP, 99.3% and 97.5%; tibial MNCV, 14.1% and 14.0%; tibial MNDL, 20.2% and 23.2%; sural SNAP, 92.4% and 90.6%; and sural SNDL, 11.2% and 12.0%. Clearly, motor NC velocities and distal latency were much less variable than were motor or sensory amplitudes (findings from earlier studies).

No statistically significant intraobserver differences were found for the 4 clinical neurophysiologists’ assessments of measurement of attributes of NC between days 1 and 2 (Table S1). By contrast, significant interobserver differences were observed for 8 of 8 attributes on day 1 and for 7 of 8 attributes on day 2 (shown as bold entries in Table 1); the only exception on day 2 was fibular MNCV.

Table 1.

Significant interobserver differences (bold) of measured attributes of nerve conduction in Cl vs. NPhys Trial 3.^*

	Friedman χ2 test for differences among 4 Cl vs. NPhys teams
	Day 1		Day 2
Nerve conduction attribute	χ2	P	χ2	P
Fibular CMAP (mV)	22.06	<0.01	33.10	<0.01
Fibular MNCV (m/s)	9.97	0.02	1.17	0.76
Fibular MNDL (ms)	28.43	<0.01	36.55	<0.01
Tibial CMAP (mV)	40.55	<0.01	32.03	<0.01
Tibial MNCV (m/s)	21.70	<0.01	12.95	<0.01
Tibial MNDL (ms)	20.15	<0.01	39.14	<0.01
Sural SNAP (µV)	15.00	<0.01	15.20	<0.01
Sural SNDL (ms)	15.16	<0.01	24.45	<0.01

A full table showing the raw values of clinical neurophysiologists’ measurements and intraobserver agreement of measured attributes of nerve conduction is shown in Table S1 (see online Supplementary Material).

^*No significant intraobserver differences were observed between days 1 and 2 for any group of electromyographers.

Intra- and Interobserver Agreement of Judgment of Abnormality of Attributes of NC

Using the kappa-coefficient, highly significant intraobserver agreement between days 1 and 2 was observed for the 4 clinical neurophysiologists’ judgments of abnormality of all attributes of NC, with the exception of the group 3 assessment of fibular MNDL (Table S2).

By contrast, statistically significant interobserver differences in judgment of abnormality of attributes of NC were observed 6 of 16 times (shown as bold entries in Table 2); that is, tibial CMAP and sural SNAP, each 2 times, and tibial MNCV and sural SNDL, 1 time. However, in addition, nearly significant differences (P ~ 0.1) were observed for 3 other attributes (Table 2).

Table 2.

Significant interobserver differences (bold) of clinical neurophysiologist judgment of abnormality of attributes of nerve conduction in the Cl vs. NPhys Trial 3.^*

	Cochran’s Q-test for differences among 4 Cl vs. NPhys Trial 3 teams
	Day 1		Day 2
Nerve conduction attribute	Q	P	Q	P
Fibular CMAP abnormal (doctor’s judgment)	6.60	0.09	3.00	0.39
Fibular MNCV abnormal (doctor’s judgment)	6.14	0.10	4.00	0.26
Fibular MNDL abnormal (doctor’s judgment)	3.00	0.39	6.00	0.11
Tibial CMAP abnormal (doctor’s judgment)	13.86	<0.01	10.50	0.01
Tibial MNCV abnormal (doctor’s judgment)	9.58	0.02	3.38	0.34
Tibial MNDL abnormal (doctor’s judgment)	6.00	0.11	5.53	0.14
Sural SNAP abnormal (doctor’s judgment)	11.90	<0.01	19.91	<0.01
Sural SNDL abnormal (doctor’s judgment)	6.43	0.09	15.89	<0.01

Intraobserver clinical neurophysiologist agreement of judgment of abnormality of attributes of nerve conduction is shown in Table S2 (online Supplementary Material).

^*Significant intraobserver differences were not observed between days 1 and 2 for any group of electromyographers, with 1 exception (i.e., electromyographer no. 3 for fibular MNDL).

Intraobserver Agreement of Diagnosis Based on “Evaluators’ Judgment” vs. “Standard Reference Values” (i.e., Σ5 NC nds ≤2.5th percentile) from RDNS-HS

As shown in Table S3, we tested for difference of clinical physiologists’ “judgment of abnormality” as compared with “referenced abnormality”; that is, ≤2.5^th and ≥97.5th percentile, based on RDNS-HS data. Regardless of which attribute was evaluated, a significant difference was not observed between days 1 and 2. By contrast, strikingly significant differences were found between “judged” and “referenced” abnormality (significant values shown as shaded boxes in last 2 columns). For fibular CMAP, “judged” was more frequently abnormal than was “referenced” abnormality, whereas, for fibular and tibial MNCV and sural SNAP and SNDL, the converse was found—“referenced” was more frequently abnormal than was “judged.” Other attributes were much more inconsistent, such as tibial CMAP.

Intraobserver Agreement of Clinical Neurophysiologists’ “Judged” vs. “Referenced” Electrodiagnosis of DSPN

Whereas no intraobserver difference was observed between days 1 and 2, significant differences between “judged” and “referenced” diagnosis were observed for 7 of 8 comparisons (shaded area in Table S4).

DISCUSSION

Accurate assessment of the diagnosis and severity of polyneuropathies such as DSPN is needed for quality medical practice, epidemiological surveys, and the conduct of therapeutic trials. Proficiency (accuracy among evaluators) of such assessments has not been emphasized sufficiently in the past, perhaps due to the assumption that clinical evaluations are expertly and proficiently done, especially among professionals who have received specialty training and certification. However, when actually tested, clinical proficiency was not as good as expected. Prototypical studies of intra- and interexaminer reliability (reproducibility) of measured attributes of NC were performed by Chaudhry et al. in healthy subjects¹⁶ and in patients with diabetic neuropathy.¹⁷ In their first study, 7 experienced electromyographers assessed NCs of 4 other members of their group on 2 occasions. In their second study, 6 experienced electromyographers performed duplicate NC studies on 6 patients with polyneuropathy. In both studies, intraexaminer agreement was high, but significant interobserver differences were found, indicating differences in performance of NC assessments. Chaudhry and colleagues concluded that, “…if NC studies are to be used longitudinally (i.e., for therapeutic trials) they should optimally be performed by a single examiner to minimize the degree of variability associated with different examiners.” Additional reports focused on test–retest reproducibility of and among evaluations of different attributes of NC,^24–29 but these studies differ from Chaudhry et al. and our study by not comparing NC as is usually done in medical practice. These latter studies often provided specific directions on how to perform NC and offered a degree of quality control of the NC examination. Also, in most of the latter studies, differences among clinical neurophysiologists were not assessed. Instead, the focus was on which attribute showed the lowest test–retest reliability.^{4,23,24,26,29}

From a review of medical publications, there is ample evidence that the methods used to test and stage severity of DSPN are not as proficient as intended.³⁰ Lack of clinical proficiency of the assessment of signs and symptoms of polyneuropathy was demonstrated unequivocally in the Cl vs. NPhys Trial 1. In that study, the investigators markedly overreported signs and, to a lesser degree, DSPN symptoms and diagnosis, presumably emphasizing sensitivity over specificity.¹⁸ When they became aware of their overreporting, and the trial was repeated using “only unequivocal abnormality” (taking age, gender, physical fitness, and physical variables into account) as a criterion for abnormality, their clinical proficiency improved dramatically.¹⁹ The trial investigators suggested that, for medical practice, epidemiological surveys, and therapeutic trials, the more specific criteria should be used.

The present trial (Cl vs. NPhys Trial 3) has focused on proficiency of NC assessment of 4 independent expert clinical neurophysiologists and associated technologists drawn from different medical centers in North America. It extends the Johns Hopkins studies by: (1) study of a larger number of patients, specifically 24 patients; (2) involving clinical neurophysiologists and their technologists from 4 widely distributed North American medical schools; (3) employing different EMG instruments and stimulating and recording peripherals; and (4) assessing judgment of abnormality of individual attributes and of the diagnosis of DSPN.

We found that, although intraobserver variability was not significantly different, interobserver variability was significantly different for many NC attributes and for the judgments of abnormality.

We judge interrater variability to be excessive, especially for the conduct of therapeutic trials. Using the raw values (median and range), one observes that measured attributes of NC were similar and overlapping among the 4 clinical neurophysiologists, suggesting that these measured values are sufficiently accurate for use in the identification and characterization of DSPN in medical practice. However, when variability was expressed as coefficient of variation, quite large differences among the NC attributes were observed, and this variability was greater for some attributes (motor and sensory amplitude) than for others (conduction velocity and distal latency). Because amplitude measurements probably relate more closely to clinical deficit, their excessive interobserver variability is of concern, especially for the conduct of therapeutic trials. This significant interobserver variability extended to judgment of attribute abnormality and to a lesser degree to electrodiagnosis of DSPN—that is, it was not statistically significant for day 1, but it was significant (P=0.05) for day 2.

What is the likely explanation for the interobserver variability of measured attributes of NC? By nature of the design of these studies, the differences cannot be explained readily by physiological alterations of nerves, because only inter- and not intraobserver differences were observed, whereas measurements underlying intra- and interobserver differences were performed concurrently. Also, the interobserver difference cannot be due to the superior or inferior performances of individual clinical neurophysiologists, because results of all groups were overlapping. Therefore, we conclude, as did Chaudhry et al.,^20,21 that the variation must relate to differences in test performance; that is, testing is still not standardized sufficiently.

Is the interobserver variability we found clinically meaningful such that improvement is needed? The answer probably depends on what use is to be made of the results. Some degree of variability may be acceptable for some purposes; however, for therapeutic trials it is not acceptable to have the degree of interobserver variability as reported here. The observed differences are sufficiently large to be of concern, especially for therapeutic trials in which small differences between treatments (treatment and sham) can markedly affect the power of the trial.

How can proficiency be improved? As suggested by Chaudhry et al.,^20,21 the same clinical neurophysiologists could be asked to do all serial evaluations of individual patients. A second approach would be to standardize and optimize the performance of NC so that interobserver differences would be eliminated. For conduct of therapeutic trials, preliminary training sessions and quality control of examinations are probably needed. We believe that, with sufficient standardization and training, interobserver variability can be eliminated. To test this possibility a further trial has been planned. Finally, it is evident that use of high-quality reference values, although meritorious, cannot take the place of accurate and highly standardized NC measurements.

Abbreviations

CMAP

compound action potential

CL vs. NPhys Trial 3

Clinical vs. Neurophysiology Trial 3

DM

diabetes mellitus

DSPN

diabetic sensorimotor polyneuropathy

MNCV

motor nerve conduction velocity

MNDL

motor nerve distal latency

NC

nerve conduction examination

RDNS-HS

Rochester Diabetic Neuropathy Study, healthy subjects

SNAP

sensory nerve action potential

SNDL

sensory nerve distal latency

Σ5 NC nds ≤2.5th

summated 5 attributes of nerve conduction at ≤2.5th percentile (expressed as standard normal deviation)

APPENDIX

Additional Cl vs. NPhys Trial 3 investigators: P. James B. Dyck, MD, Phillip A. Low, MD, and Carol J. Overland, MD (all from Department of Neurology, Mayo Clinic, Rochester, MN; additional members of the coordinating committee); Henning Andersen, MD (Aarhus University Hospital, Aarhus, Denmark); John D. England, MD (Department of Neurology, Louisiana State University, New Orleans, LA); Gareth Llewelyn, MD (University Hospital of Wales, Cardiff, Wales, UK); Michelle L. Mauermann, MD (Department of Neurology, Mayo Clinic, Rochester, MN); Dinesh Selvarajah, MD (Royal Hallamshire Hospital, Sheffield, UK); Wolfgang Singer, MD (Department of Neurology, Mayo Clinic, Rochester, MN); A. Gordon Smith, MD (University of Utah, Salt Lake City, UT); and Solomon Tesfaye, MD, and Adrian Vella, MD (Division of Endocrinology, Diabetes, Metabolism and Nutrition, Mayo Clinic, Rochester, MN; study neurologists and diabetologists).