Evaluation and recommendations for effective data visualization for seizure forecasting algorithms

Abstract

Objective

Seizure forecasting algorithms have become increasingly accurate and may reduce the morbidity and mortality caused by seizure unpredictability. Translating these benefits into meaningful health outcomes for people with epilepsy requires effective data visualization of algorithm outputs. To date, no studies have investigated patient and physician perspectives on effective translation of algorithm outputs into data visualizations through health information technology.

Materials and Methods

We developed front-end data visualizations as part of a Seizure Forecast Visualization Toolkit. We surveyed 627 people living with epilepsy and caregivers, and 28 epilepsy healthcare providers. Respondents scored each visualization in terms of international standardized software quality criteria for functionality, appropriateness, and usability.

Results

People with epilepsy and caregivers ranked hourly radar charts highest for protecting against errors in interpreting forecasts, reducing anxiety from seizure unpredictability, and understanding seizure patterns. Accuracy in interpreting visuals, such as a risk gauge, was dependent on seizure frequency. Visuals showing hourly/daily forecasts were more useful for patients who experienced seizure cycling than those who did not. Hourly line graphs and monthly heat maps were rated highest among clinicians for ease of understanding, anticipated integration into clinical practice, and the likelihood of clinical usage. Epilepsy providers indicated that daily heat maps, daily line graphs, and hourly line graphs were most useful for interpreting seizure diary patterns, assessing therapy impact, and counseling on seizure safety.

Discussion

The choice of data visualization impacts the effective translation of seizure forecast algorithms into meaningful health outcomes.

Conclusion

This effort underlines the importance of incorporating standardized, quantitative methods for assessing the effectiveness of data visualization to translate seizure forecast algorithms into clinical practice.

1. INTRODUCTION

1.1 Background and significance

Epilepsy affects almost 1% of people worldwide,¹ and 30% of people living with epilepsy have seizures intractable to anti-seizure medications.² The paroxysmal, unpredictable nature of seizures is a major source of suffering, morbidity, and mortality in epilepsy.³

In the last decade, the field of seizure forecasting and prediction has accelerated to a forefront.⁴ Seizure forecasting algorithms continue to improve in accuracy, obtaining forecasting performances over the past years with area under receiving operating curves up to more than 90%.^5–7 With algorithmic advances comes the need for guiding principles in health information technology (HIT) to provide data visualization solutions to effectively communicate algorithm outputs to primary end-users, including people living with epilepsy (PWE) and healthcare providers. Although seizure forecasting algorithms hold the promise of enabling patients to anticipate and take preventative measures against seizures,^5–9 guidelines for the development of effective HIT solutions to translate seizure forecast algorithm outputs into meaningful health outcomes in epilepsy have not been studied.

Systematic evaluation of the usability of HIT solutions for communicating seizure forecasting algorithm outputs to end-users is an essential next step for translating algorithms into meaningful clinical change. International standards are available to aid in the design of patient- and healthcare provider-facing solutions. However, despite increasing interest in effective visualization of patient-reported outcomes,^10–12 and while international standards have been used to assess visualization solutions in conditions including diabetes¹³ and heart failure,¹⁴ standards have not yet been employed in epilepsy to understand how best to effectively communicate output of seizure forecasting algorithms to patients and healthcare providers.

1.2 Objective

Patient portals including electronic seizure diaries such as Seizure Tracker^TM and My Seizure Diary offer a unique venue for providing patient- and clinician-facing data visualization solutions to communicate seizure forecast algorithm outputs to end-users. In this study, we employed an evidence-based approach to evaluating data visualization solutions within a Seizure Visualization Toolkit in SeizureTracker.com for communicating forecast algorithm outputs to PWE and healthcare providers. To address gaps in testing HIT solutions for forecasting algorithms, we developed mockups of data visuals to communicate output of seizure forecast algorithms to PWE and healthcare providers, taking into consideration principles of graphical perception. We employed a standardized method assessing major software quality standards recommended by the International Organization for Standardization/Internal Electrotechnical Commission (ISO/IEC) 25010:2011 software quality standards. We discuss conclusions on designing effective data visualizations to communicate the output of forecasting algorithms to end-users.

2. METHODS

2.1 Forecast visualization toolkit development

Mockups of visualization solutions for communicating seizure forecast algorithm outputs to end-users were first developed as part of a Seizure Forecast Visualization Toolkit through an agile software development framework (Figure 1), incorporating principles of effective IT dashboard design.¹⁵ Each solution shows the same algorithm output (forecasted seizure risk) but varies in the usage of graphical encoding features, including length, area, color, color saturation, angle, slope, curvature, position along a common scale, and direction (Figure 2, Supplementary Table S1).^16,17

Figure 1.

Agile software development framework for developing front-end data visualization solutions. (A) Iterative cycle for development of data visualization solutions. (B) Process of defining inputs and data visualization outputs for incorporating seizure forecast algorithms into SeizureTracker.com electronic seizure diary.

Figure 2.

Data visualization solutions for seizure forecast outputs in Forecast Visualization Toolkit. Mockups of data visualization solutions were developed by Boston Children’s Hospital and SeizureTracker.com for visualization output of seizure forecast algorithms, including: (A) Daily heat map, (B) Monthly heat map, (C) Hourly line plot, (D) Hourly radar chart, (E) Rose plot, (F) Daily line plot, and (G) Daily risk gauge. Each solution was evaluated with respect to established software quality criteria.

2.2 Toolkit evaluation

2.2.1 Study population

The study received Institutional Review Board exemption (#P00032463) from Boston Children’s Hospital, Boston, Massachusetts, USA. Study participants included people with epilepsy, caregivers, and epilepsy providers from the SeizureTracker.com and American Epilepsy Society (AES) databases. Patients were drawn from the SeizureTracker.com database of 21 695 e-mail addresses, including 6538 e-mail opens by people with self-reported seizures using SeizureTracker.com. Inclusion criteria for patients/caregivers were (1) adults ≥18 years with (2) self-reported seizures/epilepsy or a caregiver of a person with seizures/epilepsy, (3) mobile or web access, and (4) able to understand the English language. A separate survey was distributed to epilepsy clinicians (physicians, nurses, and nurse practitioners) from the AES and SeizureTracker.com mailing lists. Inclusion criteria for clinicians were (1) adults ≥18 years who were (2) a physician, nurse, or nurse practitioner for patients with epilepsy, with (3) mobile or web access, and (4) able to understand the English language. Data were collected from April 30th, 2019 to June 6th, 2019, and the survey was sent once to each potential participant. Survey responses were anonymous and no identifying information was recorded. Informed consent was obtained from all participants (Table 1).

Table 1.

Demographic characteristics of study sample of people living with epilepsy and caregivers

	Percentage (%)
Relationship to patient
Person with epilepsy	41.5
Caregiver	56.5
Age of person affected by epilepsy
<10 years	18.2
11–19 years	17.7
20–35 years	27.1
>36 years	32.9
Declined to respond	4.1
Seizure type
Focal/partial (starts on one side)	39.2
Focal/partial with secondary generalization (starts on one side and goes to other side)	31.4
Typical or atypical absence (staring and unresponsiveness)	44.0
Generalized (starts on both sides)	36.0
Atonic (sudden head and/or body drop)	19.0
Myoclonic (quick jerks of arms and/or legs)	34.4
Status epilepticus (any seizure greater than five minutes)	20.7
Clusters (any seizures that occur back to back)	34.3
Average seizure frequency
More than once per day	18.3
Between daily and weekly	22.6
Between weekly and monthly	29.8
Less than one per month	13.6
Currently seizure free	12.1
Do you notice the seizures typically happening during a specific time of day?
Yes	58.9
No	37.0
I do not understand the question	0.5
Declined to respond	3.7
Do you ever change your behavior depending on feeling a seizure is going (or not going) to happen?
Yes	60.6
No	13.9
I do not think seizure risk varies	11.6
Declined to respond	13.9

2.2.2 ISO/IEC standard software metrics

Methodology for evaluating visuals was based on the Guideline for Good Evaluation Practice in Health Informatics (GEP-HI).^18,19 We investigated the following properties of each visual in the Forecast Visualization Toolkit:

• Ability to improve patient understanding of seizure patterns

• Ability to reduce patient anxiety from seizure unpredictability

• Ability to improve clinician efficiency

Corresponding research questions are shown in Figure 3A (gray boxes). To answer each research question, visuals were evaluated based on ISO/IEC 25010:2011 international software evaluation criteria (Figure 3A, white boxes; Figure 3B). Correspondence between the software evaluation criteria and our three primary research questions are shown in Figure 3A. Definitions of ISO/IEC software criteria are in Supplementary Table S2.

Figure 3.

(A) Evaluation criteria definitions. (B) Evaluation criteria classifications. Adapted from reference 20.

1. PWE/caregiver survey: Two primary research questions were asked of each visualization solution for PWE/caregivers (Figure 3A): (1) Do patients anticipate the solution will improve patient understanding of seizure patterns? and (2) Do patients anticipate the solution will reduce patient anxiety from seizure unpredictability?

 

To answer these research questions, we evaluated two primary ISO/IEC metrics for each visualization solution among PWE/caregivers: functional suitability and usability. Functional suitability was determined based on functional appropriateness, by asking PWE/caregivers whether the visualization solution would relieve anxiety from seizure unpredictability (Table 2B) and improve understanding of seizure patterns (Table 2C).

Table 2.

Survey questions administered in (top) patient/caregiver survey and (bottom) clinician survey

IOC/IEC software quality characteristic	IOC/IEC software quality sub-characteristic	Data visual	Survey question	Survey response options (correct value in italics)	Survey question scoring
Patient/Caregiver Community Survey
(a) Patient usability	Error protection	Daily heat map (Figure 2A)	Using the visual above, what day would you think has the highest risk of a seizure happening?	January 27   February 3   February 5	0 = incorrect   1 = correct
	Error protection	Rose plot (Figure 2B)	Using the visual above, what time of day would you think has the highest risk of a seizure?	4 am   8 am   4 pm	0 = incorrect   1 = correct
	Error protection	Monthly heat map (Figure 2C)	Using the visual above, what month would you think has the highest risk of a seizure happening?	May 2019   October 2019   February 2020	0 = incorrect   1 = correct
	Error protection	Hourly line plot (Figure 2D)	Using the visual above, what time of day would you think has the highest risk of a seizure?	7 am   11 am   8 pm	0 = incorrect   1 = correct
	Error protection	Hourly radar chart (Figure 2E)	Using the visual above, what time of day would you think has the highest risk of a seizure happening?	1 pm   3 pm   5 pm	0 = incorrect   1 = correct
	Error protection	Daily line chart (Figure 2F)	Using the visual above, what day would you think has the highest risk of a seizure happening?	February 7   February 15   March 1	0 = incorrect   1 = correct
	Error protection	Risk gauge (Figure 2G)	Using the visual above and thinking about when the highest risk of a seizure is occurring, please select the most appropriate answer.	I’m currently at the highest risk level of today   My risk level may increase throughout the day   My risk level is the lowest it will be today	0 = incorrect   1 = correct
(b) Patient functional suitability—reducing unpredictability	Functional appropriateness (for reducing unpredictability)	When thinking about the risk level of a seizure happening, does it seem like this visual would help relieve anxiety and provide a better way to prepare for seizures?		Seems helpful but does not apply to me   Seems very helpful   I do not understand this visual   Other (please specify)	0 = I do not understand this visual or other   1 = Seems very helpful, or seems helpful but does not apply to me
(c) Patient functional suitability—evaluating seizure patterns	Functional appropriateness (for evaluating seizure patterns)	Please rate the usefulness of this visual in the context of your seizure risk and predictability.		5-item Likert rating	Likert 1 (not useful)   Likert 2   Likert 3   Likert 4   Likert 5 (very useful)
IOC/IEC software quality Characteristic	IOC/IEC software quality Sub-Characteristic	Survey question		Response	Score
Clinician Community Survey
(d) Clinician usability	Appropriateness recognizability	I would use this visualization frequently in clinical care.		5-item Likert ranking	Likert 1 (strongly disagree)   Likert 2   Likert 3   Likert 4   Likert 5 (strongly agree)
	Operability	The visualization was easily understood.
	Learnability	I found understanding this visualization to have a steep learning curve.
(e) Clinician functional suitability	Functional appropriateness (for interpreting seizure diary patterns)	This visualization will help interpret seizure diary patterns.
	Functional appropriateness (for evaluating need for medication changes)	This visualization will help assess patients to identify needed therapy changes.
	Functional appropriateness (for guiding counseling on safety)	This visualization will help counsel patients on seizure safety until next clinic visit.
(f) Clinician performance efficiency	Time behavior (ease of integration into clinical workflow)	This visualization would integrate well into my clinical workflow in patient management.

 Usability was determined based on error protection, or the number of errors when the user was asked to complete a representative user task, consisting of asking the user to identify which date/time was associated with the highest risk of a seizure (Table 2A). An error was coded when the subject failed to correctly identify the date/time with highest projected seizure risk.

Definitions of user error protection and functional appropriateness are in Supplementary Table S2.

• Clinician survey: The primary research question when evaluating each visualization solution mockup for clinicians was (Figure 3A): Will the data visualization solution reduce clinician time spent in manual seizure diary review?

 To answer this research question, the following ISO/IEC measures were evaluated: (1) clinician usability (learnability, operability, appropriateness recognizability), (2) clinician functional suitability (functional appropriateness), and (3) clinician efficiency (time behavior). Learnability, operability, appropriateness recognizability, functional appropriateness, and time behavior are software subcharacteristics of usability, functional suitability, and efficiency, and are formally defined in Supplementary Table S2. Questions to evaluate learnability and operability were adapted from the System Usability Scale,²¹ a standardized instrument for assessing the usability of interactive systems (Table 2D). Clinician functional suitability and efficiency were used to quantify the anticipated improvement in streamlining clinical workflow. These aspects were evaluated using a five-item Likert scale ranking of four abilities: (1) Interpreting seizure diary patterns, (2) Evaluating need for medication changes, (3) Counseling patients on safety, and (4) Ease of integration into clinical workflow (Table 2E).

The questionnaire assessing each ISO/IEC software quality criterion is available in Table 2.

2.3 Statistical analysis

For binary software criteria, the omnibus test for a difference among visualizations was performed using the Cochran’s Q-test, with post-hoc pairwise testing through McNemar’s test. For categorical software criteria, the omnibus test for a difference among visualizations was performed using Friedman’s test, with post-hoc pairwise tests performed through the Wilcoxon rank-sum test. Family-wise error rate was controlled through the Bonferroni correction at the 0.05 level. Association of effectiveness rankings with baseline seizure frequency and seizure cycling was evaluated using Fisher’s exact test. P-values are reported after multiple testing correction.

3. RESULTS

3.1 Study sample demographics: PWE and caregivers

Data were collected from 627 PWE/caregivers (260 PWE, 354 caregivers, 13 status missing), corresponding to a 9.6% response rate. Demographic characteristics are in Table 2 and included an even distribution across age, seizure types, and seizure frequencies. The most common seizure frequency was weekly to monthly (29.8%). Most PWE/caregivers reported rhythmicity to clinical seizures (58.9%), and the majority (60.6%) reported they would change their behavior depending on when they thought a seizure was going to occur.

3.2 Effectiveness of data visuals: PWE and caregivers

Graphical perceptual advantages and disadvantages of each data visual are described in Supplementary Table S1. Adjusted P-values after multiple testing corrections are in Table 3. Overall, seizure forecasts visualized through the hourly radar chart (Figure 2D) had the highest scores among patients for error protection, anxiety reduction from seizure unpredictability, and understanding seizure patterns (Figure 4A). The risk gauge (Figure 2G) was the lowest-scoring visualization among patients (Figure 4A). Detailed findings are discussed below.

Table 3.

Significant differences between forecast visuals for patient-facing and clinician-facing software quality standards. Significant P-values are shown

		P-value
Patient or clinician metric	Software quality standard	Omnibus test	Posthoc test (adjusted P-value)
Patient	Error protection	<.001a
	Monthly heat map > Daily line plot		6.2e−4c
	Hourly radar chart > Daily line plot		1.8e−6c
	Hourly line plot > Daily line plot		9.2e−3c
	Daily heat map > Daily line plot		6.1e−5c
	Monthly heat map > Risk gauge		7.0e−14c
	Rose plot > Risk gauge		1.4e−10c
	Daily line plot > Risk gauge		5.5e−4c
	Hourly radar chart > Risk gauge		1.8e−18c
	Hourly line plot > Risk gauge		3.4e−12c
	Heat map by day > Risk gauge		6.3e−17c
Patient	Functional appropriateness for reducing seizure unpredictability	<.001a
	Hourly radar chart > Daily line plot		2.8e−2c
	Hourly radar chart > Risk gauge		4.8e−13c
	Monthly heat map > Rose plot		5.7e−3c
	Hourly radar chart > Rose plot		4.1e−7c
	Hourly line plot > Rose plot		7.3e−5c
	Daily heat map > Rose plot		9.9e−3c
	Daily line plot > Risk gauge		3.2e−6c
	Hourly line plot > Risk gauge		7.4e−10c
	Daily heat map > Risk gauge		1.9e−7c
	Monthly heat map > Risk gauge		2.9e−8c
Patient	Functional appropriateness for evaluating seizure patterns	<0.001b
	Hourly radar chart > Daily line plot		.02d
	Hourly radar chart > Daily heat map		<.001d
	Hourly radar chart > Rose plot		<.001d
	Hourly radar chart > Monthly heat map		.001d
	Hourly line plot > Daily heat map		.005d
	Hourly line plot > Rose plot		<.001d
	Hourly line plot > Monthly heat map		<.001d
	Daily line plot > Rose plot		.004d
	Daily line plot > Monthly heat map		<.001d
	Daily line plot > Risk gauge		<.001d
	Daily heat map > Risk gauge		<.001d
	Hourly line plot > Risk gauge		<.001d
	Rose plot > Risk gauge		.03d
	Hourly radar chart > Risk gauge		<.001d
Clinician	Appropriateness recognizability	.006b
	Hourly line plot > rose plot		.03d
Clinician	Operability	.001b
	Hourly line plot > Rose plot		.009d
	Monthly heat map > Rose plot		.03d
Clinician	Learnability	.08b	Not applicable
	Functional appropriateness for evaluating seizure patterns (Interpret)	Unable to assess due to sample size	Unable to assess due to sample size
Clinician	Functional appropriateness for identifying therapy changes (Assess)	Unable to assess due to sample size	Unable to assess due to sample size
Clinician	Functional appropriateness for guiding counseling (Counsel)	Unable to assess due to sample size	Unable to assess due to sample size
Clinician	Time behavior (Integrate)	.03b
	Hourly line plot > rose plot		.03d

^aCochran's Q test.

^bFriedman test^.

^cMcNemar’s test.

^dConover test.

Figure 4.

Comparison of Forecast Visualization Toolkit solutions for various ISO metrics: (A) PWE/caregiver usability and functional suitability and (B) Clinician functional suitability, time behavior, and usability.

3.2.1 Error protection in correctly interpreting algorithm outputs

Visualization solutions differed significantly in protecting PWE/caregivers against errors in interpreting algorithm outputs to correctly identify when they were at high risk for having a seizure (omnibus test, P < .001). The greatest number of errors occurred using the risk gauge (Figure 2G), with the second greatest number of errors using the daily line chart (Figure 2F). Significantly higher levels of error protection were attained by the hourly radar chart (Figure 2D), monthly/daily heat maps (Figure 2A,B), and hourly line chart (Figure 2C).

3.2.2 Functional appropriateness for reducing seizure unpredictability

The hourly radar chart was ranked most highly by PWE/caregivers for depicting forecasts in a way that would relieve anxiety from seizure unpredictability (Figure 4A). The lowest scored visualizations were the risk gauge (Figure 2G), followed by the rose plot (Figure 2E; Table 3).

3.2.3 Functional appropriateness for understanding seizure patterns

The hourly radar chart (Figure 2D) was ranked most highly by PWE/caregivers for depicting forecasts in a way that facilitated understanding of seizure patterns, followed by the hourly line chart (Figure 2C) and daily line chart (Figure 2F). The risk gauge was the lowest ranked for evaluating seizure patterns (Figure 2G; Table 3).

Mean patient software quality scores are in Figure 4A and Supplementary Table S3.

3.3 Does effectiveness of data visualization depend on seizure cycling characteristics?

Forecast visualization preferences were largely robust across seizure frequencies but varied based on presence or absence of seizure cycling.

3.3.1 Dependence of data visualization effectiveness on seizure frequency

Error protection for the risk gauge depended on the patient’s seizure frequency (P = .007) and was more likely to be accurately interpreted by patients with frequent seizures (79.6% of patients with more than daily seizures, 69.6% of patients with daily to weekly seizures, 64.7% of patients with weekly to monthly seizures, 54.7% of those with seizures less than once per month, and 64.9% of seizure-free patients). Functional appropriateness for relieving anxiety from unpredictable seizures and understanding seizure patterns for visual solutions did not significantly vary based on seizure frequency. As expected, PWE/caregivers with more than one seizure per day reported that visuals with hourly forecasts (eg, hourly radar chart and hourly line chart) provided greater anxiety relief and understanding of seizure patterns than monthly or daily forecasts, whereas patients with less frequent seizures had more similar valuations between visuals employing hourly, daily, and monthly forecasts (Figure 5).

Figure 5.

Error protection and functional appropriateness of Forecast Visualization Toolkit data visualization solutions among PWE/caregivers, stratified by seizure frequency subgroups. Mean scores are shown.

3.3.2 Dependence of data visualization effectiveness on seizure cycling

Three patients/caregivers were not familiar with the concept of seizure cycling. Among patients familiar with the concept of seizure cycling, error protection and appropriateness of forecast visualizations for relieving anxiety related to unexpected seizures for each visual solution were robust to whether or not the patient’s seizures cycled. The monthly heat map, rose plot, and daily risk gauge yielded similar ratings for their ability to communicate forecasts in a way that improved understanding of seizure patterns, both for patients whose seizures did or did not cycle. In contrast, the daily line plot (P = .0001), daily heat map (P = .02), hourly line plot (P = 02), and hourly radar chart (P = .0005) were less effective in improving understanding seizure patterns among patients whose seizures did not cycle, compared to their effectiveness in patients whose seizures did cycle. Those whose seizures tended to cycle were more likely to rate these visuals as useful for understanding seizure patterns, whereas those without seizure cycling were more likely to rate these visuals as not useful for understanding seizure patterns.

3.4 Demographics: Clinicians

Twenty-eight clinicians treating people with epilepsy provided input on visualization solutions. The sample was composed predominantly of physicians (n = 18, 64.3%), and eight nurses (28.6%) and two nurse practitioners (7.1%).

3.5 Effectiveness of data visuals: Clinicians

Clinicians rated hourly line charts (Figure 2C) as significantly more easily understood (P = .009; Table 3), better integrated into clinical practice (P = .03; Table 3), and more likely to be used in clinical care (P = .03; Table 3) than rose charts (Figure 2E). Clinicians also rated monthly heat maps (Figure 2B) as significantly more easily understood than rose charts (P = .03; Table 3). There was no significant difference in the learning curve between the visualization solutions (omnibus test, P = .08; Table 3). Although limited by sample size, the most highly ranked visualizations for tasks of interpreting seizure diary patterns, assessing patients to identify therapy changes, and counseling patients on seizure safety were the daily heat map (Figure 2A), daily line graph (Figure 2F), and hourly line graph (Figure 2C). In contrast, the rose plot (Figure 2E) scored poorly across all quality characteristics for clinicians. Mean values of software quality rankings by clinicians are in Figure 4B and Supplementary Table S3.

3.6 Platform development and data communication

Each visual solution for communicating seizure forecast algorithm outputs is available within SeizureTracker.com. Seizure Tracker is one of the world’s largest online and mobile electronic seizure diaries, with a registered user base of over 20 000 PWE, and contains the programming infrastructure to interact with upcoming seizure forecast algorithms. Patients interact with the SeizureTracker.com system by entering the date and time of each seizure. Back-end inputs to the SeizureTracker.com application programming interface (API) provide inputs into forecast algorithms. Each visual solution in the Toolkit is then used to visualize standard seizure forecast algorithm outputs, employing the consensus statement for common data elements in mobile health when applicable.²² Inclusion of data visualization solutions into SeizureTracker.com provides a health platform for open data communication that integrates external forecast algorithms with electronic seizure diaries (Figure 6A).

Figure 6.

(A) Schematic of open communication platform for interaction between electronic seizure diaries and forecast algorithms. (B) “External server” and “internal server” models for data communication between electronic seizure diaries and forecast algorithms.

Figure 6B shows the two models the SeizureTracker.com system makes available for open data communication between forecast algorithms and electronic seizure diary platforms. Both models are available within the SeizureTracker.com API. In the “external server” model, algorithms are stored on an external server. Data are exported from SeizureTracker.com to the external server using standardized data interaction models. The algorithm is run on the external server on exported data. The external server has limited access to the electronic seizure diary system with confirmation that server interaction does not store input/output data. Algorithm outputs are then imported back to the SeizureTracker.com API and the pre-established data visualization solutions are used to visualize algorithm outputs to the end-user. In contrast, the “integrated server” model stores algorithms internally on the SeizureTracker.com server. Both the “external” and “internal server” models are based on the same API of designed data inputs and outputs. Back-end forecast algorithm outputs required to produce each data visualization solution available in the SeizureTracker.com platform are shown in Table 4.

Table 4.

Back-end inputs to generate each data visualization in Forecast Toolkit available in SeizureTracker.com platform

Toolkit visual solution	Granularity of forecasted seizures	Measurement scale	Back-end data point(s) required from algorithm	Format of back-end data points
Daily line plot	Daily	Continuous	Dates of forecasted seizures	YYYY-MM-DD+UTC
			Numeric value of risk or seizure probability on forecasted dates (current and future dates)	(Numeric values from 0 to 100)
			Numeric value of error value of forecasted days (current and future dates)	(Numeric values from 0 to infinity)
			Seizure risk or probability on day of last seizure prior to current date	(Numeric values from 0 to 100)
			Mapping of percentage values to categorical seizure risk levels (if not provided, default mapping available)	(Mapping of numeric range to categorical)
Daily heat map	Daily	Categorical	Dates of forecasted seizures	YYYY-MM-DD+UTC
			Categorical values of risk or seizure probability on forecasted days (current and future time points)	Low, Low-Medium, Medium, High
			Mapping of categorical seizure risk values to continuous seizure probabilities (if not provided, default mapping available)	(Mapping of categorical to numeric range)
Hourly line chart	Hourly	Continuous	Hours of forecasted seizures	YYYY-MM-DD HH+UTC
			Numeric value of risk or seizure probability on preceding and forecasted hours in a 24 hour time window	(Numeric values from 0 to 100)
Rose plot	Hourly	Continuous	Hours of forecasted seizures	YYYY-MM-DD HH+UTC
			Numeric value of risk or seizure probability on preceding and forecasted hours in a 24 hour time window	(Numeric values from 0 to 100)
Monthly heat map	Monthly	Categorical	Months of forecasted seizures	YYYY-MM +UTC
			Categorical values of risk or seizure probability on forecasted months (past 3 months and future months)	Low, Low-Medium, Medium, High
			Mapping of categorical seizure risk values to continuous seizure probabilities (if not provided, default mapping available)	(Mapping of categorical to numeric range)
Hourly radar chart	Hourly	Categorical	Hours of forecasted seizures	YYYY-MM-DD HH+UTC
			Categorical values of risk or seizure probability on forecasted hours (current hour and future hours in a 10 h time window)	Low, Low-Medium, Medium, High
			Mapping of categorical seizure risk values to continuous seizure probabilities (if not provided, default mapping available)a	(Mapping of categorical to numeric range)
Risk gauge	Single day	Continuous	Current date	YYYY-MM-DD+UTC
			Numeric value of current risk or seizure probability	(Numeric values from 0 to 100)
			Numeric value of median/mean level of seizure risk today, from midnight to midnight	(Numeric values from 0 to 100)

^aDefault mapping in visuals in SeizureTracker.com platform displays four categories of risk unless specified by developer: 0–39% low risk; 40–59% low–medium risk; 60–79% medium risk; 80–100% high risk.

4. DISCUSSION

We demonstrate the application of systematic software evaluation standards for developing an effective data visualization system for seizure forecasting within electronic health platforms to meet end-user needs. ISO/IEC international software standards can be employed to assess usability, functional appropriateness, appropriateness recognizability, operability, learnability, and time behavior for translating seizure forecast algorithm outputs into improved healthcare outcomes. The effectiveness of translating a given seizure forecast into meaningful outcomes depends highly on the data visualization used. Furthermore, clinicians and patients differ in data visualization needs. Whereas radar charts work best for patients/caregivers, hourly line charts are generally preferred by clinicians. Visual solutions evaluated in this study are available through the open communication platform established by SeizureTracker.com. These results allow increased understanding of principles to guide front-end user interface development and back-end algorithmic end-goals.

4.1 Seizure risk visualization for people living with epilepsy

People living with epilepsy often live in fear of seizures. The continual lack of knowledge of when a seizure will occur is often equally or more psychologically impactful than the direct injuries caused by seizures themselves.^23,24 Providing patients and caregivers with seizure forecasts has the potential to reduce morbidity and mortality associated with unpredictable seizures, by enabling patients and caregivers to take preventative steps to avoid injury/death during anticipated seizures.^5–8 The effective translation of numerical algorithm outputs into improved healthcare outcomes will require testing of data visualization solutions, in order to prevent errors in interpretation or ineffective communication of the sensitivity and specificity of algorithms to patients. Understanding what patients living with epilepsy most value in algorithm outputs is necessary to translate algorithms into meaningful healthcare outcomes.

We find that radar charts are a good option for communicating seizure forecast outputs to patients with regards to helping patients correctly interpret algorithm forecasts, reducing patient anxiety from seizure unpredictability, and understanding seizure patterns. This visualization uses a color gradient to distinguish between discrete seizure risk levels, with a circular structure to encode time of day. The risk gauge, which shows forecast outputs using both discrete and continuously valued predictions, but only for a single point in time, had the worst performance for all three software criteria. In this visual, a color gradient is used to distinguish between discrete seizure risk levels, and angle is used to denote continuous values of seizure risk. A measure of the current day’s overall seizure risk is compared to the current value of risk, but notably, no comparison is made to previous or future days. The degree of error protection afforded by the daily risk gauge varied based on seizure frequency, with more accurate interpretation by patients with seizures occurring daily. The rose plot, which shows hourly forecast outputs using the angle to encode risk level and color to distinguish daytime versus evening, also was ranked poorly for relieving anxiety from unexpected seizures. These findings were consistent across frequencies, although people with high seizure frequencies (more than daily) tended to derive greater functionality from seizure risk predictions made on an hourly/daily basis, and found both discrete risk levels and point projections useful. This suggests that the granularity of the risk algorithms should be tailored to seizure frequency. The finding that the appropriateness of visualizations for relieving anxiety from unexpected seizures is robust across seizure frequency concurs with prior findings that seizure unpredictability affects the quality of life across all seizure frequencies.³ Patients whose seizures did not cycle were equally able to use visualizations to accurately identify when seizure risk was highest and reported similar anxiety relief but was less likely to find visualizations useful for improving understanding of seizure patterns.

These findings suggest the following guiding principles when integrating patient-facing seizure forecast algorithms into health information technology: (1) color gradients can be used to effectively encode seizure risk levels, (2) using a circular structure to encode time of day is effective, (3) patients value forecasts with relation to surrounding timepoints, not only for a single point in time, (4) using the angle to encode seizure risk level may reduce effectiveness for reducing seizure anxiety, and (5) individual seizure frequency and cycling tendency should be taken into account when developing patient-facing healthcare platforms to forecast seizure risk.

4.2 Seizure risk visualization for clinicians

Next, we evaluated the usability, functionality, and efficiency of Toolkit visualizations for algorithm outputs for clinicians. Ensuring electronic health record (EHR) usability is important for reducing cognitive workload and improving efficiency among clinicians.^25–28 Clinicians reported that hourly line graphs were most easily understood, well-integrated into clinical practice, and likely to be frequently used in clinical care. There was no significant difference in the learning curve among forecast visualizations. The preference for line graphs among clinicians is contrasted to end-user feedback from patients, for whom the radar chart was most highly ranked. Whereas the radar chart shows discrete seizure risk forecasts using a color gradient encoding along a circular time axis, the line graph shows continuous seizure risk forecasts using a position encoding along a linear time axis, and additionally uses slope to encode changes in forecasts from hour to hour. Clinicians may be more accustomed to linear time axes and working with continuous laboratory values in clinical practice, as these are a customary visual display in the healthcare setting. The clinician reported that the hourly line graph, daily line graph, and daily heat map were likely to help them interpret seizure diary patterns, identify needed therapy changes, and counsel patients on seizure safety. All visualizations except the rose plot were rated as having high functional appropriateness for interpreting seizure diaries; monthly, daily, or hourly granularities provided good functionality (score 3 or higher) for assessing the impact of therapies or interpreting diaries, and daily and hourly granularities had good functionality (score 3 or higher) for counseling patient safety. Clinician functionality of monthly forecasts for assessing therapy impact and interpreting seizure diaries may be due to the fact that anti-seizure therapies are often changed during outpatient encounters, which may be months apart. The poor performance of the rose plot for clinicians is congruent with its poor performance among patients and may reflect a higher cognitive burden for using angle compared to position along a common scale to encode seizure risk levels.¹⁶

Taken together, these findings suggest that although data visualizations of seizure risk are useful to clinicians, information technology must focus on providing solutions that minimize the additional burden on the clinician. If solutions can be structured into the clinical workflow and electronic health record in a manner that minimizes the additional workload for clinicians, these visualizations may reduce error rates and improve care.

4.3 Seizure risk visualization for data scientists and platform developers

Lastly, we provide several useful perspectives to data scientists and platform developers involved in seizure forecast algorithm development. We show that clinicians and patients differ in data visualization needs, suggesting that front-end developers should focus on designing systems that provide options for different patient-facing and clinician-facing data visualization systems for forecast algorithms to accommodate different needs. We recommend that open communication platforms provide patients and clinicians with the most highly chosen three options within software interfaces (radar charts, line plots, and heat maps) with the ability to choose among the three. The poor performance of the rose plot concurs with research showing that human visual perception does not support accurate decoding of area or angles along curves^16,29 and suggests that angle and area be avoided in visualizing forecast algorithm outputs. Algorithms that target patients with severe epilepsy (eg, daily seizures) should provide seizure forecasts at least hourly. For algorithms that target patients with less frequent seizures, hourly, daily, or monthly granularity of forecasts can be useful. For algorithms that target clinicians, forecast granularity depends on the intended use. For seizure safety counseling, daily or hourly granularity is needed; for assessing the effect of treatment or aiding diary interpretation, monthly, daily, or hourly granularity is needed.

5. CONCLUSION

We provide an example process implementation for front-end incorporation of seizure forecast and risk assessment algorithm outputs into electronic health platforms. The value of the guidelines provided here is that they are developed with patient and clinician needs in mind. We identify which data visualization outputs are anticipated to most highly impact patient quality of life and clinician workflow using end-user feedback and provide a map for software developers in order to determine the outputs that algorithms should provide in order to meet end-user needs. These considerations are important in seizure forecast efforts, which, depending on the algorithm, can potentially output any number of values, including continuous, hourly, daily, or monthly seizure forecasts, and can provide anything from categorical risk values to continuous risk percentages.

There are several limitations to this study. (1) As survey responses are drawn from the SeizureTracker.com and AES databases and participation was not mandatory, patient/caregiver and clinician feedback may be subject to response or selection bias. Our study is also restricted to English. (2) Patient-reported seizure frequencies and time of day may be subject to recall bias, and forecast algorithms must communicate this uncertainty in their displays. Real-time forecasts will also rely on the usability of real-time data uploads by patients, requiring the development of devices that remove the burden of manual data uploads. (3) People living with epilepsy and caregivers are the main stakeholders that seizure forecasting algorithms are intended to benefit, and our study focuses on the visualization preferences in this group. Low clinician response rates may contribute to selection bias and limited power in the clinician sample, requiring follow-up studies with targeted larger samples of clinician recruitment.

As the size and complexity of medical data grow, the need for health information technology platforms to share and process data effectively and efficiently is similarly increasing.^20,30 Effective visualization of seizure forecasting within electronic seizure diaries is needed to translate algorithm advances into meaningful health outcomes.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Conflict of interest statement

R.M. is the co-founder/owner of Seizure Tracker, LLC, which has received funding from Cyberonics, Courtagen, Engage Therapeutics, Greenwich Biosciences Neurelis, Union Chimique Belge (UCB), Brain Sentinel, and grants from Tuberous Sclerosis Alliance. T.L. serves on the Council of the American Clinical Neurophysiology Society, on the American Board of Clinical Neurophysiology, as founder and consortium PI of the pediatric Status Epilepticus Research Group (pSERG), as an Associate Editor for Wylie’s Treatment of Epilepsy 6th and 7th editions, and as a member of the New Onset Refractory Status Epilepticus (NORSE) Institute, PACS1 Foundation, and Critical Care EEG Monitoring Research Consortium (CCEMRC). T.L. served as Associate Editor of Seizure, and served on the Laboratory Accreditation Board for Long Term (Epilepsy and Intensive Care Unit) Monitoring. He is part of patent applications to detect and predict clinical outcomes, and to manage, diagnose, and treat neurological conditions, epilepsy, and seizures. T.L. is co-inventor of the TriVox Health technology and may receive financial compensation from this technology in the future. T.L. has received research support from the Epilepsy Research Fund, National Institutes of Health (NIH), the Epilepsy Foundation of America, the Epilepsy Therapy Project, and the Pediatric Epilepsy Research Foundation, and received research grants from Lundbeck, Eisai, Upsher-Smith, Mallinckrodt, Sunovion, Sage, Empatica, and Pfizer, including past device donations from various companies, including Empatica, SmartWatch, and Neuro-electrics. T.L. previously has served as a consultant for Zogenix, Upsher Smith, Eisai, Amzell, Engage, Elsevier, UCB, Grand Rounds, Advance Medical, and Sunovion. T.L. has received speaker honorariums from national societies including the American Academy of Neurology (AAN), American Epilepsy Society (AES) and American Clinical Neurophysiology Society (ACNS).

Contributors

Sharon Chiang: Formal analysis, investigation, data interpretation, writing-original draft preparation. Robert Moss: Conceptualization, resources, data curation, investigation, writing-review & editing, supervision. Angela P. Black: Investigation, writing-review & editing. Michele Jackson: Investigation, writing-review & editing. Chuck Moss: Investigation, writing-review & editing. Jonathan Bidwell: Investigation, writing-review & editing. Christian Meisel: Investigation, writing-review & editing. Tobias Loddenkemper: Conceptualization, resources, investigation, writing-review & editing, supervision.

DATA AVAILABILITY STATEMENT

The data underlying this article will be shared on reasonable request from Robert Moss (info@seizuretracker.com).