Table 1.

Learning Analytic Categories

Analytic Category	Brief Description	Techniquesa	Requirements	Worked or Potential Example in Medical Education
Descriptive analyses of quantitative data	Analyses that characterize past performance or learning process data.	Summary statistics, visual analytics, cluster analysis, social network analysis, heat maps. Currently, this is mainly what medical educators are using to describe their trainees.	Data that can be combined due to their similarity in scaling (i.e., both scores are out of 7) and conceptually (i.e., both items could be aggregated because they are both measuring communication skills).	Time series graphs of average achievement scores (e.g., average in‐training exam scores). Radar Plots39 (graphical representation of multiple data points overlying a grid that marks out various competencies).
Descriptive analyses of qualitative data (i.e., automated text analyses)	Computational analyses of digital resources and/or products created during learning to improve the understanding of learning activities. The automated processing of naturally generated human language.	Computational linguistic analyses of text coherence, syntactic complexity, lexical diversity, semantic similarity. Collaborative filtering techniques, content‐based techniques.	Large database of qualitative comments with or without human supervision to allow for primarily text‐based processing.	Natural Language Processing (where artificial intelligence is applied to human language) Automated content analysis (where a MLA “learns” to reliably replicate human coding across large volumes of data). e.g., Automated flagging of concerning qualitative comments by faculty based on learned models of textual analysis.b
Explanatory modeling	The use of data to identify causal relationships between constructs and outcomes within a data set to refine cognitive models.	Difficulty factors assessment, learning factors analysis, linear or logistic regression, decision trees, automated cognitive model discovery.	Variables that map to defined constructs. Data representative of the target population.	Learning trajectories (regression based).6, 11 Learning curves.9 Cumulative sum scores plots.40
Predictive modeling	The use of historical data to generate models that predict unknown future events (e.g., academic success, timeline to program completion) based on timely observations.	Linear and logistic regression, nearest neighbors classifiers, decision trees, naive Bayes classifiers, bayesian networks, support vector machines, neural networks, ensemble methods.	Quantifiable subject characteristics, clear outcome of interest, a large data set, and an ability to intervene. Training and test data for which you know the outcome (e.g., passing boards).	Development of a predictive model for difficulty in residency (e.g., remediation, probation, or failure to complete) for current residents using data from their residency applications.38 Future unsupervised MLAs may be able to spot patterns in performance without human supervision.b
Evaluative analyses	The use of aggregated assessment data can also be flipped on its head and become an analysis of a program's functioning.	All of the above may be used, the difference is the flip in the item of interest from the trainee toward evaluating the residency program, the faculty, or the system at large.	The ability to link trainee data to the rest of the healthcare or educational system. e.g., If you are evaluating rater gender bias, then your data must be linked to rater data (e.g., the identities of who rated whom when).	A six‐program evaluation study of point‐of‐care daily assessments of milestone data was recently conducted.41, 42 This data has opened up a larger conversation about gender bias in EM trainee assessments.

^aThese techniques are not exclusively used for each type of analysis, but are those which are considered the typical uses. Most techniques can be modified for various uses.

^bStarred items represent items which are newer learning analytic techniques on the horizon.