Causal Inference, Statistical Assumptions, and Natural-Language-Processing Artefacts in Evaluating Medical Students’ Use of Generative Artificial Intelligence [Letter]

Dear editor

I read with interest the cross-sectional survey by Loizou et al¹ in Advances in Medical Education and Practice, which examined first-year medical students’ awareness, use, and perceptions of generative artificial intelligence (AI) tools, and compared reported encounters with incorrect AI-generated content between cohorts with and without prior AI-literacy exposure. The topic is timely and educationally consequential,^2,3 and the integration of quantitative items with free-text natural-language processing (NLP) is commendable. However, from the standpoint of educational research methodology, I believe the central inference is underdetermined and several analyses contain correctable flaws that warrant clarification by the authors.

The Headline Inference is Directionally Ambiguous and Confounded

The study’s highlighted finding is that the MD2031 cohort, which had completed a structured activity requiring identification and correction of intentionally embedded errors in AI-generated text, reported fewer encounters with incorrect AI-generated information than the MD2030 cohort (52.2% vs 84.1%; chi-square = 10.73, p = 0.0047). The authors suggest this may reflect AI-literacy exposure improving critical evaluation, while noting that it may also reflect altered thresholds for what counts as an AI error. These interpretations are not equivalent. If such training genuinely strengthens critical appraisal, one would expect heightened, not reduced, detection of errors in students’ own AI use; conversely, lower reporting is equally consistent with weaker detection, more careful prompting that avoids errors, or a measurement artefact whereby trained students exclude the exercise’s errors from their notion of “encountered” errors.^4,5 The data cannot adjudicate among these. Moreover, the contrast is a non-randomised comparison of two cohorts surveyed in different academic years (2024–25 vs 2025–26), differing in age, gender, prior degree, and prior AI use, and embedded in a period during which the underlying models and their error profiles changed substantially.⁶ Therefore, an “association” between training and reporting differences cannot be isolated. I suggest the authors reframe the finding as a cohort difference of uncertain origin rather than an indication, even tentatively, of a training effect.

Statistical Assumptions and Reporting

Three statistical issues merit attention. First, the principal chi-square test includes the GEMD2030 cohort (n = 12); the expected count in the smallest cell is approximately 4.0, below the conventional threshold of 5 for the validity of the chi-square approximation, for which Fisher’s exact test is preferable. The analytic role of this small cohort also appears inconsistent: it is included in the omnibus test yet described as restricted to descriptive analysis elsewhere. Second, only p-values are reported; effect sizes (for example, rank-biserial correlations or odds ratios) are absent, so the practical magnitude of the cohort difference cannot be judged. Third, multiple inferential tests (two analyses of variance, several chi-square tests, a Wilcoxon signed-rank test, and a Spearman correlation) appear without correction for multiplicity, inflating the family-wise Type I error rate. I also note that no response rate is reported, leaving the representativeness of the 102 respondents unclear.⁷

Construct Validity of the Measures

The usefulness and trust constructs are reported as single-number Likert means without reliability estimates (for example, Cronbach alpha), making it difficult to judge whether the “high usefulness, moderate trust” gap reflects a genuine attitudinal structure or measurement noise.⁴ More substantively, the item on having “encountered incorrect AI-generated information” is unlikely to be measurement-equivalent across cohorts: MD2031 students had just completed an exercise centred on AI errors, so their interpretation of the same item plausibly differs from that of MD2030 - a form of differential item functioning by exposure rather than evidence of a skill difference.⁵ In addition, the comparison of sentiment between Q17 (which asks about AI errors) and Q24 (which asks for improvement suggestions) is confounded by question content: Q17 responses are necessarily more negative because the prompt solicits problems, not because students psychologically “shift” from evaluating limitations to constructive engagement. The Wilcoxon p < 0.001 therefore reflects prompt framing rather than a meaningful within-student transition.

Natural-Language-Processing Artefacts

The free-text analysis raises concrete validity concerns. Non-negative matrix factorisation topic modelling on roughly 100 short survey responses operates well below the corpus size required for stable topic solutions, and no inter-rater reliability or external semantic validation was performed - limitations the authors themselves acknowledge. A visible artefact reinforces this concern: the sentiment or negation label “NEG_correct” appears among the highest-weight terms of a Q17 topic, indicating that preprocessing tags leaked into the term matrix and undermining the interpretability of that topic. Separately, VADER was developed and validated for social-media text;⁸ its applicability to formal, reflective survey responses is unestablished, and the sentiment scores were not validated against human coding.

Implications and Suggestions

From an educational standpoint, the question that matters - whether structured AI-literacy instruction causally improves students’ critical appraisal of AI outputs - cannot be answered by these data. I encourage the authors to: (a) reframe the cohort difference as unexplained pending controlled study; (b) report effect sizes, multiplicity corrections, and the survey response rate; (c) establish measurement invariance for items compared across exposure groups, or revise the item wording; (d) replace prompt-confounded cross-question sentiment contrasts with within-question cohort comparisons; and (e) validate the NLP pipeline with independent human coding and a documented preprocessing workflow. A pre-post or longitudinal design using objective error-detection tasks as the outcome would more directly test the educational hypothesis.

Conclusion

Loizou et al offer a timely description of how students in their first year of medical school engage with generative AI, and their combination of survey data with free text NLP is a valuable methodological step. Nevertheless, the study’s central educational inference is not yet warranted. The reported cohort difference is confounded by academic year, student characteristics, prior AI use, and evolving model error profiles. It is also directionally ambiguous, because lower reporting of incorrect AI generated content may indicate stronger critical appraisal, weaker detection, more careful prompting, or a changed interpretation of the survey item. Statistical concerns, including an unstable chi square approximation, missing effect sizes, uncorrected multiplicity, and an unreported response rate, further limit the strength of the conclusion. The NLP analysis also requires caution given the small corpus, possible preprocessing leakage, and unvalidated sentiment measurement. I therefore interpret the highlighted finding as an unexplained cohort difference rather than evidence that AI literacy instruction improves critical evaluation. To meet the standards of rigorous educational research, future work should use pre/post or longitudinal designs, objective error detection tasks, measurement invariance testing, and NLP pipelines validated against human coding. Such evidence would clarify whether structured AI literacy instruction genuinely strengthens medical students’ critical appraisal of generative AI and under what conditions it does so.

Disclosure

The author reports no conflicts of interest in this communication.