Causal Inference, Statistical Assumptions, and Natural-Language-Processing Artefacts in Evaluating Medical students’ Use of Generative Artificial Intelligence [Response to Letter]

Dear editor

We thank Feng for the careful reading of our article¹ and for raising several methodological considerations concerning the interpretation of cross-sectional survey data and the analysis of free-text responses.² We welcome the opportunity to clarify the scope of our study and several aspects of its statistical and natural-language-processing (NLP) methodology. We agree with a number of the points raised, several of which were explicitly acknowledged as limitations in our original article. We also believe that some points require clarification regarding the intended interpretation and methodology of the study.

Interpretation of the Cohort Difference

We agree that the observed difference in reported encounters with incorrect AI-generated information between MD2030 and MD2031 cannot establish that prior AI-literacy exposure caused improved critical appraisal. Our objective was exploratory and was to compare students’ reported experiences across cohorts that differed, among other characteristics, in prior exposure to AI-literacy activities. The design does not permit the observed difference to be attributed specifically to that exposure.

Indeed, we explicitly stated in the Limitations section that “the cross-sectional design precludes conclusions about causality” and that it was not possible to determine whether the observed cohort differences were directly attributable to prior AI-related coursework. We also acknowledged that interpretation of survey items concerning incorrect AI-generated information might vary according to prior exposure.

We therefore agree with Feng that several explanations for the lower reporting of incorrect AI-generated information in MD2031 are possible. These include differences in recognition or interpretation of errors, patterns of AI use, prompting behaviour, cohort characteristics, and changes in the underlying AI systems between academic years. The finding should consequently be regarded as an observed cohort difference whose origin cannot be established from the present study. This does not alter the descriptive finding itself, but it appropriately limits its interpretation.

Statistical Considerations

We appreciate the observation concerning the small GEMD2030 cohort and the expected cell count in the chi-square analysis. An exact analysis is a reasonable alternative where expected frequencies are small, and we agree that reporting an appropriate effect-size measure in addition to the observed proportions and significance test would provide further information concerning the magnitude of the cohort difference.

We also agree that the analyses were exploratory and that the reported p-values should therefore be interpreted in the context of multiple testing. The tests addressed distinct prespecified survey questions rather than a single confirmatory hypothesis family; nevertheless, explicit adjustment or reporting of multiplicity-sensitive results would strengthen future analyses.

We agree that the distinction regarding the role of the GEMD2030 cohort could have been stated more clearly. GEMD2030 was included in the quantitative survey analyses, including the comparison of reported encounters with incorrect AI-generated information. It was excluded from inferential NLP comparisons because of the small number of informative free-text responses. The NLP inferential comparisons were therefore restricted to the two MD cohorts.

One factual point requires clarification. The Letter states that the response rate was not reported. Response rates were explicitly presented in Table 1 of the published article: 23% for MD2030, 21% for MD2031, and 26% for GEMD2030. We nevertheless recognize that these relatively modest response rates represent a potential source of non-response bias and should be considered when assessing the generalizability of the findings.

Measurement Considerations

We agree that the survey instrument was not subjected to formal psychometric validation, as was explicitly stated in the Methods. The questionnaire was developed from themes identified in the literature and underwent expert review by three faculty members for content validity, clarity, and relevance.

We would, however, clarify the suggestion that reliability coefficients such as Cronbach’s alpha should have been reported for the usefulness and trust measures. These variables were assessed using individual five-point Likert items rather than multi-item psychometric scales. Internal-consistency coefficients such as Cronbach’s alpha are therefore not applicable to these individual items. We nevertheless agree that future studies employing validated multi-item instruments could provide more robust measurement of these constructs.

We also agree that prior exposure to an activity explicitly involving AI-generated errors may have influenced how MD2031 students interpreted the question concerning encounters with incorrect information. This possibility was explicitly recognized in our Limitations section and further supports treating the observed result as a cohort difference rather than attributing it specifically to the educational exposure.

Sentiment Analysis

Feng appropriately observes that Q17 and Q24 differed substantially in their framing: Q17 invited students to describe experiences with erroneous or misleading AI-generated information, whereas Q24 invited suggestions for improving AI tools. Consequently, greater negative sentiment in Q17 cannot be interpreted independently of this difference in question content.

The Wilcoxon test establishes a difference in the distributions of sentiment scores elicited by the two prompts, but we agree that this difference has limited independent substantive interpretation because question framing provides an obvious explanation for it. The comparison is therefore best regarded as a descriptive characterization of the language elicited by the two questions rather than evidence of a psychological transition from criticism to constructive engagement.

Importantly, within each question, sentiment did not differ significantly between MD2030 and MD2031. These within-question comparisons are not subject to the same cross-prompt framing issue.

NLP Topic Modelling

We agree that the relatively small corpus of short free-text responses limits the stability and generalizability of the topic solution. The analysis was intended as exploratory rather than as validation of a latent thematic structure. Candidate solutions containing two to eight topics were compared using reconstruction error, topic diversity, and average topic assignment, and the resulting topics were inspected against the underlying responses before being consolidated into broader themes.

We also explicitly acknowledged that the topic analysis lacked external semantic validation and formal inter-rater reliability assessment, and we recommended independent human coding in future work. We agree with Feng that such validation would strengthen subsequent studies.

Feng also correctly draws attention to the term NEG_correct appearing in the presentation of the topic-modelling results. On reviewing this point, we identified this as a typographical error in the table: the term should read correct_NEG, consistent with the negation-tagging convention described in the Methods. The preprocessing intentionally applied negation tagging—for example, converting “not accurate” to accurate_NEG—to preserve the semantic context of negated terms during topic modelling. Thus, the presence of a negation-tagged term in the feature matrix is expected and does not represent unintended preprocessing leakage. We thank Feng for drawing attention to the incorrect presentation of this term.

Finally, we agree that VADER was originally developed and validated in the context of social-media text and that its use with reflective educational survey responses represents an application to a different type of text. In addition, the sentiment classifications in the present study were not independently validated against human coding. We therefore regard the VADER results as exploratory quantitative descriptors of linguistic polarity rather than validated measures of students’ underlying attitudes. Future studies could usefully combine automated sentiment analysis with independent human annotation.

Conclusions

We appreciate Feng’s emphasis on the value of longitudinal, pre-post, and controlled studies using objective measures of AI error detection. We agree with this recommendation. Our article similarly concluded that the cross-sectional design could not establish causality and recommended longitudinal and controlled studies incorporating objective learning outcomes.

The principal contribution of our study should therefore be understood within its intended scope: it provides a cross-sectional description of how first-year medical students use and perceive generative AI, together with exploratory comparisons of reported experiences and free-text responses. The observations regarding cohorts with differing prior AI-literacy exposure are hypothesis-generating and should not be interpreted as demonstrating an educational intervention effect.

We thank Feng for providing an opportunity to clarify these methodological and interpretative points. These methodological considerations do not alter the principal descriptive observations of the study: generative AI was widely used by the students surveyed, was perceived as highly useful but only moderately trusted, and was generally regarded as a complement to rather than a replacement for conventional learning approaches.

Disclosure

The authors report no conflicts of interest in this communication.