Evaluating psychiatric conference posters: Benchmarking a custom generative pre-trained transformer against human inter-rater variability using a structured assessment framework

ABSTRACT

Background:

Scientific poster assessment lacks standardized and discipline-neutral rubrics. Assessment by human reviewers (HRs) is subject to inter-rater variability.

Aim:

To assess PA²IRS (Poster Assessment via AI-Integrated Rubric System) framework for AI-assisted psychiatric poster evaluation, and conducted a reliability study comparing AI and HR agreement.

Methods:

PA²IRS was developed through an AI-assisted iterative criterion refinement process modelled on Delphi principles. Sixty posters (30-case reports/series [CR], 20 original research [OR], 10-systematic review-Meta-analysis [SRMA]) were randomly sampled. Three qualified mental health professionals served as independent reviewers. A custom-GPT (GPT-5.2, GO-subscription) provided AI assessments across three domains: Domain-A (content quality, poster-type specific), Domain-B (visual), and Domain-C (impact). PA²IRS is a 100-point instrument combining an AI-assessable poster component and an in-person interview component. This study concerns only the poster component. Intraclass correlation coefficients (ICCs), Passing–Bablok regression, Bland–Altman analysis, and variance component analysis were performed using appropriate statistical tools.

Results:

AI–human single-measure ICCs [Overall (0.62), Domain-A (0.63), Domain-B (0.44), Domain-C (0.55)] met or exceeded human-human ICCs (0.42, 0.40, 0.27, 0.48) across all domains. Four-rater ICC (with AI) reached 0.75. Variance ratios (AI–human vs inter-human spread) were ≤1.0 across all domains for all posters combined. The SRMA subgroup showed variance ratios of 0.09–0.13 for Domains A and B (Bartlett P ≤ 0.002). Overall score bias was 0.11 percentage points (pp); Domain-A showed a consistent maximum positive bias of 5.5 pp across subgroups.

Conclusion:

AI–human agreement was within or exceeded the inter-human reliability range across three domains. The domain-dependent agreement pattern is consistent with dual-process cognitive theory. PA²IRS supports use as a scalable, standardized, and cross-disciplinarily competent formative biomedical poster assessment tool.

INTRODUCTION

Scientific poster presentation is a widely used means of academic communication, enabling researchers to disseminate findings, and demonstrate competencies. Traditionally, posters have been assessed using analytical rubrics evaluating discrete criteria such as design, content, and verbal presentation.[1] However, these methods are often time-consuming, subject to inter-rater variability, and influenced by subjective biases such as first-impression effects, design preferences, and inconsistent application of scoring criteria.[2,3,4]

Current poster evaluation literature highlights persistent challenges: lack of standardization across disciplines, evaluator fatigue and visual impression influencing scoring.[2,4,5] Even structured human evaluation remains resource-intensive and constrained by limited expert availability.[6] Visual impression also significantly influences scoring, with layout, logical sequencing, and conciseness presenting persistent challenges for consistent human assessment.[1,3,6] The evolving visual and digital landscape also demands assessment tools that adapt to diverse visual structures while maintaining objectivity and reliability.[6] The essence is to preserve core science communication over the visual distractions.

Two structural problems affect the validity of poster assessment, and Artificial Intelligence (AI) is uniquely positioned to address them. First, no individual reviewer possesses deep knowledge across all sub-specialties and methodologies. Second, human evaluation naturally engages a holistic visual impression before criterion-level analysis. AI, integrated with a structured framework, can automate preliminary evaluations, standardize feedback, and maintain scoring consistency.

Emerging generative AI models, particularly GPT (Generative Pre-trained Transformer) and multimodal large language models (LLMs), offer promising avenues. Such advances signal the potential for AI not only to describe scientific content but also to evaluate it against structured, predetermined criteria in a consistent and automated manner.[7,8,9]

To address the gap, we propose PA²IRS (Poster Assessment via AI-Integrated Rubric System), designed to systematically evaluate biomedical posters across domains. This study aimed to evaluate AI–human agreement using PA²IRS and determine whether AI performance falls within inter-rater variability. This study applies a dual-process cognitive framework to interpret domain-dependent agreement patterns and quantifies variance components to benchmark AI disagreement against natural inter-human variability across three poster types from Royal College of Psychiatrists International Congress 2025, a major international psychiatry congress.

MATERIALS AND METHODS

Development of poster assessment via AI-integrated rubric system framework

PA²IRS was developed from published rubrics and literature [Supplemental File 1]. It is a criterion-based assessment rubric rather than a psychometric instrument. Criteria were systematically collated from previously published poster assessment rubrics and the empirical literature, then synthesized and deduplicated into three assessment domains. Given the multidisciplinary scope of the rubric, encompassing case reports (CR), original research (OR), and systematic reviews-meta-analyses (SRMA), [Table S1 in Supplemental File 2] assembling a single human expert panel with equivalent expertise across all content domains was considered challenging. Therefore, an AI-assisted iterative criterion refinement process modelled on Delphi principles was undertaken to improve criterion clarity, consistency, and completeness. This exploratory process supported rubric development and does not intend to replace formal multidisciplinary human expert content validation that remains a future validation step.

Table-S1.

Operational Definitions Used in This Study

Original research	Generating new data through systematic investigation, with defined methodology and analysis)
Case report/case series	Descriptive studies detailing one or multiple patients with similar clinical features, without a comparison group or inferential analysis
Systematic review–meta-analysis (SRMA)	Studies that systematically identify, appraise, and synthesise existing literature using predefined criteria, with meta-analyses additionally performing quantitative pooling of results. Narrative review excluded

It is discipline-neutral, spanning domains A (content quality, type-specific scoring), B (visual presentation, 30 points), and C (impact/innovation, 10 points). Domain-A scores are type-specific (CR: 40 points; OR: 45 points; SRMA: 50 points) and were converted to percentages for cross-type comparability; Domains B and C were analyzed as raw scores, and overall scores were expressed as percentages.

All analyses reflect the poster component that constitutes 80–90% of the full 100-point instrument, depending on poster type. The interview component (verbal defense) was not assessed here. The initial development and refinement of PA²IRS (rubrics) criteria was by using three architecturally distinct LLMs across three rounds [Supplemental File 1].

Study setting and poster sample

Posters from the Royal College of Psychiatrists International Congress 2025 were accessed via PosterSessionOnline (https://www.postersessiononline.eu/) in December 2025. Of 875 submissions, 758 were retrievable. A total of 875 poster submissions were recorded; 758 were listed and retrievable, comprising 199 research presentations (OR, SRMA), 79 case studies and the rest excluded. Operational definitions and pre-specified criteria for AI assessment suitability and governance requirements were applied [Table S1, S2]. The study aimed for 60 posters: 30 CR, 20 OR, and 10 SRMA. Using a 1:4 sampling ratio, 240 poster IDs were randomly selected (randomizer.org); posters were screened sequentially until category targets were met, with a second sampling round to complete the 30-CR target. A formal sample size calculation was not performed; the target of 60 posters was determined based on feasibility and representational balance across poster types.

Table-S2.

Pre-specified criteria for AI assessment suitability and governance requirements

Domain	Unsuitability threshold	Observed value	Threshold met?	Governance implication
Overall score — AI–human ICC	Below lower bound of human-human ICC	0.62 vs human lower bound 0.42	No — threshold not met	Suitable for deployment
Any domain — variance ratio	Exceeds 1.5 at full-sample level	Max observed 0.986 (Domain B)	No — threshold not met	Suitable for deployment
Any domain — systematic bias	Exceeds ±10 percentage points	Max observed +5.5pp (Domain A)	No — threshold not met	Calibration recommended; deployment suitable
Human override authority	Must be structurally preserved at all adjudication stages	By design in proposed operational model	Requirement satisfied	Non-negotiable in all deployment contexts
AI score disclosure to presenters	Must not occur without prior human review	Not applicable in validation study	Requirement noted	Required before operational use
Score audit trail	Model version, Domain scores retained	Documented in this study	Requirement satisfied	Required for retrospective calibration

These thresholds were defined prior to analysis. A negative finding was defined as breach of any ICC, variance ratio, or bias threshold. None were breached. Governance requirements are stated as minimum standards for operational deployment and are independent of validation outcome.

Human reviewers

Three independent human reviewers (HRs, authors 3-5) were qualified psychiatrists with 32 years of cumulative academic and prior conference evaluation experience. All three HRs had experience in conducting and peer-reviewing research, making and judging poster presentations, and SRMA methodology (no formal certification). All received individual digital poster files with standardized PA²IRS scoring instructions and scored each poster independently, anonymized from each other’s and the AI’s scores in January 2026. Only domain-level (Domains A, B, C) and overall scores were used in analyses. Sub-domain scores were excluded from analysis.

Artificial Intelligence assessment system

A customized GPT was developed using OpenAI’s custom GPT builder (chatgpt.com/create), powered by GPT-5.2, accessed via a ChatGPT “Go,” an Indian subscription tier, in December 2025–January 2026. The tool was configured with system-level instructions embedding the PA²IRS framework criteria and scoring logic. Domain-A was evaluated in a single prompt; Domains B and C were evaluated together in a second prompt to maximize tokenization. The AI generated structured scores for each domain with concise criterion-linked justifications, and did not retain memory across assessments. Posters were uploaded as PDFs and assessed in single-pass evaluations to minimize contextual and carryover bias and prevent memory retention across assessments. The AI was permitted to use external web browsing for verifying factual claims and based all evaluations on the content of the uploaded poster.

Artificial Intelligence calibration and prompt engineering

Before formal assessment, the customized GPT was calibrated by one of the authors (RT, a postgraduate dentist with 24 years of academic experience) using a sequential, consensus-based approach with 50 dental posters representing all three poster types. Where scores deviated from PA²IRS’ intent, targeted prompt revisions were made, and the poster was re-evaluated until author-AI consensus was reached. Calibration was complete when prompts produced scores and rationales consistent with PA²IRS criteria across all poster types; the calibration set was drawn exclusively from dentistry to ensure no overlap with study data.

Prompt engineering followed a hierarchical instruction structure of the rubrics, with system-level rules defining PA²IRS domains, scoring logic and poster-type-specific criteria when the AI is supplied with a poster. The AI was explicitly instructed to: (i) adhere strictly to PA²IRS criteria; (ii) avoid inferential assumptions beyond poster content but verify claims if necessary; (iii) assign scores independently for each domain; and (iv) provide brief, criterion-linked rationales without narrative embellishment.

AI scoring reproducibility was assessed in a randomly selected 10% subset (n = 6) under two conditions: (i) re-analysis using the same account at a 30-day interval (temporal stability); and (ii) re-analysis using a different account at the same time point (inter-account variability). Identical scoring protocols were maintained across all repetitions.

The pre-specified thresholds for operational statistics were defined before analysis [Table S1, S2 in Supplemental File 2]. A negative finding was defined as a breach of any Intraclass correlation coefficients (ICC), variance ratio, or bias threshold. As the study did not contain any human participants, the work does not require a formal ethical review process.

Statistical analysis

Descriptive statistics (mean ± SD) were computed for AI and each HR by domain and poster type. ICC (ICC-2, two-way mixed effects, consistency model) were computed for: (i) inter-rater reliability among 3-HRs; (ii) agreement between AI scores and the mean of three HR scores; and (iii) four-rater reliability across all three-HRs and AI simultaneously. Single-measure and average-measure ICCs with 95% confidence intervals are reported; Cronbach’s α equals average-measure ICC under this model and is reported for completeness. Significance was set at P < 0.05.

Systematic bias and proportional agreement were assessed by Passing–Bablok regression and Bland–Altman analysis, performed using the online comparison web application (version 1.0.0, September 2022). Passing–Bablok regression is non-parametric and robust to outliers; Bland–Altman analysis provides the mean difference (bias) and 95% limits of agreement (LoA). Both analyses were conducted for each domain within each poster subgroup. Pre-specified criteria are shown in S2 [Supplementary File 2].

Variance component analysis benchmarked AI–human score disagreement against natural inter-human variability. For each domain and poster subgroup, a variance ratio was computed as: (a) variance of AI score minus mean HR score per poster (n values); divided by (b) variance of all pairwise absolute differences between HRs, pooled across three HR pairs (3n values), selected over averaged pairwise differences per poster to avoid artificially deflating the denominator. A ratio ≤1.0 indicates AI–human disagreement no greater than inter-human disagreement. Bartlett’s test was the primary significance test; Levene’s test (center = median) was applied as a robustness check. Main analyses were performed in IBM SPSS Statistics Version 25.0 and variance component analyses in Python 3.12 (SciPy 1.11).

RESULTS

Mean (±SD) scores by domain, poster type, and assessor are presented in Table S3. Domain-A scores were broadly comparable between AI and HR means across all poster types; Domain-B showed a consistent pattern of AI scoring lower than the HR mean; and Domain-C scores were closely aligned across all poster types.

Table-S3.

Mean (± SD) Scores by Domain, Poster Type, and Assessor of All Posters Used in This Study

Poster Type	Domain	AI	HR-1	HR-2	HR-3	Mean HR
Case Report/ Series (n=30)	Domain A (%)	65.58 ±11.65	59.33 ±14.65	60.83 ±17.65	52.08 ±13.10	57.42 ±12.32
	Domain B*	15.00 ±4.08	21.63 ±3.67	18.23 ±6.43	16.00 ±3.68	18.62 ±3.51
	Domain C*	6.63 ±0.67	7.07 ±2.08	6.03 ±2.61	6.07 ±1.93	6.39 ±1.83
	Overall (%)	59.83 ±9.66	65.54 ±12.86	60.75 ±17.93	53.63 ±9.43	59.97 ±11.50
Original Research (n=20)	Domain A (%)	58.22 ±14.08	52.22 ±15.47	66.11 ±19.09	49.44 ±9.88	55.93 ±12.80
	Domain B*	18.95 ±3.30	20.85 ±3.41	22.85 ±4.64	17.00 ±5.09	20.23 ±3.32
	Domain C*	5.95 ±1.23	5.45 ±2.58	6.10 ±2.10	5.65 ±2.08	5.73 ±1.89
	Overall (%)	60.12 ±10.34	57.76 ±14.04	69.06 ±16.53	52.82 ±10.42	59.88 ±11.38
Systematic Review-Meta-Analysis (n=10)	Domain A (%)	55.60 ±10.23	47.40 ±14.85	66.80 ±16.74	40.00 ±8.79	51.40 ±7.78
	Domain B*	17.90 ±2.77	21.50 ±2.88	23.00 ±5.46	14.30 ±4.03	19.60 ±2.94
	Domain C*	5.60 ±0.70	5.40 ±2.27	5.60 ±1.90	5.30 ±1.16	5.43 ±1.18
	Overall (%)	57.00 ±7.41	56.22 ±11.77	68.89 ±14.01	44.00 ±7.43	56.37 ±7.75

Domain A expressed as percentage of domain-specific maximum (Case reports/Series: 40pts; Original Research: 45pts; Systematic Review- Meta-Analysis: 50pts); Overall score expressed as percentage of total scores of Domain-A, B, C (Case reports/Series: 80pts; Original Research: 85pts; Systematic Review- Meta-Analysis: 90pts). *Domains B (30pts) and C (10pts) reported as raw scores. AI = Artificial Intelligence HR = Human Reviewer; Mean HR = mean of three reviewers

Human-human and AI-human ICC values are presented in Table 1 (average-measure, Cronbach’s α, and P values) and Table S4. Across all 60 posters, human-human average-measure ICC ranged from 0.52 (Domain-B) to 0.69 (Domain-C). The SRMA subgroup showed the lowest reliability: Domain-A ICC = −0.03 (95% CI − 0.3 to 0.44, P = 0.53), the only negative point estimate in the dataset, indicating near-zero to negative human-human reliability for content quality scoring in SRMA posters. AI–human ICCs met or exceeded human-human values across all four domains. SRMA that had the least reliable human-human agreement, showed the highest AI–human agreement: overall ICC 0.83 (95% CI: 0.45–0.95), domain-A 0.76, and domain-B 0.86, the highest domain-level value in the dataset. CR domain-B was the only non-significant AI–human ICC (P = 0.09). When AI was included as a fourth rater, the average-measure ICC for the overall score reached 0.75 (95% CI 0.63–0.84, P < 0.001), compared to 0.68 for three HRs alone.

Table 1.

Table showing average measures of intra-class correlation between three human reviewers, between AI and average of human reviewers and three human reviewers with AI

Domain/Poster Type	Average Measures of Intra Class Correlation
	Between 3 Human Reviewers			Between AI and Average of 3 Human Reviewers			Among 3 Human Reviewers and AI
	ICC	Cron Bach’s α	P	ICC	Cron Bach’s α	P	ICC	Cron Bach’s α	P
	Overall Score
All posters	0.68 (0.51–0.80)	0.68	<0.001	0.77 (0.61–0.86)	0.77	<0.001	0.75 (0.63–0.84)	0.75	<0.001
CR (n=30)	0.77 (0.58–0.89)	0.77	<0.001	0.69 (0.34–0.85)	0.69	<0.001	0.78 (0.61–0.88)	0.78	<0.001
OR (n=20)	0.75 (0.48–0.90)	0.75	<0.001	0.82 (0.55–0.93)	0.82	<0.001	0.81 (0.63–0.92)	0.81	<0.001
SRMA (n=10)	0.42 (-0.70–0.84)	0.42	0.16	0.91 (0.62–0.98)	0.91	<0.001	0.66 (0.09–0.90)	0.66	0.02
Domain-A
All posters	0.67 (0.49–0.79)	0.67	<0.001	0.77 (0.62–0.86)	0.77	<0.001	0.76 (0.64–0.84)	0.76	<0.001
CR (n=30)	0.73 (0.51–0.86)	0.73	<0.001	0.79 (0.56–0.90)	0.79	<0.001	0.80 (0.64–0.89)	0.80	<0.001
OR (n=20)	0.79 (0.55–0.91)	0.79	<0.001	0.73 (0.31–0.89)	0.73	0.003	0.81 (0.63–0.92)	0.81	<0.001
SRMA (n=10)	-0.09 (-2.20–0.70)	-0.09	0.53	0.87 (0.46–0.97)	0.87	0.003	0.45 (−0.45–0.85)	0.45	0.11
Domain-B
All posters	0.52 (0.26–0.70)	0.52	<0.001	0.61 (0.34–0.77)	0.61	<0.001	0.62 (0.43–0.75)	0.62	<0.001
CR (n=30)	0.58 (0.22–0.78)	0.58	0.003	0.40 (−0.26–0.71)	0.40	0.09	0.57 (0.26–0.78)	0.57	<0.001
OR (n=20)	0.61 (0.17–0.83)	0.61	0.01	0.69 (0.20–0.88)	0.69	0.01	0.69 (0.38–0.86)	0.69	<0.001
SRMA (n=10)	0.45 (-0.60–0.85)	0.45	0.13	0.92 (0.69–0.98)	0.92	<0.001	0.69 (0.17–0.91)	0.69	0.01
Domain-C
All posters (n=60)	0.73 (0.59–0.83)	0.73	<0.001	0.71 (0.52–0.83)	0.71	<0.001	0.76 (0.65–0.85)	0.76	<0.001
CR (n=30)	0.76 (0.57–0.88)	0.76	<0.001	0.51 (−0.03–0.77)	0.51	0.03	0.74 (0.55–0.87)	0.74	<0.001
OR (n=20)	0.78 (0.54–0.91)	0.78	<0.001	0.83 (0.57–0.93)	0.83	<0.001	0.83 (0.66–0.92)	0.83	<0.001
SRMA (n=10)	0.28 (-1.10–0.81)	0.28	0.26	0.69 (−0.27–0.92)	0.69	0.05	0.44 (−0.46–0.84)	0.44	0.12

CR=Case Report/Series; OR=Original Research; SRMA=Systematic Review-Meta-analysis; ICC=Intra Class Correlation

Table-S4.

Table showing Single Measure Intra-Class Correlation between 3 Human Reviewers, between AI and Average of Human Reviewers and 3 Human Reviewer with AI

Domain / Poster Type	Single Measures ICC
	Between 3 HRs	Between AI and Average of 3 HR	Among 3 HR and AI
Overall Score
All posters (n=60)	0.68 (0.51–0.80)	0.77 (0.61–0.86)	0.75 (0.63–0.84)
Case report/series (n=30)	0.77 (0.58–0.89)	0.69 (0.34–0.85)	0.78 (0.61–0.88)
Original research (n=20)	0.75 (0.48–0.90)	0.82 (0.55–0.93)	0.81 (0.63–0.92)
SRMA (n=10)	0.42 (-0.70–0.84)	0.91 (0.62–0.98)	0.66 (0.09–0.90)
Domain-A
All posters (n=60)	0.67 (0.49–0.79)	0.77 (0.62–0.86)	0.76 (0.64–0.84)
Case report/series (n=30)	0.73 (0.51–0.86)	0.79 (0.56–0.90)	0.80 (0.64–0.89)
Original research (n=20)	0.79 (0.55–0.91)	0.73 (0.31–0.89)	0.81 (0.63–0.92)
SRMA (n=10)	-0.09 (-2.20–0.70)	0.87 (0.46–0.97)	0.45 (−0.45–0.85)
Domain-B
All posters (n=60)	0.52 (0.26–0.70)	0.61 (0.34–0.77)	0.62 (0.43–0.75)
Case report/series (n=30)	0.58 (0.22–0.78)	0.40 (−0.26–0.71)	0.57 (0.26–0.78)
Original research (n=20)	0.61 (0.17–0.83)	0.69 (0.20–0.88)	0.69 (0.38–0.86)
SRMA (n=10)	0.45 (-0.60–0.85)	0.92 (0.69–0.98)	0.69 (0.17–0.91)
Domain-C
All posters (n=60)	0.73 (0.59–0.83)	0.71 (0.52–0.83)	0.76 (0.65–0.85)
Case report/series (n=30)	0.76 (0.57–0.88)	0.51 (−0.03–0.77)	0.74 (0.55–0.87)
Original research (n=20)	0.78 (0.54–0.91)	0.83 (0.57–0.93)	0.83 (0.66–0.92)
SRMA (n=10)	0.28 (-1.10–0.81)	0.69 (−0.27–0.92)	0.44 (−0.46–0.84)

Systematic bias is summarized in Table 2. For overall scores across all 60 posters, the mean AI–human difference was 0.11 percentage points (pp), negligible aggregate bias. Domain-A showed a consistent positive bias across all subgroups (all 60: +5.55 pp; CR: +8.17 pp; OR: +2.30 pp; SRMA: +4.20 pp), indicating AI assigned modestly higher content quality scores than the human average. Domain-B showed a consistent negative bias (all 60: −2.52 raw; CR: −3.62; SRMA: −1.70). Domain-C bias was negligible (0.17–0.24). The SRMA subgroup showed the narrowest LoA for the overall score (−8.08 to +9.34 pp, width 17.42 pp) and the only Passing–Bablok slope that approximated unity (1.00). [Supplemental File 3]

Table 2.

Bland–Altman agreement analysis and Passing–Bablok regression: AI vs mean of human reviewers scores

Subset	Domain	Bias	Lower LoA	Upper LoA	LoA Width	Passing–Bablok Equation
All Posters (n=60)	Overall (%)	0.11	−17.29	17.51	34.80	y=−12.95+1.23x
	Domain A (%)	5.55	−15.24	26.34	41.58	y=3.15+0.89x
	Domain B (raw)	−2.52	−10.28	5.23	15.51	y=5.66+0.78x
	Domain C (raw)	0.22	−2.44	2.89	5.33	y=−8.71+2.34x
Case Report (n=30)	Overall (%)	−0.14	−20.46	20.18	40.64	y=−15.83+1.27x
	Domain A (%)	8.17	−11.47	27.80	39.27	y=−7.50+1.00x
	Domain B (raw)	−3.62	−12.76	5.51	18.27	y=5.65+0.83x
	Domain C (raw)	0.24	−2.86	3.35	6.21	— (Unstable)
Original Research (n=20)	Overall (%)	0.24	−16.29	16.76	33.05	y=−14.89+1.30x
	Domain A (%)	2.30	−22.10	26.70	48.80	y=4.72+0.90x
	Domain B (raw)	−1.28	−7.63	5.06	12.69	y=−1.33+1.11x
	Domain C (raw)	0.22	−2.17	2.60	4.77	y=−5.18+1.84x
Systematic Review- Meta-Analysis (n=10)	Overall (%)	0.63	−8.08	9.34	17.42	y=0.39+1.00x
	Domain A (%)	4.20	−8.08	9.34	17.42	y=9.93+0.78x
	Domain B (raw)	−1.70	−4.68	1.28	5.95	y=−4.45+1.33x
	Domain C (raw)	0.17	−1.69	2.02	3.71	y=−4.36+1.67x

Bias=mean (AI−mean HR score). LoA=95% limits of agreement. Domain-A: percentage; Domain-B, C: raw scores. Bias values exceeding±5 pp highlighted. Domain-C CR Passing–Bablok equation not estimable due to near-zero slope variability. LoA=Limits of agreement; pp=percentage points; HR=Human reviewer; CR=Case reports/series; AI=Artificial intelligence

Variance ratios are presented in Table 3. Across all 60 posters, all variance ratios were ≤1.0, including domain-B (0.99) and domain-C (0.940), confirming AI–human disagreement did not exceed inter-human disagreement for any domain. Bartlett’s test reached significance only for the overall score (P = 0.049), reflecting differences in distributional shape; all domain-level tests were non-significant (P ≥ 0.170). CR ratios remained below 1.5 (Domain-B highest: 1.43, P = 0.23); OR ratios were similarly contained (Domain-A highest: 1.28, P = 0.51).

Table 3.

Complete variance statistics by domain and poster subgroup

Subset	Domain	Mean AI−HR	SD AI−HR	Mean HR pairs	SD HR pairs	Var Ratio	Bartlett P	Levene P
All (n=60)	Overall (%)	0.11	8.88	13.83	11.04	0.65	0.049	0.147
	Domain A (%)	5.55	10.61	15.20	12.32	0.74	0.170	0.375
	Domain B (raw)	−2.52	3.96	5.74	3.98	0.99	0.949	0.656
	Domain C (raw)	0.22	1.36	1.78	1.40	0.94	0.773	0.790
CR (n=30)	Overall (%)	−0.14	10.37	12.31	9.74	1.13	0.679	0.444
	Domain A (%)	8.17	10.02	13.06	10.44	0.92	0.787	0.969
	Domain B (raw)	−3.62	4.66	5.60	3.90	1.43	0.229	0.423
	Domain C (raw)	0.24	1.58	1.71	1.55	1.05	0.874	1.000
OR (n=20)	Overall (%)	0.24	8.43	14.39	10.70	0.62	0.226	0.414
	Domain A (%)	2.30	12.45	15.48	11.01	1.28	0.505	0.660
	Domain B (raw)	−1.28	3.24	5.50	3.51	0.85	0.677	0.669
	Domain C (raw)	0.22	1.22	1.80	1.22	0.1	0.989	0.691
SRMA (n=10)	Overall (%)	0.63	4.44	17.26	14.41	0.1	<0.001	0.004
	Domain A (%)	4.20	6.24	21.07	17.50	0.13	0.002	0.006
	Domain B (raw)	−1.70	1.52	6.67	5.02	0.09	<0.001	0.004
	Domain C (raw)	0.17	0.95	1.93	1.31	0.52	0.264	0.244

AI−HR=AI score minus mean of three human reviewer scores per poster (n values per cell); HR pairs=all three pairwise absolute differences between human reviewers, pooled (3n values per cell); Var Ratio=Var (AI−HR)/Var (HR pairs). Bartlett P and Levene P compare variance of n AI−HR differences against 3n pooled HR pairwise differences. HR=Human reviewer; AI=Artificial intelligence; CR=Case reports/series; OR=original research; SRMA=Systematic reviews-meta-analyses

The SRMA subgroup produced a qualitatively distinct pattern. Variance ratios for overall score (0.10), domain-A (0.13), and domain-B (0.09) were all substantially below 1.0, with Bartlett’s and Levene’s tests both highly significant (P < 0.001, 0.002, and <0.001, respectively). AI–human disagreement was approximately one-tenth of inter-human disagreement (SD 14.41–17.50), while AI scoring remained consistent (SD 4.44–6.24). Domain-C was the exception (ratio 0.52, P = 0.26).

AI scoring reproducibility in the 10% subsample (n = 6) is presented in Table 4. Findings are directionally informative only. Temporal stability was highest for domain-A (ICC 0.84, r = 0.84, P = 0.035), acceptable for domain-C (ICC 0.5) and overall score (ICC 0.7), and lowest for domain-B (ICC 0.54). A counterintuitive pattern emerged: Domain-B ICC was substantially higher inter-account (0.73, r = 0.97, P = 0.002) than temporally (0.54), while domain-A remained high (ICC 0.87) and domain-C showed markedly lower cross-account reliability (ICC 0.17, P = 0.66). Mean differences were small throughout, confirming no systematic directional drift.

Table 4.

AI Scoring Reproducibility: Temporal Stability and Inter-Account Variability (n=6 subsample)

Domain	Pearson r	P	ICC	Mean Difference±SD
Panel A—Temporal Stability: Same Account, 30-Day Interval (n=6)
Domain A (%)	0.84	0.04	0.84	1.86±6.35
Domain B (raw)	0.75	0.08	0.54	−2.83±2.79
Domain C (raw)	0.46	0.37	0.50	0.00±1.10
Overall (%)	0.74	0.09	0.7	−3.00±6.20
Panel B—Inter-Account Variability: Different Account, Different Time (n=6)
Domain A (%)	0.85	0.03	0.87	0.66±6.32
Domain B (raw)	0.97	0.002	0.73	−1.67±1.03
Domain C (raw)	0.23	0.66	0.17	−0.33±1.03
Overall (%)	0.87	0.02	0.79	−2.92±3.40

n=6 (10% random subsample). ICC=intraclass correlation coefficient. Mean Difference=second evaluation minus first. Findings are directionally informative; confidence intervals not reported due to subsample size. Domain A and Overall: percentage; Domains B, C: raw scores

Representative verbatim AI evaluation statements across assessment domains and poster types are presented in Table 5, providing a mechanistic context for the quantitative agreement patterns. In domain A, AI demonstrated criterion-sequential, taxonomy-sensitive reasoning across all poster types. For case reports, the statement “The case is positioned as educational/illustrative, not first-in-literature” reflects a precise differentiation between novelty and educational value, a distinction that domain-A criteria explicitly operationalize but that HRs frequently conflate. For OR posters, AI identified both present and absent methodological elements (“Robust single-blind, cluster-randomized design… predefined non-inferiority margin” followed by “Power calculation details… allocation concealment… not fully described”), applying criteria to gaps as rigorously as to documented features. For SRMA posters, AI deployed language aligned with Cochrane and PRISMA frameworks (“structured research question framing and formal risk-of-bias assessment”).

Table 5.

Representative Verbatim AI Evaluation Statements: Qualitative Analysis of Criterion-Referenced Reasoning

Poster Type	Domain	Representative AI Quote	Interpretation
Case Report	A	“The case is positioned as educational/illustrative, not first-in-literature.”	Differentiation of novelty vs educational value
Case Report	B	“High cognitive load due to long paragraphs and ‘wall of text’ sections… Gestalt principles minimally applied.”	Application of perceptual design concepts
Case Report	C	“Clear clinical lesson: proactive counselling, early monitoring, and clinician vigilance… meaningful educational value.”	Recognition of translational relevance
Original Research	A	“Robust single-blind, cluster-randomized design… validated instruments… predefined non-inferiority margin.”	Methodological rigor appraisal
Original Research	A	“Power calculation details… allocation concealment… not fully described.”	Identification of reporting gaps
Original Research	B	“Key results are not visually foregrounded… no prominent ‘Key Findings’ box or visual emphasis.”	Communication efficiency critique
Original Research	C	“Directly inform service delivery models and task-shifting strategies.”	Translational value recognition
Systematic Review	A	“Key elements expected of high-quality systematic reviews are incompletely reported, including structured research question framing and formal risk-of-bias assessment.”	Evidence synthesis evaluation
Systematic Review	B	“Multiple appropriate analytical visuals… figures substantially aid interpretation, though annotation could further enhance clarity.”	Data visualization interpretation
Systematic Review	C	“Incremental but legitimate contribution that refines safety understanding rather than transforming practice.”	Nuanced impact assessment

Statements selected to represent the range of criterion-referenced reasoning observed across domains and poster types

In domain-B, AI statements made explicit use of perceptual design theory: “High cognitive load due to long paragraphs and ‘wall of text’ sections… Gestalt principles minimally applied.” This represents the operationalized System-2 equivalent of the immediate, holistic visual impression that HRs experience, the same underlying principles applied as explicit sequential criteria rather than as an automatic pre-analytical response.

In domain-C, AI demonstrated nuanced impact grading: “Incremental but legitimate contribution that refines safety understanding rather than transforming practice”—distinguishing incremental from transformative contributions at a level that parallels expert reviewer judgment. The consistent domain-C Passing–Bablok slope amplification (range 1.5–3.37 across poster types, Table 5) indicates that AI assigns impact scores within a narrower central band, whereas HRs utilize the full-scale range.

DISCUSSION

Principal findings and novel contributions

This study provides the first reliability evidence for AI-assisted biomedical poster assessment using the PA²IRS (rubrics) framework, with two novel contributions: (i) AI performance is benchmarked against inter-human variability rather than an assumed human gold standard, and (ii) inclusion of AI as a fourth rater maintains or improves collective panel reliability.

The poster assessment via Artificial Intelligence-integrated rubric system scoring architecture: Intentional separation of Artificial Intelligence and human components

PA²IRS is a 100-point instrument with explicit structural separation between the poster component (assessed here) and the interview component (reserved for human evaluators). The interview weighting follows a principled inverse gradient relative to poster content complexity: highest for CR (20/100), where live interaction tests communication skills the poster alone cannot demonstrate; lowest for SRMA (10/100), where methodological rigor is entirely encoded in the written content. This architectural logic maps directly onto the agreement findings: AI agreement was strongest where the poster carries the most evaluative weight, SRMA (poster 90/100, AI-human ICC 0.83), and most variable where the interview compensates most, in CR (poster 80/100, domain-B ICC 0.250, non-significant).This maps directly onto the agreement findings: AI agreement was strongest for SRMA (poster 90/100, ICC 0.83) and most variable for CR (poster 80/100, domain-B ICC 0.25, non-significant).

Psychiatry as a conservative test case

Validation in psychiatry constitutes a deliberately conservative test. LLMs have previously demonstrated inconsistent performance on nuanced psychiatric constructs, and psychiatric content may be underrepresented in LLM training distributions relative to other biomedical domains.[10,11,12,13,14] Agreement achieved here therefore represents a lower-bound estimate. PA²IRS performance in more methodologically homogeneous disciplines is likely to be equal or superior.

Score agreement and systematic bias

Near-zero aggregate bias for overall scores (0.11 pp) and domain C (0.17–0.24 raw) indicates excellent calibration. Domain-A showed a consistent positive bias of approximately 5.5 pp across all poster types:. AI credits each verifiable content element without the moderating influence of holistic impression, potentially assigning full credit to well-structured content that HRs partially discount when the overall visual or impact impression is suboptimal. Domain-B showed a consistent negative bias, interpretable as AI applying visual organization criteria that HRs may apply more generously, yielding more holistic impressions.

Domain-C Passing–Bablok slopes consistently exceeded unity (range 1.50–3.37), indicating systematic range compression. AI assigns impact scores within a narrower central band, while HRs utilize the full scale. Domain-C AI scores are therefore better suited to rank-ordering than to direct scalar substitution, with compression greatest for SRMA posters where interpretive expertise and experience-weighted inference are most consequential.

A dual-process account of domain-dependent agreement

The domain-dependent agreement pattern is consistent with dual-process models of cognition applied to clinical reasoning. Croskerry proposed a universal model of diagnostic reasoning involving System-1 (heuristic, intuitive) and System-2 (systematic, analytical) modes across healthcare domains.[15,16] This study extends the framework to AI–HR divergence. AI may operate exclusively in System-2 across all domains. HRs may have engaged System-1 involuntarily for visually and affectively loaded content, shaping criterion scores differentially. Domain-A’s verifiable, presence-based criteria engage both assessors analytically, producing the highest convergence. Domain-B’s immediate aesthetic response before rubric application produces the highest inter-human variance and a variance ratio of 0.986, AI–human disagreement statistically indistinguishable from inter-human disagreement, achieved through entirely different cognitive routes. Domain-C occupies an intermediate position. Domain-B’s lowest agreement should not be read as AI struggling with visual assessment; rather, HRs differ from each other, and AI’s divergence from the human average is no greater than the divergence between HRs themselves.

Among experienced reviewers, System-1 responses may not be arbitrary “noise;” they could encode cumulative clinical and conference exposure, constituting a richly informed prior that criterion-level scoring refines rather than replaces.[17,18] AI applies each criterion independently and without experiential prior, directly underlying the domain-C range compression (slopes 1.50–3.37): AI occupies the scale center because it lacks the experience-weighted anchor points that allow expert reviewers to score confidently at the extremes.[19,20]

Available evidence corroborates these domain-differentiated patterns. LLMs outperform humans on on analytical rubrics, but are outperformed on holistic judgment, a distinction the PA²IRS domain structure directly addresses.[21,22,23] The present study assessed HRs under controlled, non-conference conditions. In real settings, cognitive load, travel fatigue, and social-affective priming selectively impair System-2 capacity while leaving System-1 intact, meaning the reliability gaps reported here likely underestimate AI’s operational advantage in real conference conditions.[24,25,26]

Artificial Intelligence as a complementary rather than substitute evaluator

The agreement pattern is consistent with AI and HRs operating through structurally different evaluative frameworks rather than the same process with different outcomes. Where criteria are verifiable and presence-based (domain-A), HR and AI converge. Where holistic impression precedes rubric application (domains B and C), HRs diverge from each other, and AI diverges from the human average by no greater margin. This supports a complementary rather than substitutive deployment model, consistent with findings showing humans and LLMs demonstrate non-overlapping strengths across holistic and analytical scoring contexts.[21,22]

For formative and pre-conference screening, AI’s sequential, consistent scoring may offer a practical advantage. Every submission receives rubric-anchored feedback irrespective of disciplinary background or evaluator cognitive state. The present data do not support AI for standalone final evaluation or communicative immediacy, disciplinary significance, and the verbal defense component remain outside the scope of this poster-only validation. They are explicitly reserved for human evaluation within PA²IRS.[5,27,28]

The present findings should not be interpreted as evidence that AI can independently replace human judgement in assessing methodological rigor. While AI demonstrated agreement with HRs comparable to natural inter-human variability, methodological quality frequently requires contextual interpretation, discipline-specific knowledge, and consideration of nuances beyond explicit rubric criteria. Accordingly, PA²IRS was intentionally designed as a hybrid framework in that AI provides structured, criterion-based assessment of the poster component, whereas HRs retain responsibility for contextual interpretation, evaluation of the verbal defense, adjudication, and final scoring decisions. These findings therefore support AI-assisted standardization of preliminary assessment rather than autonomous evaluation.

Disciplinary breadth and the systematic review-meta-analysis boundary condition

Assembling reviewers with adequate SRMA literacy to consistently apply PRISMA, GRADE, and AMSTAR criteria is rarely feasible. The present data reflect this directly: human-human ICC for domain-A was −0.03 (P = 0.53), indicating that reviewer disagreement exceeded the between-poster variance. AI–human agreement for the same domain was substantially higher (ICC 0.76), with a variance ratio of 0.13. Until reviewer panels include those with formal epidemiological or evidence synthesis training, AI-assisted assessment will likely provide more consistent domain-A evaluation for SRMA posters than any realistically assembled human panel.

Artificial Intelligence scoring reproducibility and the stochastic ceiling

AI-generated scores may exhibit inherent variability due to stochastic sampling and session-level contextual sensitivity in LLM architectures, thereby imposing a ceiling on the reproducibility achievable through prompt-based instructions alone. This underscores the value of safeguards such as temperature minimization, chain-of-thought prompting, and multi-run score averaging in structured scoring tasks.[29,30,31,32,33,34,35,36,37,38] The reproducibility subsample (n = 6, directional only) demonstrated acceptable to good temporal stability for domain-A and overall score, with score variation attributable to model stochasticity rather than user-specific factors. The domain-B reversal, higher cross-account (ICC 0.73) than temporal reliability (ICC 0.54), indicates visual domain scoring is more strongly determined by poster structural features than session context, with practical implications for multi-instance AI deployment.

Limitations

Several limitations apply including n = 60 posters. The HR panel was not matched for sub-disciplinary expertise across poster types. The reproducibility subsample (n = 6) precludes precise estimation. SRMA findings are directionally strong but imprecise at n = 10; replication requires a minimum of 30 SRMA posters. Assessment under controlled non-conference conditions constitutes a best-case estimate of HR reliability. Construct validity was not addressed and represents the necessary next step before high-stakes summative application. Comprehensive threat-to-validity analysis and future research directions are provided in supplemental tables S5 and S6.

Table-S5.

Threat-to-Validity Analysis – Table highlighting limitation of the study due to factors.

Threat	Direction of bias	Estimated magnitude	Mitigation
Reviewers not matched to exact poster sub-discipline	Inflates inter-human variance; favours AI agreement	Moderate — demonstrated in SRMA subgroup	Subgroup ICC reported separately
Non-conference conditions	Underestimates real-world inter-human variability	Moderate — neuroscientific evidence cited	Stated as best-case human estimate
Dentistry calibration set	May reduce Domain-A AI precision in psychiatric context	Small to moderate	Psychiatric calibration recommended
Reproducibility subsample n=6	Wide CIs; directional only	High uncertainty	Not used for primary inference
Fixed AI model version	Findings reflect one LLM generation	Unknown	Model version documented
Proprietary criteria	External criterion-level replication not possible	Moderate	Instrument limitation, not study design
Purposive stratified sampling	Pooled estimates do not reflect conference prevalence	Moderate	Equal-weighting stated in Methods
Human Construct validity not assessed	Reliability without construct evidence is necessary but insufficient	Material	Identified as future validation priority

Table-S6.

Future Research Directions Arising from PA²IRS Validation: Priorities for Subsequent Investigation

#	Research question	Rationale	Suggested design	Primary outcome
1	Does criterion-sequential AI feedback translate into measurable improvement in presenter performance?	Validation is reviewer-facing; educational impact on presenters remains untested	Randomised comparison: AI feedback vs human feedback vs no feedback prior to conference presentation	Poster quality score improvement; presenter self-efficacy
2	Does AI pre-scoring attenuate anchoring effects in sequential human poster review?	Human reviewers recalibrate scores relative to early session posters; AI does not exhibit this bias	Counterbalanced within-session design; reviewers with and without AI pre-scores across session order conditions	Variance in human scores across session position
3	Does AI literal criterion application disadvantage presenters operating in English as a second language?	Human reviewers make implicit language accommodations; AI applies criteria literally regardless of language proficiency	Stratified analysis by presenter language background; AI vs human score differences as primary comparison	Differential item functioning by language group
4	What is the minimum number of AI scoring runs required for stable domain-level scores?	Single-run AI assessment is subject to stochastic variability; multi-run averaging is recommended but not yet formally quantified	Repeated-measures reliability study across increasing run numbers per poster	ICC stabilisation curve by number of runs
5	Does AI pre-scoring improve collective human panel reliability in live conference conditions?	Controlled conditions in this study represent best-case human reliability; real-conference advantage of AI may be larger	Randomised controlled design: human panels with vs without AI pre-scores in operational conference setting	Human-human ICC with and without AI baseline
6	Can PA²IRS Domain C impact scoring be improved through experience-anchored prompt design?	Range compression in Domain C is consistent with absence of experiential prior; anchor-based prompting may partially compensate	Prompt engineering study using exemplar-anchored vs standard prompts across impact score distribution	Passing-Bablok slope and intercept toward unity
7	Does PA²IRS perform equivalently across biomedical disciplines beyond psychiatry?	Psychiatric validation provides conservative lower-bound estimate; generalisability to other disciplines unestablished	Multi-site replication across medicine, surgery, and basic science conference settings	AI-human ICC and variance ratio replication
8	What is the construct validity evidence for PA²IRS domain scores?	Reliability without construct validity confirms consistent measurement, not valid measurement	Convergent validity against independent expert ratings; discriminant validity across poster quality strata; content validity indexing	Messick unified validity framework criteria

CONCLUSION

AI-assisted poster assessment using PA²IRS produced agreement with HRs within the natural reliability range of trained humans across all domains and poster types sampled from an International Psychiatrist conference. Validation in psychiatry provides a conservative lower-bound estimate.PA²IRS-based AI assessment will perform at least as well as, if not better than, across less methodologically complex biomedical disciplines. The domain-dependent agreement pattern reflects two complementary evaluative orientations rather than differential accuracy between raters. AI applies criteria sequentially and consistently across disciplinary breadth, while HRs bring holistic judgment and localized domain expertise. These are distinct but compatible frameworks, and their divergence across domains is structural rather than indicative of one rater outperforming the other. The domain-dependent pattern reflects complementary evaluative orientations. AI’s sequential application of criteria versus human holistic judgment, rather than differential accuracy. Divergence is structural, not hierarchical. A. multi-stage model with AI for standardized preliminary scoring and formative feedback, and HRs for interactive evaluation and final scoring, represents the most evidence-consistent operational framework. Validation across disciplines, conference contexts, and live presentation settings, together with formal content validity indexing, constitutes the necessary next step. Operational deployment should preserve human override authority at every adjudication stage, with full score audit trails retained for retrospective calibration and accountability.

AI use statement

• A customized GPT (OpenAI, GPT-5.2 and above via “Go”, subscription model) was used as a part of framework evaluation of PA²IRS the assessment instrument in this study, as described in the Methods section in detail. This constitutes the research intervention and is reported in full.

• Additionally, a LLM – Claude Sonet 4.6 and Grammarly was used for manuscript editing assistance, the former to polish the language and reduce the word count while the latter for reducing typographical errors and language handling.

• All intellectual content, data interpretation, and conclusions are entirely the work of the authors, who take full responsibility for the accuracy and integrity of this manuscript.

Ethics approval

This study did not involve patients, biological materials, or identifiable human participants. The poster data were accessed from a publicly available online repository. The human reviewers were co-investigators/authors who participated voluntarily. Formal ethics committee review was therefore not applicable to this study.

Supplementary material

This article has supplementary material and can be accessed at this link. Supplementary Files 1-3 at https://links.lww.com/IPSY/A20, https://links.lww.com/IPSY/A21 and https://links.lww.com/IPSY/A22.

Conflicts of interest

Prof. Rooban Thavarajah and Prof. Anusa Mohandoss Arunachalam are co-copyright holders of the PA²IRS framework used in this study. Mr. R Santosh, the third co-copyright holder and acknowledged contributor above, is the son of Prof. Rooban Thavarajah and Prof. Anusa Mohandoss Arunachalam. These relationships are declared in the interest of transparency; no financial benefit was derived from the use of the framework in this research. All other authors declare no conflict of interest

Supplementary file 2

PASSING-BABLOK REGRESSION SCATTER PLOT ALL POSTERS IN THE STUDY OVERALL SCORES

BLAND-ALTMAN PLOT ALL POSTERS IN THE STUDY OVERALL SCORES

PASSING-BABLOK REGRESSION SCATTER PLOT CASE REPORT POSTERS IN THE STUDY OVERALL SCORES

BLAND-ALTMAN PLOT CASE REPORT POSTERS IN THE STUDY OVERALL SCORES

PASSING-BABLOK REGRESSION SCATTER PLOT ORIGINAL RESEARCH POSTERS IN THE STUDY OVERALL SCORES

BLAND-ALTMAN PLOT ORIGINAL RESEARCH POSTERS IN THE STUDY OVERALL SCORES

PASSING-BABLOK REGRESSION SCATTER PLOT SYSTEMATIC REVIEW-META-ANALYSIS POSTERS IN THE STUDY OVERALL SCORES

BLAND-ALTMAN PLOT SYSTEMATIC REVIEW-META-ANALYSIS POSTERS IN THE STUDY OVERALL SCORES

Funding Statement

Nil.