Reassessing the validity of AI cephalometric landmark identification: an analytical perspective on Ye et al. (2023)

Abstract

To provide the public and non-specialist readers with clear insight, we re-evaluate a published study claiming that artificial intelligence (AI) software performs as accurately as orthodontist experts in identifying key points on dental X-ray images. Our analysis reveals flaws in the original research methods: inconsistent image processing led to unreliable measurements, and critical details about data sources were missing. Most importantly, when we repeated the experiment with new X-ray images and protocols, over 80% of the AI’s landmark identifications showed significantly higher error rates than originally reported—contradicting the claim that AI matches expert accuracy. We urge methodological rigor in future studies on AI’s role in dentistry.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12903-026-08728-2.

This article is a critique of a published article [1] (Is automatic cephalometric software using artificial intelligence better than orthodontist experts in landmark identification? BMC Oral Health. 2023 Jul 8,23(1):467. 10.1186/s12903-023-03188-4). We have identified several fundamental methodological issues that undermine the reliability of its conclusions. First, we raise concerns about image analysis standardization, specifically regarding inconsistent image resolution handling across different software platforms, with MyOrthoX notably failing to provide original-resolution images due to its limitation on downloadable outputs, potentially distorting measurements. Second, reproducibility concerns emerged starkly through our experimental replication: using new lateral cephalometric radiographs from diverse sources and stricter protocols, we identified significantly higher error rates than those reported, challenging the reliability of the original findings. Third, the lack of a comparative analysis between manual and automated landmarking obscures validation of the AI's alleged equivalence, a critical omission for clinical application. Fourth, insufficient data source clarification regarding the software used for manual landmarking and the origin of the 43 sample images (including inconsistencies in pixel dimensions across platforms) complicates reproducibility and accuracy assessment. Overall, this critique argues that the original study's conclusions are potentially unreliable due to these methodological flaws and lack of transparency. We urge enhanced rigor in future research to ensure trust in AI tools for orthodontic practice.

Standardization of image analysis

The method of uploading images to the three software (MyOrthoX, Angelalign, and Digident) and then scanned into ImageJ (obtain the coordinates of each landmark) may introduce discrepancies due to image resolution changes in different software. As we known, if the pictures input into Image J were captured by screenshot, the image resolution of a screenshot corresponds to the resolution of the display screen, and if the pictures input into Image J were downloaded from the three software, the software may change the original pictures and result in different image pixel dimensions. It is recommended that the scale be recalibrated within ImageJ to account for these potential variations. We would like to emphasize that we found that the software MyOrthoX, on its official website (https://mydentalx.com/worktable/ceph), does not allow downloading of original resolution images and cannot export images with key points. We can only obtain images through screen capture for further processing with ImageJ. On the other hand, Angelalign and Digident, on their official websites (https://iortho.angelalign.com/ and https://www.zhibeicloud.com), allow downloading of original resolution images. Such differences can directly affect the inconsistent resolution of images in subsequent processing, thus affecting the experimental results.

Experiment’s reproducibility and credibility

To better demonstrate our ideas and address our concerns, we repeated the previous study's scenarios using 49 new lateral cephalometric radiographs. We used the same image processing software (Image J) as the original paper to extract 31 landmarks (The landmark Pcd was excluded due to uncertainty regarding whether the AI was accountable for its localization in Digident), and for each landmark, we tested 49 samples from different sources, using different scales and publicly available datasets (Supplementary materials, Image Dataset). The mean radial errors (MRE) were assessed up to thresholds of 1.0 mm, 1.5 mm, and 2 mm to compare the successful detection rate (SDR). The results show that the MyOrthoX dataset had the lowest average MRE among the 31 landmarks at 1.66 ± 1.03 mm, while the Digident and Angelalign datasets had average MREs of 1.98 ± 1.17 mm and 1.80 ± 1.15 mm, respectively (Table 1).

Table 1.

The comparison of cephalometric landmark prediction errors by three AI system between Ye et al. (https://doi.org/10.1186/s12903-023-03188-4) and our study, and the landmark Pcd was not included

Landmark	MyorthoX				Angelalign				Digident
	Ye et al. (n = 43)	Our (n = 49)	p1	p2	Ye et al. (n = 43)	Our (n = 49)	p1	p2	Ye et al. (n = 43)	Our (n = 49)	p1	p2
G	0.65 ± 0.42	3.37 ± 2.75	0.000	0.000	0.46 ± 0.54	2.81 ± 1.85	0.000	0.000	0.45 ± 0.37	2.38 ± 2.21	0.000	0.000
Prn	0.84 ± 0.43	1.00 ± 0.61	0.155	0.146	0.70 ± 1.21	2.81 ± 0.88	0.000	0.000	2.43 ± 1.57	3.61 ± 0.99	0.000	0.000
Sn	0.55 ± 0.32	0.91 ± 0.46	0.000	0.000	0.63 ± 0.71	0.37 ± 0.22	0.017	0.025	0.71 ± 0.39	0.72 ± 0.36	0.899	0.899
Ls	0.89 ± 0.66	1.94 ± 0.90	0.000	0.000	0.69 ± 0.83	1.67 ± 0.76	0.000	0.000	0.57 ± 0.33	3.29 ± 1.01	0.000	0.000
Li	0.90 ± 0.49	1.51 ± 0.82	0.000	0.000	0.60 ± 0.63	0.90 ± 0.55	0.017	0.018	0.56 ± 0.29	0.86 ± 0.64	0.006	0.004
Si	1.26 ± 0.68	1.01 ± 0.61	0.066	0.068	0.73 ± 0.68	0.78 ± 0.54	0.695	0.700	1.10 ± 0.65	1.03 ± 0.73	0.630	0.628
Pog’	0.61 ± 0.35	1.93 ± 1.74	0.000	0.000	0.58 ± 0.72	2.19 ± 1.79	0.000	0.000	0.60 ± 0.37	3.32 ± 1.80	0.000	0.000
Gn’	0.68 ± 0.39	2.62 ± 1.59	0.000	0.000	0.57 ± 0.62	2.95 ± 2.11	0.000	0.000	0.60 ± 0.47	2.60 ± 2.03	0.000	0.000
Me’	0.78 ± 0.83	1.92 ± 1.40	0.000	0.000	0.63 ± 0.56	4.40 ± 1.92	0.000	0.000	0.64 ± 0.57	2.43 ± 1.44	0.000	0.000
C	0.93 ± 0.45	2.90 ± 2.03	0.000	0.000	0.87 ± 0.76	4.11 ± 3.26	0.000	0.000	0.82 ± 0.92	3.51 ± 3.05	0.000	0.000
Ba	0.81 ± 0.84	2.03 ± 1.63	0.000	0.000	0.96 ± 0.97	2.90 ± 2.33	0.000	0.000	0.82 ± 0.67	2.15 ± 1.55	0.000	0.000
S	0.73 ± 0.45	0.62 ± 0.36	0.196	0.203	0.48 ± 0.34	0.58 ± 0.29	0.131	0.136	0.67 ± 0.42	0.62 ± 0.33	0.525	0.531
N	1.11 ± 0.70	1.25 ± 0.61	0.308	0.313	0.80 ± 0.69	0.86 ± 0.59	0.654	0.657	1.12 ± 0.63	0.96 ± 0.67	0.243	0.241
Po	0.92 ± 0.59	1.41 ± 1.26	0.022	0.017	1.48 ± 1.39	1.62 ± 1.45	0.639	0.638	0.97 ± 0.67	2.20 ± 1.51	0.000	0.000
Or	1.05 ± 0.77	1.68 ± 0.88	0.000	0.000	0.70 ± 0.48	2.09 ± 1.11	0.000	0.000	0.99 ± 0.65	1.62 ± 0.84	0.000	0.000
ANS	2.06 ± 1.32	1.77 ± 1.12	0.257	0.263	0.90 ± 0.99	1.23 ± 0.90	0.097	0.100	2.16 ± 1.15	1.66 ± 1.08	0.034	0.035
PNS	0.65 ± 0.36	1.20 ± 0.69	0.000	0.000	0.67 ± 0.52	1.01 ± 0.67	0.008	0.008	0.66 ± 0.47	1.12 ± 0.61	0.000	0.000
Pt	1.42 ± 1.03	2.54 ± 1.45	0.000	0.000	1.49 ± 1.24	2.44 ± 1.49	0.001	0.001	1.15 ± 1.27	2.82 ± 1.78	0.000	0.000
Ptm	0.91 ± 0.86	1.62 ± 0.98	0.000	0.000	0.64 ± 0.43	2.39 ± 1.33	0.000	0.000	0.85 ± 1.32	3.60 ± 1.85	0.000	0.000
A	1.70 ± 0.93	1.47 ± 0.88	0.226	0.228	1.06 ± 0.72	1.88 ± 1.05	0.000	0.000	1.30 ± 0.59	1.56 ± 0.79	0.080	0.075
U1	0.82 ± 0.58	0.73 ± 0.58	0.460	0.460	0.50 ± 0.70	0.50 ± 0.59	1.000	1.000	1.00 ± 0.66	0.67 ± 0.65	0.018	0.018
U1A	2.39 ± 1.05	1.71 ± 1.11	0.003	0.003	0.80 ± 0.48	1.68 ± 1.19	0.000	0.000	1.31 ± 0.82	2.35 ± 1.09	0.000	0.000
L1	1.17 ± 0.67	0.94 ± 0.36	0.040	0.049	0.74 ± 0.79	0.62 ± 0.52	0.386	0.399	0.61 ± 0.41	0.87 ± 0.54	0.012	0.010
L1A	1.60 ± 0.81	1.58 ± 1.14	0.924	0.922	0.85 ± 0.94	2.08 ± 1.45	0.000	0.000	0.73 ± 0.53	1.41 ± 0.78	0.000	0.000
B	1.34 ± 0.65	1.25 ± 0.66	0.513	0.512	0.70 ± 0.68	1.55 ± 0.89	0.000	0.000	0.96 ± 0.54	1.47 ± 0.93	0.002	0.002
Pog	0.81 ± 0.47	1.29 ± 0.80	0.001	0.001	0.57 ± 0.56	1.16 ± 0.74	0.000	0.000	0.50 ± 0.28	2.19 ± 1.00	0.000	0.000
Gn	0.80 ± 0.74	1.15 ± 0.66	0.019	0.020	0.50 ± 0.60	0.74 ± 0.46	0.033	0.036	0.43 ± 0.29	1.37 ± 0.72	0.000	0.000
Me	0.79 ± 0.62	2.36 ± 1.30	0.000	0.000	0.62 ± 0.56	1.79 ± 1.13	0.000	0.000	0.47 ± 0.28	2.22 ± 1.20	0.000	0.000
Go	1.87 ± 1.38	2.16 ± 1.16	0.276	0.282	1.20 ± 0.96	2.09 ± 1.28	0.000	0.000	1.04 ± 1.71	2.89 ± 1.67	0.000	0.000
Ar	0.95 ± 0.63	1.45 ± 1.26	0.021	0.017	1.19 ± 0.87	1.68 ± 1.26	0.035	0.031	0.86 ± 0.54	1.89 ± 1.27	0.000	0.000
Co	1.18 ± 0.80	2.09 ± 1.03	0.000	0.000	1.10 ± 0.67	1.91 ± 0.93	0.000	0.000	1.38 ± 0.74	1.94 ± 1.11	0.006	0.005
Average	1.11 ± 0.48	1.66 ± 1.03	0.002	0.001	0.80 ± 0.26	1.80 ± 1.15	0.000	0.000	0.97 ± 0.51	1.98 ± 1.17	0.000	0.000

p₁, Significance with Equal variances assumed, analyzed by Summary Independent-Samples T Test

p₂, Significance with Equal variances not assumed, analyzed by Summary Independent-Samples T Test

In addition, we found that among the 31 landmark measurements, the detection of subnasale (Sn) showed the highest SDR in the Angelalign dataset, while the detection of menton of soft tissue (Memark) showed the lowest SDR. In the MyOrthoX dataset, sella (S) showed the highest SDR, and ptergoid (Pt) showed the lowest SDR. In the Digident dataset, sella (S) showed the highest SDR, and pronasale (Prn) showed the lowest SDR, as shown in Table 2.

Table 2.

All the landmarks successful detection rate (SDR) within 1.0, 1.5 and 2.0 mm in MyOrthoX, Angelalign, and Digident

Landmark	MyOrthoX SDR			Angelalign SDR			Digident SDR
	< 1.0 mm	< 1.5 mm	< 2.0 mm	< 1.0 mm	< 1.5 mm	< 2.0 mm	< 1.0 mm	< 1.5 mm	< 2.0 mm
G	0.20	0.29	0.37	0.16	0.22	0.35	0.27	0.37	0.55
Prn	0.67	0.84	0.92	0.00	0.08	0.16	0.00	0.00	0.02
Sn	0.63	0.94	0.98	1.00	1.00	1.00	0.73	0.96	1.00
Ls	0.16	0.31	0.51	0.22	0.43	0.61	0.02	0.02	0.10
Li	0.31	0.49	0.69	0.61	0.84	0.96	0.67	0.84	0.94
Si	0.61	0.78	0.92	0.63	0.90	0.98	0.61	0.76	0.90
Pog’	0.35	0.47	0.63	0.35	0.47	0.53	0.14	0.16	0.31
Gn’	0.20	0.27	0.39	0.24	0.29	0.39	0.27	0.35	0.49
Me’	0.29	0.49	0.65	0.04	0.06	0.10	0.14	0.33	0.41
C	0.10	0.35	0.49	0.12	0.31	0.39	0.18	0.39	0.41
Ba	0.22	0.41	0.63	0.18	0.27	0.41	0.18	0.47	0.53
S	0.86	0.98	1.00	0.94	1.00	1.00	0.84	1.00	1.00
N	0.63	0.82	0.94	0.71	0.88	0.90	0.57	0.82	0.88
Po	0.39	0.63	0.84	0.43	0.59	0.71	0.18	0.45	0.55
Or	0.24	0.51	0.71	0.16	0.31	0.53	0.29	0.51	0.71
ANS	0.29	0.43	0.63	0.49	0.69	0.84	0.35	0.53	0.71
PNS	0.43	0.71	0.90	0.57	0.78	0.94	0.47	0.69	0.94
Pt	0.12	0.29	0.39	0.18	0.27	0.41	0.16	0.29	0.41
Ptm	0.33	0.49	0.69	0.18	0.33	0.41	0.06	0.12	0.22
A	0.33	0.53	0.73	0.24	0.39	0.59	0.33	0.55	0.69
U1	0.78	0.96	0.96	0.92	0.94	0.96	0.90	0.92	0.98
U1A	0.27	0.43	0.65	0.39	0.51	0.65	0.08	0.20	0.43
L1	0.57	0.94	1.00	0.86	0.88	0.98	0.69	0.90	0.94
L1A	0.35	0.57	0.76	0.29	0.43	0.55	0.35	0.59	0.76
B	0.43	0.69	0.90	0.29	0.47	0.69	0.39	0.53	0.65
Pog	0.37	0.61	0.84	0.47	0.73	0.9	0.06	0.29	0.49
Gn	0.45	0.76	0.86	0.78	0.94	0.96	0.29	0.69	0.86
Me	0.08	0.27	0.39	0.27	0.47	0.67	0.14	0.24	0.49
Go	0.20	0.39	0.51	0.24	0.39	0.55	0.06	0.24	0.37
Ar	0.37	0.67	0.78	0.31	0.59	0.78	0.22	0.53	0.65
Co	0.16	0.31	0.53	0.14	0.43	0.59	0.18	0.37	0.61
Average	0.37	0.57	0.72	0.40	0.54	0.66	0.32	0.49	0.61

Furthermore, we conducted statistical analysis to the validate the credibility of the original study. Due to the inability to acquire the raw data of landmark localization errors from the original publication ([1] 10.1186/s12903-023-03188-4), we were constrained to conducting an inferential statistical analysis using the Summary Independent-Samples T Test. The aforementioned publication included 43 samples, whereas our initial dataset encompassed 55 samples, with 6 samples being subsequently excluded due to their failure to be accurately recognized by the neural network system for scale calibration.

As depicted in Table 1, we have presented the significance test results under the assumption of equal and unequal variances. In terms of the outcomes, a significant statistical difference was observed in 21 out of 31 (67.74%) landmarks in MyorthoX, 25 out of 31 (80.65%) landmarks in Angelalign, and 27 out of 31 (87.10%) landmarks in Digident, with a p-value less than 0.05. Considering that the commercially available software we used is stable, and the lateral cephalometric radiographs we selected for testing are commonly seen in clinical practice, sourced from multiple centers, and have clear images, such significant differences in results should not have occurred. Even if there are no methodological issues with the original publication, the significant differences in our replication experiment cannot lead to the conclusion that “AI assistance may increase efficiency without compromising accuracy with cephalometric tracings in routine clinical practice and research settings.”

It is evident that the error rates of most landmarks reported among the three systems in Ye et al.’s paper are significantly lower than those we observed.

In clinical practice, cephalometric landmarks are finally used for conducting cephalometric measurements. In order to further verify the credibility of AI system in clinical applications, we compared 11 measurements errors in the three AI systems, as shown in Table 3.

Table 3.

Mean error and standard derivation (SD) for the 11 Measurement index measured by MyOrthoX, Angelalign, and Digident compared with manual standard

Measurements	MyOrthoX	Angelalign	Digident
SNA (°)	1.23 + 091	1.17 ± 091	1.23 ± 0.84
SNB (°)	0.91 ± 0.68	0.84 ± 0.76	0.94 ± 0.73
ANB (°)	0.64 ± 0.52	0.55 ± 0.46	0.55 ± 0.44
U1-NA (mm)	1.05 ± 0.90	1.04 ± 1.05	0.90 ± 0.79
L1-NB (mm)	0.57 ± 0.35	0.44 ± 0.39	0.55 ± 0.54
U1-SN (mm)	3.27 ± 2.26	2.05 ± 1.77	2.94 ± 2.03
UL-EP (mm)	0.38 ± 0.30	0.26 ± 0.19	0.83 ± 0.52
LL-EP (mm)	0.81 ± 0.47	0.27 ± 0.27	0.48 ± 0.45
MP-SN (°)	1.27 ± 0.94	1.13 ± 0.83	1.38 ± 1.09
FMA (°)	1.51 ± 0.94	1.39 ± 1.25	1.37 ± 1.12
IMPA (°)	2.32 ± 1.62	1.99 ± 1.51	2.08 ± 1.56
Average	1.27 ± 1.33	1.01 ± 1.14	1.20 ± 1.25

Among these systems, the automatic measurements calculation in the Angelalign system showed the highest accuracy (SDR within 2 mm/degree reached 86%), while the accuracy of MyOrthoX (SDR within 2 mm/degree was 81%) and Digident (SDR within 2 mm/degree was 82%) were relatively similar (Table 4).

Table 4.

Measurement calculation results in terms of successful detection rate (SDR) within 1.0, 2.0 and above 2.0 mm / degree (°) in MyOrthoX, Angelalign, and Digident

Measurements	MyOrthoX SDR			Angelalign SDR			Digident SDR
	<1	≤2	>2	<1	≤2	>2	<1	≤2	>2
SNA (°)	0.49	0.86	0.14	0.59	0.84	0.16	0.51	0.84	0.16
SNB (°)	0.63	0.92	0.08	0.65	0.92	0.08	0.53	0.86	0.14
ANB (°)	0.69	1.00	0.00	0.84	1.00	0.00	0.86	0.98	0.02
U1-NA (mm)	0.57	0.86	0.14	0.57	0.86	0.14	0.63	0.94	0.06
L1-NB (mm)	0.90	1.00	0.00	0.90	1.00	0.00	0.80	1.00	0.00
U1-SN (mm)	0.22	0.35	0.65	0.35	0.63	0.37	0.14	0.39	0.61
UL-EP (mm)	0.96	1.00	0.00	1.00	1.00	0.00	0.61	0.98	0.02
LL-EP (mm)	0.67	1.00	0.00	0.94	1.00	0.00	0.90	1.00	0.00
MP-SN (°)	0.39	0.78	0.22	0.49	0.86	0.14	0.47	0.69	0.31
FMA (°)	0.31	0.69	0.31	0.49	0.73	0.27	0.45	0.80	0.20
IMPA (°)	0.27	0.47	0.53	0.33	0.63	0.37	0.29	0.59	0.41
Average	0.55	0.81	0.19	0.65	0.86	0.14	0.56	0.82	0.18

Comparison of manual and automated landmarking results

The article lacks a comparative presentation of manual and automated landmarking results. To more clearly illustrate your research results, comparative data or images of manual and automatic point-making results are essential, so that readers can fully understand your research findings. In addition, from the three pictures provided in Figs 1 and 2, it can be seen that there are significant differences in resolution and landmark dimensions. MyOrthoX's cephalometric image shows obvious poor positioning of the mentum, Angelalign's nasal tip positioning is seriously erroneous, and Digident's has no obvious errors and performs best. However, this is clearly not consistent with the conclusion given in the article.

Fig. 1.

[From original study https://doi.org/10.1186/s12903-023-03188-4] The anatomical landmarks used in that challenge. All landmarks are defined and explained

Fig. 2.

[From original study https://doi.org/10.1186/s12903-023-03188-4] Cephalometric tracing of anatomical structures in three AI-assisted programs. Sample lateral cephalometric radiograph with a 30-mm ruler uploaded to the MyOrthoX, Angelalign, and Digident programs.The automated detection group: the cephalograms were uploaded to three commercial software (MyOrthoX, Angelalign, and Digident),The system automatically identified the landmarks

Clarification of data source

The software used for manual landmarking of the 43 samples, which serves as a benchmark, is not specified. Software and parameter settings used in data analysis and image processing can have a direct impact on the results. Therefore, to ensure reproducibility and accuracy of the research, please provide detailed information about the manual point-making method, including the name, version, and specific parameter settings of the software used.

The origin of the data from the 43 samples is not clearly stated. It is unclear whether the samples collected from multiple or a single imaging equipment, which affected the image pixel resolution and AI performance. In Fig. 2, the resolutions between samples may be different, with the landmark points in MyorthoX being significantly larger than that in Angelalign and Digident, which may have an important impact on the accuracy and reliability of the data analysis. Therefore, it is crucial to clarify the source and shooting conditions of the sample data for assessing the reliability of the research results.

Authors’ contributions

Jiangwei Liao: Conceptualization; data curation; formal analysis; investigation; visualization; writing-original draft preparation; writing-reviewing and editing. Jinghong Yu: Data curation; formal analysis; methodology; software; visualization. Fulin Jiang: Conceptualization; methodology; funding acquisition; resources; project administration; writing-original draft preparation; writing-reviewing and editing. All authors gave their final approval and agreed to be accountable for all aspects of the work.

Funding

Science and Technology Research Program of Chongqing Municipal Education Commission (KJZD-K202400104), Chongqing Wanzhou District PhD Direct-Acess Research Project (grant number wzstc20230409).

Data availability

The datasets generated and analyzed during the current study are not publicly available due to data sensitivity but are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study was approved by the Ethics Review Committees of Chongqing University Affiliated Three Gorges Hospital (Approval Nos. CQTGHIRB-2024–153). The investigation was conducted in strict compliance with the pertinent guidelines and regulations. All procedures were conducted in accordance with approved guidelines, confirming that informed consent was obtained from all patients or their legal guardians.

Consent to publication

Not applicable.

Competing interests

The authors declare no competing interests.