Validation design and experimental provenance shape apparent machine-learning transferability in legacy Mössbauer spectroscopy

Abstract

Machine-learning models in chemistry are increasingly trained on values compiled from many published papers and are usually validated by splitting the records at random or by publication; the latter assumes that each publication is an independent experiment. We tested this assumption in a legacy ⁵⁷Fe Mössbauer corpus. All 45 source publications were re-examined against their original tables and figures, and papers reporting the same alloy composition with the same fitted spectral areas were identified and checked against the sources. Eleven publications reported measurements on three shared specimen series, so the corpus formed 37 experimental-provenance groups containing 171 model-eligible sample states. The target was an operational high-field ferromagnetic spectral fraction, not a martensite phase fraction. At a matched test-set size, median R² fell from 0.670 under random state-level splitting to 0.322 when publications were held out and 0.287 when provenance groups were held out. With the test states held fixed, retaining related publications in training was associated with a median reduction in apparent error of 6.72 percentage points, which varied strongly between clusters and partly reflects the larger training set. The flexible chemistry-and-temperature model reached a pooled R² of 0.422 against 0.354 for a calibrated martensite-start comparator but had the lower error in only 17 of 37 groups (group-balanced difference in mean absolute error −1.73 percentage points; 95% CI −7.87 to +4.52). Transferability improved as the evidence base expanded, although provenance-group count could not be isolated as the cause, and a retrospective open-access audit added only 12 model-eligible states. Thus, a publication identifier, which records the paper a value came from, does not guarantee experimental independence, validation design can change the apparent R² by more than a factor of two, and a pooled advantage need not hold when each experiment is weighted equally. We provide practical recommendations for machine learning on literature-derived chemistry data and release the curated corpus, provenance assignments and code.

Data compiled from different papers can come from the same experiment. Holding out whole experiments lowered a Mössbauer model's median R² from 0.670 to 0.287; despite a higher pooled R², it outperformed a baseline in only 17 of 37 experiments.

1. Introduction

Many machine-learning studies in chemistry and materials science rely on data that were not measured by the modeling team but were compiled from tables and figures in published papers. A single model can thus draw on measurements from many separate studies. Literature-compiled datasets have supported machine-learning prediction of the martensite-start temperature in steels.^1,2 Such corpora can aggregate measurements spanning many compositions, treatments and source studies without requiring a new experimental campaign for every record. That same construction carries a cost. A corpus assembled from separate publications inherits differences in instrumentation, fitting conventions, reporting basis and specimen preparation, and it inherits whatever relationships hold between the publications themselves, relationships that are not recorded in any field of the compiled table. Older records compound the problem, since quantities may be reported in mixed units and composition bases, so a compiled table can be internally inconsistent unless each value is checked against the source it came from.

How such models are evaluated therefore determines what their reported performance means. Materials-machine-learning studies have adopted group-disjoint evaluation to test transfer beyond records sharing experimental or chemical-group structure.^3,4 More broadly, data leakage has been identified as a recurring source of optimistic performance estimates in machine-learning-based science.⁵ In materials prediction, strong benchmark performance likewise does not guarantee transfer when the evaluation distribution shifts beyond the training support.⁶ These are distinct failure modes with a common prerequisite: the variable along which records are grouped, or across which the distribution is held to shift, has to be known before the split is made. Where that variable is available (a chemical family, a database release, or a measurement campaign), the corresponding evaluation can be constructed directly.

The present study addresses a different source of dependence, one that becomes visible only after source-level reconstruction: nominally distinct publications may contain measurements originating from a shared experimental series. Where this occurs, the publication identifier is available to the analyst but does not mark experimental independence, so a split keyed on that identifier can place related measurements on opposite sides of the partition while appearing to separate them. The grouping variable that would be needed is not a field of the corpus; it has to be inferred from the sources and adjudicated against them. Reconstruction of this kind is therefore part inference and part judgment, and what it yields is a grouping variable that did not previously exist. This makes both the reconstruction and its effect on measured performance worth reporting explicitly. We examine this in a curated database of experimental ⁵⁷Fe Mössbauer measurements on Fe-based alloys, in which every source was re-opened and audited against its original tables and figures, and in which the prediction target is an operationally defined high-field ferromagnetic spectral fraction rather than a phase fraction.

Two terms are used throughout this work. A publication identifier is the field of a compiled dataset that records the paper from which a value was taken, usually its DOI. It is available for every record and requires no interpretation. An experimental provenance group is a set of records that cannot be treated as independent because the publications reporting them share experimental material, for example the same alloy melt, the same specimen series or the same reference sample. In this work, publications that report at least one common specimen state were assigned to the same group together with all of their records. Experimental provenance is not recorded in compiled datasets and has to be reconstructed from the original publications. When several papers report measurements on the same specimens, the two groupings differ, and a data split based on publication identifiers can place closely related measurements in both the training set and the test set.

This work is primarily a methodological benchmark of apparent generalization rather than a model development study. Its aims are, in order: to reconstruct experimental provenance across publications and adjudicate the resulting clusters against the sources; to compare state-random, publication-disjoint and provenance-disjoint validation of the same corpus under a matched test-set size; to assess the operational effect of cross-publication relatedness while holding the evaluated states fixed; to characterize how apparent generalization changes as the evidence base expands, and to probe whether provenance-group structure contributes beyond record count at fixed sample size; to compare a flexible model against a calibrated comparator built on a composition-derived martensite-start descriptor, under both pooled and group-balanced evaluation; and to quantify group-disjoint predictive uncertainty. The analyses, their metrics and the interpretation to be adopted under each possible outcome were specified and frozen before execution, allowing prespecified hypotheses to be retained or withdrawn according to the frozen decision rules. The curated corpus, the correction log, the provenance assignments, the frozen analysis plan and the evaluation logs are publicly available, so that every judgment reported here can be re-examined and revised by others. Practical recommendations derived from this case study are summarized in Section 4.6.

2. Methods

2.1. Dataset and analysis set

We assembled a state-resolved database of experimental ⁵⁷Fe Mössbauer measurements on Fe-based alloys from the peer-reviewed literature. The literature-derived primary corpus originated from manual, non-systematic keyword-based searching in Google Scholar and data-oriented selection of relevant publications; the exact historical keywords, search dates and screening log were not preserved. No open-access filter was applied during this assembly; the primary corpus spans 1981 to 2025 and includes subscription-only publications (27 of the 42 publications with a resolvable DOI; SI Methods S1.1). A separate retrospective structured coverage audit, restricted to open-access articles and conducted after the primary analysis was frozen, is reported in SI Methods S1 and Fig. S1. The unit of record is the sample state, defined by publication, composition, thermomechanical treatment and measurement temperature, with one or more fitted spectral components carrying hyperfine field and relative spectral area.

Every source was re-opened and audited against its original tables and figures, and accepted, corrected or excluded values were recorded in a correction log that is released with the dataset (SI Methods S1 and Table S1a–c). Two eligibility flags, defined before the locked modeling analysis, required quantitative fitted component areas or fractions and an unambiguous composition basis that could be represented in wt%; one further state was excluded for conversion-electron measurement geometry, which probes the surface rather than the bulk. The primary analysis set comprises 171 sample states from 45 source publications.

All eligibility decisions and provenance assignments were made by the authors through direct examination of the source publications, according to the rules described in SI Methods S1–S3. This applies equally to the conservative chemistry criterion used in the retrospective audit; the 12 retrospective states that met this criterion are listed individually with their source evidence in SI Table S1d and e. The machine-learning models described in Section 2.4 were applied only to the frozen dataset, and generative-AI tools were not used to make eligibility or provenance decisions (Section 6.6).

2.2. Endpoint definition

In the present source audit, magnetic ordering and hyperfine-field magnitude did not identify phase uniquely: high-field magnetic components were attributable to non-martensitic phases, and austenite components in high-Mn alloys carried small hyperfine fields, so that a spectroscopic “total magnetic fraction” can increase when austenite, not martensite, increases. Antiferromagnetic ordering with small Fe hyperfine fields has previously been reported in high-Mn Fe–Mn–Si austenitic alloys.⁷

We therefore did not define the endpoint as a phase fraction. The primary endpoint is the high-field ferromagnetic fraction: the relative spectral area of magnetically split components with hyperfine field above 10 T, as a percentage of total fitted area. It is an operationally defined spectroscopic quantity and is named as such throughout; threshold sensitivity was evaluated at 5, 10 and 15 T (Fig. S2). Morphology descriptors were not used to assign component phase identity, because that field is state-level rather than component-level in a substantial minority of multi-component states. Endpoint construction, threshold sensitivity and a 49-state curated pilot of endpoint identity (Fig. S3) are detailed in SI Methods S2 and Table S2.

2.3. Experimental provenance

Within the curated corpus, distinct DOIs were not necessarily independent experiments: the same specimen series or control material was represented across multiple publications. Candidate re-use was detected by fingerprinting composition together with multisets of component areas across sources; every flagged cluster was then adjudicated against the source publications by a domain expert. Each confirmed link was supported by numerical agreement of reported values: the same nominal composition and treatment appeared in different publications with the same fitted magnetic fraction and hyperfine field (for example, 71.864% at 30.54 T for the Fe–13.4Mn–5.2Mo baseline in four publications), and in one case the later publication stated that the sample had been taken from the earlier study (SI Table S3b). Reused states were retained as separate records, as they appear in each publication; because all records of a reused state belong to the same provenance group, they are always held out together in provenance-disjoint evaluation. Confirmed clusters were merged, reducing 45 source identifiers to 37 provenance groups; three clusters span more than one publication, together covering 11 publications and 29 states (Fig. S4). Grouped cross-validation uses the provenance group throughout, except where the publication-level regime is examined deliberately (Section 2.6). Source-to-provenance assignments are detailed in SI Methods S3 and Table S3a. The primary-corpus construction, reconstructed provenance, and validation regimes are summarized schematically in Fig. 1.

Fig. 1. Primary-corpus construction, reconstructed experimental provenance, and validation design. Panel (a) summarizes the path from 45 source publications through source-level audit, correction logging, and eligibility filtering to the frozen primary set of 171 eligible states in 37 provenance groups. A provenance group is a set of records judged to share an experimental origin; panel (b) shows the three multi-publication provenance clusters (11 publications, 29 states) alongside 34 singleton publication groups, illustrating that publication identity and experimental independence are not equivalent. Panel (c) schematically contrasts state-random splitting (individual states may be separated), publication-disjoint splitting (publications are separated but a reconstructed provenance group can still span training and test data), and provenance-disjoint splitting (entire provenance groups are held out). Filled blue markers denote training states and open red markers denote test states; the schematic encodes validation logic rather than sample counts.

2.4. Features, models and baselines

The primary feature set comprises 20 elemental concentrations expressed in wt% and the measurement temperature; a secondary set adds the reconstructed thermomechanical history. The locked candidate grid comprised 22 configurations spanning ExtraTrees, RandomForest, k-nearest neighbors, ridge regression and CatBoost;^8–12 predictions are clipped to [0, 100]%. Implementations used the scikit-learn ecosystem where applicable.¹³ Chemistry-and-temperature was fixed as the reference representation before the generalization analyses ran, while the process representation was retained as a secondary locked comparison (SI Methods S4 and Table S4).

Two metallurgy-informed benchmarks were retained. B1 combines the linear six-element form of the Andrews¹⁴ expression for the martensite-start temperature, with concentrations in wt%, and Koistinen–Marburger kinetics¹⁵ using the conventional coefficient α = 0.011 K⁻¹, with no parameters fitted here. These relations were treated as empirical benchmarks rather than ground-truth laws: Koistinen–Marburger was formulated for pure Fe–C and plain-carbon steels, while the Andrews descriptor is extrapolated well beyond conventional steel compositions. B2 maps the composition-derived Ms descriptor to the endpoint by monotonicity-constrained isotonic calibration¹⁶ fitted only on outer training groups. B2 is not a physics model; it tests whether flexible models add information beyond composition-derived Ms.

2.5. Nested group-disjoint validation

The main analysis uses leave-one-provenance-group-out cross-validation. Within each outer training set, model family and hyperparameters are selected jointly by grouped inner cross-validation on the same provenance variable, using mean inner-fold mean absolute error. The held-out group is never seen during selection, imputation, calibration or fitting.

2.6. Frozen generalization analyses

Five generalization analyses, their metrics, and the interpretation to be adopted under each possible outcome were specified and frozen before execution. Exact algorithms and the prespecified interpretation limits are given in SI Methods S5 and Table S5.

2.6.1. E1: validation-regime comparison

E1 compared state-random, publication-disjoint and provenance-disjoint workflows over 40 replicates at a matched test-set size of 34 states, applying each independence assumption at both the outer and the inner level.

2.6.2. E1b: fixed-test-state cluster exclusion

E1b held test states fixed while training excluded either the target publication alone or its whole provenance cluster. Because whole-cluster exclusion also reduces training-set size, the contrast is interpreted as an operational effect rather than a causal leakage magnitude.

2.6.3. E2: evidence-base scaling

E2 evaluated evidence bases at nine levels from G = 6 to G = 37: 200 unique subsets per level below G = 37 and the single full-data subset at G = 37. One reference configuration was fixed beforehand as the modal inner selection of the locked main analysis and was therefore not chosen using E2 outcomes.

2.6.4. E3: matched-record-count sensitivity

E3 held state counts of 30, 35 and 40 fixed while probing provenance-group structure, recording group count, alloy-family count, endpoint moments and chemical dispersion per subset because the pools are not drawn from the same region of composition space.

2.6.5. E4: nested-selection fidelity

E4 repeated selected E2 levels under the full nested selection protocol to test whether the evidence-base trend survived model reselection.

2.7. Statistical analysis

Pooled performance is reported as R² and mean absolute error over out-of-fold predictions. Because pooled metrics weight every state equally and therefore give greater influence to larger provenance groups, the primary model-versus-baseline comparison is the provenance-group-balanced (hereafter group-balanced) difference in mean absolute error, in which each provenance group contributes equally. Statistical procedures are detailed in SI Methods S6.

2.7.1. Group-balanced comparisons and descriptive subset summaries

The group-balanced difference is summarized by its mean, a 95% confidence interval from 10 000 bootstrap resamples of provenance groups,¹⁷ the fraction of groups in which the model has the lower error, and a Wilcoxon signed-rank test¹⁸ over the paired group means. Distributions across sampled subsets or replicates are reported as medians, interquartile ranges and 5th–95th percentiles; because sampled subsets share provenance groups extensively, these are descriptive statistics of the subset space, not probabilities or significance tests. The secondary feature-set comparison uses the same group-balanced statistic, taken as process minus chemistry-and-temperature.

2.7.2. Predictive uncertainty

Distribution-free prediction intervals were obtained by split conformal prediction¹⁹ targeting 90% coverage over 50 repetitions, with the calibration set drawn from provenance groups disjoint from both training and test groups.

2.8. Reproducibility

Because the outer and inner grouping variables must be interchangeable across regimes, the cross-validation driver for the generalization analyses re-implements the main pipeline's logic; its equivalence is therefore verified numerically rather than asserted, by reproducing the locked main-analysis result before any analysis may run and aborting unless agreement is within 10⁻⁶. A run manifest records the analysis identifier, dataset and candidate-grid hashes, random seed and ensemble sizes. Finalized CSV result tables are the canonical evaluation records. Append-only JSONL files were used as restart/checkpoint traces; resumed execution can leave duplicate keys, and the available traces are not guaranteed to contain one row for every finalized evaluation. Their reconciliation against the canonical CSVs is documented in SI Methods S7 and Table S6. Curated data, correction log, provenance assignments, code, canonical result tables and raw checkpoint traces are deposited at https://doi.org/10.5281/zenodo.21997560.

2.9. Sensitivity analyses outside the frozen plan

Two analyses were not part of the frozen analysis plan and are reported as sensitivity analyses. First, the fixed-test-state summary of E1b was recomputed for every combination of the three multi-publication provenance clusters. Each combination approximates the result that would have been obtained had only some of the cross-publication links been identified. Because each E1b contrast is formed within a single cluster, this recomputation uses the released per-publication evaluation records and requires no additional model fitting (SI Methods S3.1 and Table S3c). Second, the recall of the retrospective open-access search was stratified by publication year, using the 47 DOI-bearing publications of the audited corpus as a reference set of known relevant publications (SI Table S1f).

3. Results

3.1. Protocol reproduction

The generalization layer reproduced the locked main analysis exactly (R² = 0.4221969136, MAE = 20.89356619 percentage points). Sections 3.2–3.6 were therefore produced by machinery numerically equivalent to the main pipeline. The modal configuration was the same RandomForest specification in 28 of 37 outer folds (Fig. S5).

3.2. Validation regime strongly changes apparent generalization

Across 40 replicates at an identical test-set size of 34 states, the median estimated performance decreased monotonically as the independence assumption was tightened (Fig. 2). Median R² was 0.670 under state-random evaluation, 0.322 under publication-disjoint evaluation and 0.287 under provenance-disjoint evaluation; median MAE rose correspondingly from 13.78 to 21.99 to 23.83 percentage points.

Fig. 2. Validation-regime distributions across 40 matched-size replicates. Panel (a) shows the coefficient of determination (R²) and panel (b) shows mean absolute error (MAE, percentage points). State-random, publication-disjoint, and provenance-disjoint workflows apply their stated independence rule at both outer evaluation and inner model selection. Boxes show medians and interquartile ranges (IQRs), points are replicate estimates, and the numerical medians are printed beside each distribution. The arms do not share identical test states, so the contrast describes changes in estimated transferability rather than a causal leakage effect.

Replicate-wise, moving from state-random to publication-disjoint evaluation increased MAE by a median of 7.52 pp (IQR 4.89 to 11.00), and the stricter regime gave the larger error in 0.950 of replicates. The further step to provenance-disjoint evaluation added a smaller and less consistent 3.00 pp (IQR -2.59 to 6.26; stricter regime worse in 0.625 of replicates). Over the full contrast the median increase was 9.83 pp, with the stricter regime worse in 0.925 of replicates. Because the three regimes do not share test states and replicates overlap substantially, these are reported as descriptive distributions rather than as significance tests.

3.3. Cross-publication dependence adds heterogeneous optimism

Holding the test states fixed and varying only whether experimentally related publications remained in training provided the most controlled assessment of cross-publication dependence (Fig. 3). Across the 11 publications belonging to multi-publication provenance clusters, retaining related publications was associated with a publication-macro median reduction in apparent error of 6.72 pp for the flexible model (7.39 pp state-weighted); the corresponding optimism for the calibrated comparator was substantially smaller, 1.05 pp (2.67 pp state-weighted). The effect was positive in 9 of 11 publications but strongly cluster-dependent: median optimism was 10.93 pp in VG-PC001-PC002 (6 publications), 0.28 pp in VG-PC003 (3 publications), 21.16 pp in VG-PC004 (2 publications). Because whole-cluster exclusion also reduces the training set, this quantity is an operational effect rather than a causal leakage magnitude.

Fig. 3. Operational effect of excluding a complete reconstructed provenance cluster rather than the test publication alone. Panel (a) shows the 11 fixed-test-state publication contrasts (P1–P11), with color identifying the three multi-publication provenance clusters and filled versus hatched bars identifying the flexible model and calibrated comparator. Panel (b) summarizes each predictor by the publication-macro median and state-weighted mean. Positive values indicate lower apparent error when experimentally related publications remain in training. Whole-cluster exclusion also reduces training size, so the contrast is not interpreted as a causal leakage magnitude.

In the sensitivity analysis described in Section 2.9, the publication-macro median optimism of the flexible model was positive for every combination of the three clusters, ranging from 0.28 pp when only VG-PC003 was included to 21.16 pp when only VG-PC004 was included, and the corresponding value for the calibrated comparator was lower in every combination (SI Table S3c). Across these seven combinations, the magnitude of the effect changed with the clusters identified, but its sign did not.

3.4. Transferability improves as the evidence base expands

Using a fixed reference configuration, performance increased monotonically with the size of the training evidence base (Fig. 4). Median R² rose from −0.332 at G = 6 (median 27 states) to 0.390 at G = 20 (92.5 states) and 0.513 at G = 37 (171 states). The fraction of sampled subsets with positive R² increased from 0.235 at G = 6 to 0.915 at G = 16; all 200 sampled subsets at G = 25 and G = 30 had positive R², and the single full-data subset at G = 37 was also positive. The group-balanced difference against the calibrated baseline changed sign between G = 20 (0.52 pp) and G = 25 (−1.52 pp).

Fig. 4. Evidence-base scaling and nested-selection fidelity. For the prespecified fixed-reference analysis, circles show median R², the darker band shows the interquartile range (IQR), and the lighter band shows the 5th–95th percentile range. The upper axis reports the median record count n at each provenance-group count G. Diamonds show nested-selection medians with IQRs at selected G values; they are displaced by +0.25 on the horizontal axis solely for visual separation. The trend direction is reproduced, whereas the model-versus-comparator crossover is not.

Under the full nested selection protocol the direction was reproduced but the crossover was not: median R² rose from −0.476 at G = 6 to −0.222 at G = 12 and 0.367 at G = 25, yet the group-balanced difference against the baseline remained positive at G = 25 (1.35 pp) where the fixed-reference curve had already crossed. Fixed-reference and nested curves therefore agree on trend but not on the location of the crossover, and no single crossover point is claimed.

3.5. Matched-n analysis does not isolate provenance-group count

At fixed record count, the matched-n analysis did not support a beneficial effect of larger provenance-group count (Fig. 5 and Table 1). At every record count examined, subsets built from few large provenance groups did not perform worse than subsets built from many small ones; the raw comparison ran in the opposite direction to the hypothesis:

Fig. 5. Matched-record-count sensitivity analysis. Median R² and interquartile ranges are shown for subsets assembled from fewer larger provenance groups and from more smaller provenance groups at n = 30, 35, and 40. Annotations give the median number of alloy families represented in each arm. Because alloy-family and composition-space coverage differ together with provenance-group structure, this analysis probes but does not isolate an independent causal effect of group count.

Table 1. Matched-record-count sensitivity analysis. Values are median provenance-group count G, median R², and group-balanced model-minus-comparator ΔMAE (percentage points, pp) within each prespecified pool.

n	Fewer, larger groups	More, smaller groups
30	G = 3.0; R2 = −0.221; ΔMAE = 7.24 pp	G = 14.0; R2 = −0.238; ΔMAE = 9.99 pp
35	G = 4.5; R2 = 0.445; ΔMAE = −9.20 pp	G = 16.0; R2 = −0.248; ΔMAE = 9.49 pp
40	G = 4.0; R2 = 0.349; ΔMAE = 5.00 pp	G = 18.0; R2 = −0.260; ΔMAE = 11.66 pp

The reverse conclusion is equally unsupported, because the two pools differ in composition space as well as in group structure: median alloy-family count was 5 in the few-large pool against 13 in the many-small pool at n = 35, with comparable differences at the other record counts. The prespecified hypothesis that the number of independent provenance groups drives generalization is therefore not supported; and because provenance-group structure remained entangled with alloy-family and composition-space coverage, the analysis did not isolate an independent effect of group count.

3.6. A pooled advantage does not become group-balanced superiority

On the full dataset the flexible model achieved a higher pooled R² than the calibrated comparator (0.422 against 0.354; MAE 20.89 against 24.25 pp). Superiority was not established under group-balanced evaluation, however: the model achieved the lower group-level MAE in only 17 of 37 provenance groups, the paired group-balanced difference was −1.73 pp with a 95% confidence interval spanning zero (−7.87 to 4.52 pp), and the Wilcoxon signed-rank test over the paired provenance-group errors was not significant (p = 0.51) (Fig. 6).

Fig. 6. Group-balanced mean absolute error (MAE) contrasts on frozen out-of-fold predictions. Points are means over 37 provenance groups and error bars are 95% percentile-bootstrap confidence intervals (CIs). Values are in percentage points (pp); negative values favor the first-named model or feature representation. The zero line marks no group-balanced difference. All three intervals span zero, including the primary chemistry-and-temperature model versus the calibrated B2 comparator.

3.7. Secondary feature-set and uncertainty analyses

The quantities in this section come from the locked main analysis rather than from the generalization layer (Fig. 6 and 7). Adding reconstructed process features improved pooled high-field performance from R² = 0.422 and MAE = 20.89 pp to R² = 0.550 and MAE = 17.78 pp. This pooled improvement did not translate into a clearly established group-balanced increment: the mean of the 37 per-group MAE differences (process minus chemistry-and-temperature) was −1.44 pp (95% bootstrap confidence interval −4.79 to 1.44 pp), with process features giving the lower error in 19 of 37 groups (Wilcoxon p = 0.60). Under the process representation the flexible model likewise did not establish group-balanced superiority over the calibrated comparator: mean paired difference −3.16 pp (95% confidence interval −9.25 to 2.88 pp), lower group MAE in 21 of 37 groups, Wilcoxon p = 0.24. Across 50 provenance-disjoint split-conformal repetitions targeting 90% coverage, mean coverage was 90.2% and median coverage 95.6%, but the median repetition-level interval width was 92.1 pp on the 0–100 pp endpoint scale, and coverage ranged from 22.7% to 100%.

Fig. 7. Provenance-disjoint split-conformal uncertainty across 50 repetitions targeting 90% marginal coverage. Each point shows observed test coverage and the repetition-level median prediction-interval width. The dashed horizontal line marks nominal 90% coverage and the dotted vertical line marks the median interval width (92.1 percentage points, pp). The broad widths relative to the 0–100 pp endpoint scale limit support for precise specimen-level prediction despite near-nominal mean coverage.

3.8. Prespecified hypotheses not advanced

Two decision rules fixed before execution were triggered. The hypothesis that model ranking reverses when the spectroscopic endpoint is replaced by a source-curated α′ endpoint could not be tested: in the curated subset the three endpoint definitions coincide numerically for every state, so the intended contrast does not exist there. The hypothesis that provenance-group count drives generalization was not supported (Section 3.5). Neither is advanced (SI Note S1).

4. Discussion

4.1. Validation design and apparent generalization

Several recent Ms-prediction studies based on literature-compiled steel data have evaluated models using ordinary record-level train–test partitions.^1,2 That convention is reasonable when records are exchangeable; our results indicate that in a literature-derived corpus they may not be. Holding the test-set size fixed while changing the independence assumption encoded by the validation workflow, the median coefficient of determination fell from 0.670 under state-random evaluation to 0.287 when whole provenance groups were held out, with the stricter regime worse in more than nine of ten replicates, rather than being an artifact of a single partition.

Group-disjoint validation has previously been used in materials machine learning both to assess extrapolation to unseen chemical families⁴ and to prevent leakage from records sharing an experimental origin.³ These grouping constructs are not equivalent to the experimental-provenance groups reconstructed here, which partition an origin that had to be inferred across publications. Data leakage is more broadly a recognized cross-disciplinary source of overoptimistic performance estimates in machine-learning-based science;⁵ to our knowledge, the particular dependence examined here has received much less attention in literature-mined materials corpora.

Two boundaries deserve emphasis. The three regimes do not share test states, so the comparison describes how an estimate changes rather than isolating a causal quantity; and the difference between random and grouped evaluation reflects what is being estimated (transfer to a new record versus transfer to a new experiment), not evidence that any published model is wrong.

4.2. Publication identity and experimental dependence

In our curated corpus, distinct publications were not always experimentally independent: 11 publications were linked into three multi-publication provenance clusters after source-level adjudication. Holding the test states fixed and varying only whether related publications remained in training, retaining them was associated with a publication-macro median reduction in apparent error of 6.72 percentage points for the flexible model, against 1.05 percentage points for the calibrated comparator. The effect was positive in nine of eleven publications but far from uniform, and near zero in one of the three clusters. Because excluding a whole cluster also reduces the training set, this is an operational effect of the exclusion rule rather than a measured leakage magnitude.

The logic is analogous to subject-wise partitioning in biomedical machine learning, where repeated measurements from one patient are kept within a single partition.²⁰ The analogy has a clear limit: experimental provenance in a literature corpus is less explicit than subject identity and had to be reconstructed and adjudicated rather than read from a known identifier. A publication identifier is available to anyone; a provenance assignment is a judgment, which is why ours is released with the dataset.

Because provenance reconstruction combines automated detection with expert confirmation, the dependence of the results on curator judgment requires explicit assessment. Domain knowledge entered the analysis at three points, and each was constrained to limit its influence. First, the endpoint was defined from measured quantities, namely fitted spectral areas and hyperfine fields, rather than from phase assignments, so it does not depend on how a curator interprets individual spectral components; model performance was also nearly unchanged when the field threshold was moved from 10 T to 5 T or 15 T (SI Table S2). Second, every correction made during the source audit was recorded together with the original value, the corrected value, the reason and the location of the supporting evidence in the source, so that each curation decision can be checked independently. Third, candidate links between publications were detected automatically, and three of the four confirmed links rest on exact numerical agreement of reported values or on an explicit statement by the source authors (Section 2.3; SI Table S3b).

The influence of the remaining judgment can be bounded with the analyses reported here. The largest change in apparent transferability does not depend on provenance knowledge at all: moving from state-random to publication-disjoint evaluation lowered the median R² from 0.670 to 0.322, and the complete reconstruction lowered it further to 0.287. The contribution of the reconstruction itself is isolated in the fixed-test-state analysis, in which the test states are identical in both conditions. There, the effect of retaining related publications was positive for every combination of the three clusters, including the combination without the least distinctive link (Section 3.3; SI Table S3c). Missing a true provenance link can allow experimentally related records to cross the training–test boundary and can therefore bias apparent transferability upward. Conversely, merging records that are genuinely independent can unnecessarily reduce the available training information and may bias the estimate downward. The magnitude and direction of these effects are not guaranteed in every finite dataset. In the seven reconstructions examined here, however, curator choice changed the magnitude of the fixed-test-state effect but not its sign. We therefore restrict our inference to the reconstructions explicitly tested rather than extrapolating beyond them. All provenance assignments and their supporting evidence are released, so that alternative assignments can be tested directly with the released code.

4.3. What the evidence-base analyses do and do not show

Transferability improved substantially as the training evidence base expanded, and the direction was reproduced under the full nested selection protocol. It would be easy to read this as a sample-size law, and we cannot support that reading. A benchmark score need not imply generalization, and distribution shift between training and evaluation data is known to degrade materials-property prediction.⁶ This is a different mechanism from the dependence examined here, but it reinforces that what changes between subsets is rarely one thing.

Our matched-n analysis held record count fixed and was intended to probe whether provenance-group structure contributed beyond record count; it did not isolate an independent effect of group count. At fixed record count, subsets built from many small provenance groups did not outperform subsets built from few large ones; the raw comparison ran opposite to the prespecified hypothesis. The reverse conclusion is equally unsupported, because the two pools differ in alloy-family coverage and composition space as well as in group structure. The prespecified hypothesis is therefore not advanced and no threshold is proposed. The learning-curve and matched-n analyses are second, differently structured sensitivity analyses of the same 171 states and 37 provenance groups, not independent lines of evidence.

4.4. Flexible models and a calibrated physical descriptor

Physically motivated descriptor sets have been compared against raw composition as inputs to the same learner for martensitic transformation temperature,²¹ and empirical-equation-derived features have been fused with learned representations for Ms in steels.²² Our comparator belongs to that family but is deliberately weaker: it is not a physics model, but a monotonicity-constrained calibration of a single composition-derived Ms descriptor, fitted only on outer training groups.

The flexible model did not establish superiority over that comparator under group-balanced evaluation. With the chemistry-and-temperature representation fixed as the reference feature set for these experiments, it achieved a higher pooled coefficient of determination, yet lower group-level error in only 17 of 37 provenance groups, with a group-balanced difference in mean absolute error whose 95% confidence interval spanned zero. The same held when the process feature set was used. Pooled and group-balanced evaluation therefore returned different answers on identical predictions. Comparison with a low-dimensional, physically motivated comparator made that discrepancy visible; without it, the pooled figure alone would have supported a stronger conclusion than the data carry.

4.5. Limitations and implications

The corpus is small by materials-informatics standards (171 sample states in 37 provenance groups), and the wide confidence intervals throughout largely reflect that. Provenance reconstruction is partly expert-adjudicated: fingerprinting proposes, a domain expert decides, and the result is reproducible because the assignments are published, not because they are algorithmically determined; Section 4.2 discusses the extent to which the results depend on this judgment. The endpoint is an operational spectroscopic quantity, not a martensite phase fraction. The fixed-test-state analysis rests on 11 publications and 29 states in three clusters, and the matched-n comparison is confounded by composition-space coverage. Process features improved pooled performance, but their incremental benefit was not clearly established under group-balanced evaluation, and whether that reflects the reconstructed metadata or the underlying physics is unresolved here. Group-disjoint conformal intervals were extremely broad relative to the 0–100 percentage-point endpoint scale, which limits support for precise specimen-level prediction. The primary corpus is not a systematic sample of the ⁵⁷Fe Mössbauer literature. A retrospective open-access coverage audit identified 24 additional source-verified publications; 11 yielded 93 exact operational-endpoint states across 10 provenance groups, while only 12 states across four provenance groups also met the conservative chemistry criterion (SI Table S1d). Accordingly, the retrospective set was retained as a coverage and eligibility audit rather than treated as a second modeling cohort (SI Methods S1).

The retrospective audit was restricted to open-access, English-language research articles, and this restriction limits its coverage. Open-access availability depends strongly on publication year.²³ Of the 47 DOI-bearing publications in the audited corpus, the open-access search retrieved 3 of the 24 published before 2010 (12.5%) and 11 of the 23 published in 2010 or later (47.8%), and the 24 additional publications identified by the audit had a median publication year of 2016.5, compared with 2009 for the primary corpus (SI Table S1f). The audit therefore under-represents older literature and provides a lower bound on the relevant work that a search without these restrictions would identify. The restriction does not affect the primary analysis. The primary corpus was assembled without an open-access filter, 27 of its 42 publications with a resolvable DOI have no open-access version, and it was frozen before the audit was carried out; no retrospectively identified record was added to it.

4.6. Practical recommendations for machine learning on literature-derived chemistry data

Incomplete or inconsistent metadata and insufficient curation are recognized barriers to the reliable reuse of materials data, particularly for downstream data-driven modeling.²⁴ General recommendations for reporting machine-learning-based research are available.²⁵ Data compiled from the chemical literature raise additional issues, because the records come from sources that differ in measurement practice, reporting conventions and experimental origin. Table 2 lists the points that this study identifies as important when machine learning is applied to such data, grouped by the stage of the work at which they arise and linked to the corresponding evidence from the present corpus.

Table 2. Recommendations for applying machine learning to chemistry data compiled from the published literature, with the supporting evidence from this study.

Stage	Recommendation	Evidence in this study
Data compilation	Verify every compiled value against the original source and keep a correction log	185 corrections for 181 records during the source audit, including 38 conversions from atomic to mass percent and the exclusion of one conversion-electron measurement (Section 2.1; SI Methods S1.3)
	Record how the literature was searched, including any filters, and assess what the search could not reach	The original search log was not preserved (Section 2.1); the retrospective audit was therefore fully documented, and its open-access recall was 12.5% for publications before 2010 and 47.8% for later publications (Section 4.5; SI Table S1f)
	State who made eligibility and grouping decisions and which rules were applied	All decisions made by the authors from the sources under stated rules; retrospective states documented individually (Section 2.1; SI Table S1d and e)
Target definition	Define the target from measured quantities and test its sensitivity to any threshold	Operational high-field fraction above 10 T; R2 of 0.422, 0.422 and 0.416 at 5, 10 and 15 T (Section 2.2; SI Table S2)
Validation design	Check whether records with different identifiers originate from the same experiment, and define the unit of independence before splitting the data	45 publications formed 37 provenance groups; 11 publications belonged to three shared specimen series (Section 2.3)
	Apply the same grouping in model selection as in model evaluation, and report how performance changes across validation designs	With grouping applied at both levels, median R2 was 0.670, 0.322 and 0.287 for state-random, publication-disjoint and provenance-disjoint evaluation (Sections 2.6.1 and 3.2)
Model comparison	Include a simple, physically motivated baseline	The calibrated martensite-start comparator reached a pooled R2 of 0.354, which made it possible to test whether the flexible model (0.422) adds information (Sections 2.4 and 3.6)
	Report group-balanced metrics with confidence intervals alongside pooled metrics	The flexible model had the lower error in only 17 of 37 groups; ΔMAE −1.73 pp (95% CI −7.87 to 4.52) (Section 3.6)
Uncertainty	Report prediction intervals under the same grouping, with both coverage and width	Mean coverage of 90.2% with a median interval width of 92.1 pp on a 0 to 100 pp scale (Section 3.7)
Reporting	Specify analyses and decision rules before running them, and report hypotheses that were not supported	Two prespecified hypotheses not advanced (Section 3.8; SI Note S1)
	Release the curated data, correction log, group assignments and code	Public repository with integrity checksums (Section 6.5)

These recommendations are derived from a single corpus and are not presented as a validated protocol. They are, however, inexpensive to implement, and in this study they changed the conclusions that would otherwise have been drawn. With random state-level splits and pooled metrics alone, the same data would have indicated a median R² of 0.670 and an advantage of the flexible model over the calibrated comparator that is not supported when each provenance group is weighted equally.

5. Conclusions

This study examined how validation design and experimental provenance affect the apparent transferability of machine-learning models trained on literature-derived Mössbauer data. Four conclusions follow from the results. First, publication identity did not guarantee experimental independence. Source-level reconstruction showed that 11 of the 45 publications reported measurements on three shared specimen series, so the corpus contained 37 rather than 45 independent provenance groups. Second, apparent performance depended strongly on the validation design. At an identical test-set size, the median R² was 0.670 for random state-level splits, 0.322 for publication-level splits and 0.287 for provenance-level splits. With the test states held fixed, retaining experimentally related publications in training was associated with a median reduction in apparent error of 6.72 percentage points. This shows that publication-disjoint evaluation can remain sensitive to cross-publication relatedness, although part of the contrast also reflects the reduction in available training information when complete provenance clusters are excluded. Third, pooled and group-balanced evaluation led to different conclusions from the same predictions. The flexible model achieved a higher pooled R² than the calibrated comparator (0.422 versus 0.354), but it had the lower error in only 17 of the 37 provenance groups, and the group-balanced difference in mean absolute error was not significant (95% CI −7.87 to +4.52 percentage points). Fourth, provenance-disjoint prediction intervals reached near-nominal coverage but had a median width of 92.1 percentage points on a 0 to 100 percentage-point scale, which is too wide to be useful for predicting individual specimens.

These results have direct consequences for studies that apply machine learning to data compiled from the literature. The unit of independence used for validation should be stated explicitly, and records with different identifiers should be checked for a shared experimental origin before the data are split. Group-balanced metrics and a simple, physically motivated comparator should be reported together with pooled metrics, because pooled performance alone can overstate the benefit of a flexible model. These and related points are summarized in Table 2.

Two questions remain open: which properties of an expanding evidence base improve transferability, and whether the flexible model outperforms the calibrated comparator when every experiment is weighted equally. The curated corpus, provenance assignments, frozen analysis plan and evaluation records are publicly available, so that these questions can be revisited with additional data and alternative provenance assignments can be tested directly.

Author contributions

Osman Armağan: conceptualization (lead), methodology (lead), investigation (lead), data curation (lead), validation (equal), formal analysis (equal), visualization (equal), writing – original draft (lead), writing – review & editing (equal), project administration (lead). Muhammed Emin Bedir: methodology (equal), software (lead), formal analysis (equal), validation (equal), data curation (supporting), visualization (equal), writing – original draft (supporting), writing – review & editing (equal), reproducibility curation (lead). Both authors approved the final manuscript and accept responsibility for the integrity of the work.

Conflicts of interest

The authors declare no known financial or non-financial competing interests that could have appeared to influence the work reported in this article.

Funding

Open access funding provided by the Scientific and Technological Research Council of Türkiye (TÜBİTAK).

Data availability

The data and code supporting this article, including the source-verified primary analysis dataset, frozen analysis notebooks, canonical model outputs, retrospective endpoint and coverage-audit materials, and reproducibility documentation, are openly available in Zenodo at https://doi.org/10.5281/zenodo.21997560. Raw Web of Science and Scopus database exports are not redistributed because they contain third-party database content; verified record counts, SHA-256 fingerprints, aggregate screening records, and a controlled re-execution protocol are provided in the repository.

Supplementary information (SI) is available. See DOI: https://doi.org/10.1039/d6ra08367f.