Skip to main content
[Preprint]. 2024 Oct 6:2024.03.31.587283. [Version 3] doi: 10.1101/2024.03.31.587283

Table 1:

Ablation study and aggregated benchmark results for gRNAde. We report metrics averaged over 100 test sets samples and standard deviations across 3 consistent random seeds. The percentages reported in brackets for the 3D self-consistency scores are the percentage of designed samples within the ‘designability’ threshold values (scRMSD≤2Å, scTM≥0.45, scGDT≥0.5).

Split Max. #states Model GNN Max. train length Perplexity (↓) Native seq. recovery (↑) Self-consistency metrics
2D – EternaFold scMCC (↑) scRMSD (↓) 3D – RhoFold scTM-score (↑) scGDT_TS (↑)
Single-state split 1 AR Equiv 500 1.77±0.07 0.438±0.01 0.624±0.07 13.01±1.18 (0.5%) 0.21±0.0 (14.3%) 0.22±0.0 (12.7%)
1 AR Equiv 1000 1.73±0.08 0.453±0.01 0.648±0.01 13.10±0.58 (1.0%) 0.20±0.0 (10.8%) 0.21±0.0 (10.6%)
1 AR Equiv 2500 1.41±0.01 0.513±0.01 0.633±0.03 11.76±0.91 (1.4%) 0.27±0.0 (28.8%) 0.27±0.0 (28.0%)
1 AR Equiv 5000 1.29±0.02 0.538±0.03 0.612±0.02 11.50±0.64 (1.9%) 0.28±0.0 (32.1%) 0.28±0.0 (26.2%)
1 AR, rand Equiv 5000 1.59±0.16 0.531±0.04 0.621±0.04 11.87±1.06 (1.9%) 0.26±0.0 (28.1%) 0.26±0.0 (24.1%)
1 AR Inv 5000 1.32±0.04 0.531±0.01 0.585±0.03 11.70±0.56 (1.3%) 0.26±0.0 (24.8%) 0.25±0.0 (20.1%)
1 NAR Inv 5000 1.54±0.04 0.571±0.00 0.430±0.02 14.26±0.51 (1.3%) 0.19±0.0 (15.9%) 0.18±0.0 (12.7%)
1 NAR Equiv 5000 1.46±0.06 0.584±0.00 0.473±0.02 13.04±0.88 (1.3%) 0.23±0.0 (24.0%) 0.22±0.0 (17.9%)
3 AR Equiv, DS 5000 1.23±0.05 0.539±0.01 0.620±0.01 11.47±1.05 (2.5%) 0.28±0.0 (31.4%) 0.28±0.0 (27.2%)
5 AR Equiv, DS 5000 1.25±0.01 0.539±0.02 0.596±0.03 11.90±1.00 (2.9%) 0.27±0.0 (31.6%) 0.26±0.0 (26.4%)
Groundtruth sequence prediction baseline: - 1.000±0.00 0.686±0.00 5.23±0.07 (27.9%) 0.56±0.0 (68.7%) 0.55±0.0 (68.7%)
Random sequence prediction baseline: - 0.251±0.00 0.012±0.00 24.40±0.34 (0.0%) 0.04±0.0 (0.0%) 0.02±0.0 (0.0%)
ViennaRNA 2D-only baseline: - 0.259±0.00 0.611±0.00 20.34±0.10 (0.0%) 0.07±0.0 (0.6%) 0.07±0.0 (1.1%)
Multi-state split 1 AR Equiv 5000 1.51±0.01 0.481±0.00 0.573±0.04 21.83±0.53 (0.0%) 0.12±0.0 (2.6%) 0.15±0.0 (5.5%)
3 AR Equiv, DS 500 1.87±0.04 0.444±0.01 0.587±0.02 22.09±0.13 (0.0%) 0.12±0.0 (2.3%) 0.14±0.0 (5.7%)
3 AR Equiv, DS 1000 1.76±0.04 0.455±0.03 0.504±0.04 22.92±1.43 (0.0%) 0.11±0.0 (2.3%) 0.14±0.0 (5.8%)
3 AR Equiv, DS 2500 1.54±0.07 0.500±0.01 0.543±0.01 22.00±0.26 (0.0%) 0.11±0.0 (2.9%) 0.14±0.0 (3.7%)
3 AR Equiv, DS 5000 1.44±0.04 0.531±0.00 0.573±0.03 22.19±0.28 (0.0%) 0.12±0.0 (4.2%) 0.15±0.0 (7.5%)
3 AR Equiv, DSS 5000 1.37±0.04 0.540±0.03 0.574±0.03 22.20±0.43 (0.0%) 0.12±0.0 (4.0%) 0.15±0.0 (7.5%)
5 AR Equiv, DS 5000 1.37±0.03 0.510±0.00 0.514±0.00 21.80±0.08 (0.0%) 0.12±0.0 (2.9%) 0.14±0.0 (6.2%)
1 NAR Equiv 5000 1.81±0.03 0.489±0.00 0.372±0.03 24.18±0.63 (0.0%) 0.09±0.0 (2.2%) 0.12±0.0 (4.7%)
3 NAR Equiv, DS 5000 1.65±0.13 0.506±0.01 0.346±0.02 24.06±0.43 (0.0%) 0.08±0.0 (2.0%) 0.11±0.0 (2.9%)
3 NAR Equiv, DSS 5000 1.60±0.10 0.520±0.02 0.352±0.03 24.18±0.55 (0.0%) 0.09±0.0 (2.2%) 0.12±0.0 (4.7%)
5 NAR Equiv, DS 5000 1.59±0.21 0.517±0.01 0.339±0.01 24.16±0.75 (0.0%) 0.08±0.0 (2.2%) 0.10±0.0 (4.5%)
Groundtruth sequence prediction baseline: - 1.000±0.00 0.525±0.00 17.52±0.32 (3.9%) 0.25±0.0 (24.2%) 0.29±0.0 (31.4%)
Random sequence prediction baseline: - 0.249±0.00 0.013±0.00 31.00±0.20 (0.0%) 0.03±0.0 (0.0%) 0.02±0.0 (0.0%)
ViennaRNA 2D-only baseline: - 0.258±0.00 0.470±0.00 29.10±0.00 (0.0%) 0.05±0.0 (0.0%) 0.05±0.0 (0.0%)