Table A1.

Analysis results of selected articles.

ID	Authors and Year	Title	Publisher, Publication Name	PHM Activity	XAI Approach	Performance	XAI Assist PHM	Metric	Human Role	Uncertainty Management	Case Study
1	[54] Wong et al., 2015	On equivalence of FIS and ELM for interpretable rule-based knowledge representation	IEEE, IEEE Transactions on Neural Networks and Learning Systems	Diagnostic	Rule- and knowledge-based	Accuracy: 85.14% Good	Yes	No	No	No	Real— Circulating cooling water system for turbine. (energy sector)
2	[55] Wu et al., 2018	K-PdM: KPI-oriented machinery deterioration estimation framework for oredictive maintenance using cluster-based hidden Markov model	IEEE, IEEE Access	Prognostic	Rule- and knowledge-based	RMSE: 14.28 Very Good	No	No	No	Probabilistic state transition model	Simulated—Turbofan engine (aerospace)
3	[56] Massimo et al., 2018	Unsupervised classification of multichannel profile data using PCA: An application to an emission control system	Elsevier, Computers and Industrial Engineering	Diagnostic	Cluster- based	MSE: 2.127 × 10−5 to 5.809 × 10−3 Very Good	Yes	No	Yes	No	Real—Emission control system (automotive, environment)
4	[57] Mathias et al, 2019	Forecasting remaining useful life: Interpretable deep learning approach via variational Bayesian inference	Elsevier, Decision Support Systems	Prognostic	Interpretable model	MAE: 13.267 Better than other methods, except LSTM	No	No	No	Uncertainty in model parameters	Simulated— Turbofan engine (Aerospace)
5	[58] Imene et al., 2019	Fault isolation in manufacturing systems based on learning algorithm and fuzzy rule selection	Springer, Neural Computing and Applications	Diagnostic	Rule- and knowledge- based	Accuracy: 97.01% Very Good	Yes	No	No	Probabilistic classification by Bayes decision rule	Real— Rotary kiln (civil engineering)
6	[59] Kerelous et al., 2019	Interpretable logic tree analysis: A data-driven fault tree methodology for causality analysis	Elsevier, Expert Systems with Applications	Diagnostic	LAD	Mean and standard errors are less than 2% and 1% Very good	Yes	No	Yes	FTA—Expert opinion	Simulated— Actuator system (manufacturing, energy, production, chemical)
7	[60] Rajendran et al., 2019	Unsupervised wireless spectrum anomaly detection with interpretable features	IEEE, IEEE Transactions on Cognitive Communications and Networking	Anomaly detection	Autoencoder	Generally better than other tested methods	Yes	No	No	Probabilistic classification error by discriminator	Real—software defined radio spectrum simulated—synthetic data (communication)
8	[61] Wang et al., 2019	An attention-augmented deep architecture for hard drive status monitoring in large-scale storag systems	ACM, ACM Transactions on Storage	Prognostic, diagnostic	Attention mechanism	Prognostic precision: 94.5–98.3% Generally, better than other methods. No comparison in diagnostic	Diag: Yes Prog: No	No	No	No	Real— Hard drive (information technology)
9	[62] Le et al., 2019	Visualization and explainable machine learning for efficient manufacturing and system operations	ASTM, Smart and Sustainable Manufacturing Systems	Diagnostic	Others	N/A 1	Yes	No	Yes	No	Simulated—turbofan (aerospace)
10	[63] Langone et al., 2020	Interpretable anomaly prediction: Predicting anomalous behavior in industry 4.0 settings via regularized logistic regression tools	Elsevier, Data and Knowledge Engineering	Anomaly detection	Interpretable model	Kappa: 0.4–0.6 AUC: 0.6–0.8 F1: 0.3–0.5 PRAUC: 0.2–0.4 Good	Yes	No	No	Statistical feature extraction	Real— High-pressure plunger pump (chemical)
11	[64] Peng et al., 2020	A dynamic structure-adaptive symbolic approach for slewing bearings life prediction under variable working conditions	Sage, Structural Health Monitoring	Prognostic	Interpretable model	RMSE: 18.19 Better than previous methods	Yes	No	No	No	Real— Slewing bearings (rotating machinery, energy, manufacturing)
12	[65] Ritto et al., 2020	Digital twin, physics-based model, and machine learning applied to damage detection in structures	Elsevier, Mechanical Systems and Signal Processing	Diagnostic	Interpretable model	Accuracy: 74.8–93.3% Good	No	No	No	No	Not specified— Spring mass system (wind turbine, energy)
13	[66] Rea et al., 2020	Progress toward interpretable machine learning based disruption predictors across tokamaks	Taylor and Francis, Fusion Science and Technology	Diagnostic	Interpretable model	N/A	No	No	No	Physic-based indicator	Real DIII—D and JET tokamaks (energy)
14	[67] Murari et al., 2020	Investigating the physics of tokamak global stability with interpretable ML tools	MDPI, Applied Sciences	Anomaly detection	Mathematic equation	Success Rate > 90% Very Good	No	No	No	No	Type unspecified—Tokamak (energy)
15	[68] Zhou et al., 2020	Fault diagnosis of gas turbine based on partly interpretable convolutional neural networks	Elsevier, Energy	Diagnostic	Tree-based	Accuracy: 95.52% Better than other tested methods	Yes	No	No	No	Simulated— Gas turbine model (energy)
16	[69] Zhou et al., 2020	Addressing noise and skewness in interpretable health-condition assessment by learning model confidence	MDPI, Sensors	Diagnostic	Rule- and knowledge- based	F1 Score: 0.8005 Very Good	No	No	No	No	Real— Aircraft structure. (aerospace)
17	[70] Jianbo et al., 2020	Knowledge extraction and insertion to deep belief network for gearbox fault diagnosis	Elsevier, Knowledge-Based Systems	Diagnostic	Rule- and Knowledge-based	Accuracy: 92.33 Very Good	Yes	No	No	No	Real— Gearbox (manufacturing, energy, automotive)
18	[71] Conde et al., 2020	Isotonic boosting classification rules	Springer, Advances in Data Analysis and Classification	Diagnostic	Rule- and knowledge-based	Total Misclassification Probability (TMP): 0.036-0.164 Good and comparable to other methods	Yes	No	No	No	Real— Induction motor (manufacturing, energy, production)
19	[72] Antonio et al., 2020	Using an autoencoder in the design of an anomaly detector for smart manufacturing	Elsevier, Pattern Recognition Letters	Anomaly detection	Autoencoder	Precision: 77.8–100% Accuracy: 94.9–100% Same as the previous best method	Yes	No	No	No	Simulated— Continuous batch washing equipment (industrial laundry)
20	[73] Abid et al., 2020	Robust interpretable deep learning for intelligent fault diagnosis of induction motors	IEEE, IEEE Transactions on Instrumentation and Measurement	Diagnostic	Filter-based	Accuracy: 99.95% ± 0.05% Better than other tested methods and previous works	Yes	No	No	No	Real— Electrical and mechanical motor (Manufacturing, Energy, Production)
21	[74] Liu et al., 2020	Tscatnet: An interpretable cross-domain intelligent diagnosis model with antinoise and few-shot learning capability	IEEE, IEEE Transactions on Instrumentation and Measurement	Diagnostic	Filter-based	Accuracy: 100% Better than other tested methods	Yes	No	No	No	Real—Bearing, drive train (manufacturing, energy, production)
22	[75] Li et al., 2020	Waveletkernelnet: an interpretable deep neural network for industrial intelligent diagnosis.	IEEE, IEEE Transactions on Systems, Man, and Cybernetics: Systems	Diagnostic	Filter-based	Accuracy: 92.61–99.91% Better than other tested methods	Yes	No	No	No	Real—Bearing, drive train (manufacturing, energy, production)
23	[76] Chen et al., 2020	Vibration signals analysis by explainable artificial intelligence approach: Application on bearing faults diagnosis	IEEE, IEEE Access	Diagnostic	Attention mechanism	N/A	No	No	No	No	Real— Rolling bearing (manufacturing, energy, production)
24	[77] Sun et al., 2020	Vision-based fault diagnostics using explainable deep learning with class activation maps	IEEE, IEEE Access	Diagnostic	Attention mechanism	Accuracy: 95.85% Precision: 100% Very good	No	No	No	No	Real— Base-excited cantilever beam, water pump system (manufacturing, energy, production)
25	[78] Oh et al., 2020	VODCA: Verification of diagnosis using CAM-based approach for explainable process monitoring	MDPI, Sensors	Diagnostic	Attention mechanism	Accuracy: 78.4–99.5% Good	Yes	No	No	True positive and true negative indicators	Simulated— Ford motor and real—sapphire grinding (automotive, production)
26	[79] Sreenath et al., 2020	Fouling modeling and prediction approach for heat exchangers using deep learning	Elsevier, International Journal of Heat and Mass Transfer	Failure Prediction	Model agnostic	Accuracy: 99.80–99.92% Very good	No	No	No	No	Simulated— Heat-exchanger model (manufacturing, energy, production)
27	[80] Hong et al., 2020	Remaining useful life prognosis for turbofan engine using explainable deep neural network with dimensional reduction	MDPI, Sensors	Prognostic	Model Agnostic	RMSE: 10.41 Very good	No	No	No	No	Simulated—Turbofan engine (aerospace)
28	[81] Grezmak et al., 2020	Interpretable convolutional neural network through layer-wise relevance propagation for machine fault diagnosis	IEEE, IEEE Sensors Journal	Diagnostic	LRP	Accuracy: 100% Very good	No	No	No	No	Real— Induction motor (manufacturing, energy, production)
29	[82] Ming et al., 2020	ProtoSteer: Steering deep sequence model with prototypes	IEEE, IEEE Transactions on Visualization and Computer Graphics	Diagnostic	Others	N/A	Yes	No	Yes	No	Real— Vehicle fault log (automotive)
30	[83] Chen et al., 2020	Frequency-temporal-logic-based bearing fault diagnosis and fault interpretation using Bayesian optimization &ANN	Elsevier, Mechanical Systems and Signal Processing	Diagnostic	Others	Better error percentage, error rate and robustness than other tested methods	Yes	No	No	No	Real—Bearings (manufacturing, energy, production)
31	[84] Steenwinckel et al., 2021	FLAGS: A methodology for adaptive anomaly detection and root cause analysis on sensor data streams by fusing expert knowledge with machine learning	Elsevier, Future Generation Computer Systems	Anomaly detection, diagnostic	Rule- and knowledge- based	Accuracy: 75% Good in anomaly detection, no result for diagnostic	Yes, for both	No	Yes	FMEA and FTA—Expert opinion	Real—Train (transportation)
32	[85] Zhang et al., 2021	A new interpretable learning method for fault diagnosis of rolling bearings	IEEE, EEE Transactions on Instrumentation and Measurement	Diagnostic	Cluster- based	Accuracy: 99.3–100% Very good	Yes	No	No	No	Real— Rolling bearing (manufacturing, energy, production)
33	[86] Onchis et al., 2021	Stable and explainable deep learning damage prediction for prismatic cantilever steel beam	Elsevier, Computers in Industry	Diagnostic	Model Agnostic	Accuracy for 19% damage: 75–92% Accuracy for 43% damage: 85–95% Good	Yes, by LIME only	Stability-fit compensation index (SFC)—Quality indicator of the explanations	No	Yes	Real— Prismatic cantilever steel beam (civil engineering, structural engineering)
34	[87] Kim et al., 2021	An explainable convolutional neural network for fault diagnosis in linear motion guide	IEEE, IEEE Transactions on Industrial Informatics	Diagnostic	Attention mechanism	Accuracy: 99.59–99.71% Very good	No	No	No	No	Real— Linear motion guide (manufacturing, energy, production)
35	[88] Ding et al., 2021	Stationary subspaces autoregressive with exogenous terms methodology for degradation trend estimation of rolling and slewing bearings	Elsevier, Mechanical Systems and Signal Processing	Prognostic	Others	MAE: 0.0375–0.0414 RMSE: 0.0482–0.0659 Better than other methods and comparable to previous works	Yes	No	No	No	Real— Rolling and slewing bearings (manufacturing, energy, production)

¹ N/A = Item not included in the studied work.