Algorithm 1.

The dual explanation algorithm.

Require: Training set $\mathcal{T}$; the black-box model f; explainable point x0; the number of nearest neighbors K

Ensure: Important features of x0 (vector $\mathbf{\text{a}}={(a_1,...,a_m)}^{\mathrm{\text{T}}}$ of the linear surrogate model coefficients)

1: Determine a set ${\mathcal{T}}_K$ of K nearest neighbors for x0 adding x0 itself

2: Construct the largest convex hull $\mathcal{P}$ of ${\mathcal{T}}_K$

3: Find extreme points of $\mathcal{P}$ and their number d ≤ K+1

4: Generate uniformly n points λ(j), j = 1, ..., n, from the unit simplex Δd−1

5: Find predictions zi of the black-box model in accordance with associated input $\sum _{i=1}^d{\text{λ}}_i^{(j)}{\mathbf{\text{x}}}_i^{*}$ for all i = 1, ..., n

6: Construct a new dual dataset $\mathcal{D}=\left\{({\text{λ}}^{(1)},z_1),...,({\text{λ}}^{(n)},z_n)\right\}$

7: Train the linear regression (Equation 8) on dataset $\mathcal{D}$ and find the vector of coefficients $\mathbf{\text{b}}={(b_1,...,b_d)}^{\mathrm{\text{T}}}$

8: Find vector a by solving optimization problem (Equation 15)