Counterfactual Supervision-Based Information Bottleneck for Out-of-Distribution Generalization

. 2023 Jan 18;25(2):193. doi: 10.3390/e25020193

Algorithm 1 Counterfactual Supervision-based Information Bottleneck (CSIB)

Input:

P (X^{e}, Y^{e})

e \in E_{t r}

r t h > 0

c \geq d i m (Z_{i n v})

M ≫ 0

, and

(x, y)

is an example randomly drawn from

P (X^{e}, Y^{e})

Output: classifier

w \in R^{c + 1}

, feature extractor

Φ = R^{c \times d}

Begin:

1:
$L v \leftarrow []$ ; $L r \leftarrow []$ ; $Φ^{'} \leftarrow I^{d \times d}$
2:
$d^{'} \leftarrow d i m (X^{e})$
3:
Apply IB-ERM method (Equation (12)) to $P (X^{e}, Y^{e})$ and obtain $w^{*} \in R^{c + 1}$ and $Φ^{*} \in R^{c \times d^{'}}$
4:
Apply SVD to $Φ^{*}$ as $Φ^{*} = U Λ V^{T} = [U_{1}, U_{2}] [Λ_{1}, 0; 0, 0] [V_{1}^{T}; V_{2}^{T}]$
5:
$r \leftarrow r a n k (Φ^{*})$
6:
$z_{1 : r}^{1} \leftarrow [- M, \dots, - M]$ ; $z_{r + 1 : d^{'}}^{1} \leftarrow V_{2}^{T} Φ^{'} x$
7:
$z_{1 : r}^{2} \leftarrow [M, \dots, M]$ ; $z_{r + 1 : d^{'}}^{2} \leftarrow V_{2}^{T} Φ^{'} x$
8:
$x^{1} \leftarrow V z^{1}$ ; $x^{2} \leftarrow V z^{2}$
9:
if $L v$ is not empty then
10:
$z_{o l d} \leftarrow []$ ; $i \leftarrow 0$ ; $x^{'} \leftarrow x$
11:
while $i < l e n (L v)$ do
12:
$z \leftarrow L v [i] x^{'}$
13:
$z_{o l d}$ .append(z)
14:
$x^{'} \leftarrow z_{L r [i] :}$
15:
$i \leftarrow i + 1$
16:
end while
17:
$i \leftarrow 0$
18:
while $i < l e n (L v)$ do
19:
$j \leftarrow l e n (L v) - i$
20:
$z^{1} \leftarrow z_{o l d} [j]$ ; $z^{2} \leftarrow z_{o l d} [j]$
21:
$z_{L r [j] :}^{1} \leftarrow x^{1}$ ; $z_{L r [j] :}^{2} \leftarrow x^{2}$
22:
$x^{1} \leftarrow L v {[j]}^{T} z^{1}$ ; $x^{2} \leftarrow L v {[j]}^{T} z^{2}$
23:
$i \leftarrow i + 1$
24:
end while
25:
end if
26:
if label( $x^{1}$ ) = label( $x^{2}$ ) then
27:
$L r$ .append(r); $L v$ .append( $V^{T}$ )
28:
$X^{e} \leftarrow V_{2}^{T} X^{e}$ ; $Φ^{'} \leftarrow V_{2}^{T} Φ^{'}$
29:
Go to Step 2
30:
end if
31:
$w \leftarrow w^{*}$ ; $Φ \leftarrow Φ^{*}$

End