Algorithm 1 Attention Routing using Scalar Product

Input: input parameters capsule $u_n^l$ from layer l

Output: output the Digicaps: $V_j$

1: affine transformation for all $u_n^l$:

$${\widehat{U}}_{(n^l,n^{l+1},d^{l+1})}^l=u_n^{Tl}\times W_{(n^l,n^{l+1},d^l,d^{l+1})}^l$$

2: Calculating self-attention weights:

$$A_{(n^l,n^l,n^{l+1})}^l=\frac{{\widehat{U}}_{(n^l,n^{l+1},d^{l+1})}^l\times {\widehat{U}}_{(n^l,n^{l+1},d^{l+1})}^{Tl}}{\sqrt{d^l}}$$

3: Use softmax to Calculate Weights C:

$$C_{(n^l,n^{l+1})}^l=\frac{exp\left({\sum }_{n^l}{A}_{(n^l,n^l,n^{l+1})}^l\right)}{{\sum }_{n^{l+1}}exp\left({\sum }_{n^l}{A}_{(n^l,n^l,n^{l+1})}^l\right)}$$

4: For all capsule j in $l+1$:

$$s_n^{l+1}={\widehat{U}}_{(n^l,n^{l+1},d^{l+1})}^l\times (C_{(n^l,n^{l+1})}^l+B_{(n^l,n^{l+1})}^l)$$

5: Compress the capsule length to between 0 and 1:

$$V_j(x_1\dots x_m)\leftarrow squash\left(s_n^{l+1}\right)$$

6: return $V_j$