Figure 6.

Illustration of window based self-attention and masking self-attention mechanism. (A) for the 3D window based multi head self-attention. (B) for the efficient batch computation approach for multi head self-attention in shifted window partitioning. MSA, multi head self-attention.