Figure 2.

Attention weight visualizations for head 9, layer 6; head 2, layer 11; and head 3, layer 8 of MFBERT on aspirin. Each layer shows the attention mechanism attending to various molecular features within the SMILES. For example, the start and end of the phenyl ring (specifically the “1” and “1=” tokens) are attended to highly by almost all other atoms in layer 11, head 2. In layer 6, head 9, the carbon in the “C=O” of the carbonyl group denoted by token “CC” has a high attention weight to the carbonyl oxygen and its closing branching token “)”.