Figure 4.

The predictions of the presented model can also be interpreted thanks to the attention mechanism. (a) Distribution of the attention given by the model to each of the input channels for a given sample: the distribution can be interpreted as the relative importance of each data source for the yield prediction of that sample. The selected architecture has 8 different attention heads, each represented by a column in the plot, and they all consider different aspects of the input, hence giving varying importance to each channel. On average, multispectral images is the channel most considered, indicating that the model prioritizes its input well, as multispectral images are known to better correlate with yield than the other modalities. (b and c) Attention of a heads 1 and 4 respectively for 50 samples, the attention is consistent across samples, indicating reproducible behavior. (d) The attention distribution can also be evaluated across time for images collected at different stages of the plant growth. The model focuses on multispectral images at the beginning of growth, multispectral, and Digital Elevation Models at mid-growth and thermal images at the end of the growth. Interestingly, the model gives more importance to images obtained at mid-growth, confirming results found in Fig. 1. MIL, multiple instance learning; DEM, digital elevation models; Multispec., multispectral; Therm., thermal.