Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

Abstract

Recent advancements in multimodal large language models (MLLMs) have significantly improved performance in visual question answering. However, they often suffer from hallucinations. In this work, hallucinations are categorized into two main types: initial hallucinations and snowball hallucinations. We argue that adequate contextual information can be extracted directly from the token interaction process. Inspired by causal inference in the decoding strategy, we propose to leverage causal masks to establish information propagation between multimodal tokens. The hypothesis is that insufficient interaction between those tokens may lead the model to rely on outlier tokens, overlooking dense and rich contextual cues. Therefore, we propose to intervene in the propagation process by tackling outlier tokens to enhance in-context inference. With this goal, we present FarSight, a versatile plug-and-play decoding strategy to reduce attention interference from outlier tokens merely by optimizing the causal mask. The heart of our method is effective token propagation. We design an attention register structure within the upper triangular matrix of the causal mask, dynamically allocating attention to capture attention diverted to outlier tokens. Moreover, a positional awareness encoding method with a diminishing masking rate is proposed, allowing the model to attend to further preceding tokens, especially for video sequence tasks. With extensive experiments, FarSight demonstrates significant hallucination-mitigating performance across different MLLMs on both image and video benchmarks, proving its effectiveness.

1. Introduction

Multimodal large language models (MLLMs) [1, 3, 9, 11, 44, 45, 89, 97] have become essential tools in addressing numerous vision tasks and performing complex visual question-answering due to their superior capabilities in content comprehension [32] and generation [15]. Despite their remarkable versatility, MLLMs often suffer from hallucinations. Specifically, MLLMs frequently generate convincing text responses that contradict the visual content of an image, describing elements not present in the image. Hallucinations can be categorized into two types: initial hallucinations and snowball hallucinations, as illustrated in Fig. 1. Specifically, initial hallucinations (e.g., bridge) stem from insufficient information within the model, while snowball hallucinations (e.g., handrails) occur when the model maintains consistency with previous hallucinations.

Figure 1.

Illustrates the phenomenon of snowball hallucinations as an extension of initial hallucinations. MLLMs produce hallucinations by asserting nonexistent objects (e.g., bridge) within the image, followed by further explanatory errors (e.g., handrails). This progression from initial to snowball hallucinations reveals the model’s tendency to build upon its own erroneous assumptions.

The key to mitigating hallucinations lies in extracting contextual information from the token interaction process. Recent studies focus on external knowledge retrieval [2, 62] and robust instruction fine-tuning [61, 73, 83], but these methods often incur substantial additional costs. Conversely, other approaches focus on training-free decoding strategies such as contrastive decoding [25, 31, 34, 70] and self-calibrating attention [24, 48, 53, 74]. They aimed to enhance the accuracy and consistency of generated responses by reducing excessive reliance on linguistic priors in the token interaction process. Though previous works have shown effectiveness, they lack analysis of the interaction process between multimodal tokens and the causes of hallucinations. For example, Fig. 2 illustrates a high proportion of snowball hallucinations, particularly in video captioning. Interestingly, these methods have not been effective in reducing the proportion of snowball hallucinations. In this study, we hypothesize that insufficient interaction between tokens may result in over-reliance on outlier tokens, thereby neglecting dense and informative contextual cues. In this work, we argue that intervening effectively in the token interaction process enhances in-context inference.

Figure 2.

Percentage of initial hallucination (IH) and percentage of snowball hallucination (SH) (calculated over the entire datasets) for LLaVA-1.5-7B [44], Video-LLaVA-7B [40] and EDVT [52].

To delve deeper into this phenomenon, we analyze the attention maps during decoding and identify two issues contributing to hallucinations. (i) Attention Collapse in MLLMs: As illustrated in Fig. 3 (a), we observe that the model tends to allocate disproportionate attention to tokens with limited informational content. These low-information yet high-attention outlier tokens, such as visual backgrounds and textual symbols, disrupt the effective propagation of relevant information. This issue arises because the softmax attention mechanism requires all attention scores to be non-zero and sum to one, causing even low-information or non-priority tokens to receive disproportionate attention. Attention collapse, akin to the findings in Opera [24] on the “summary token”, causing a gradual attenuation of vision and text information transmission as the generated text extends. (ii) Positional Information Decay: As illustrated in Fig. 3 (b), we observe a progressive decline in attention to dense vision information throughout the generation process. This occurs due to the rotational position encoding (RoPE) [63], whose long-term decay fails to provide adequate positional information to ensure sufficient interaction between vision and text tokens. As the relative distance increases, the flow of vision token information gradually diminishes, leading to potential hallucinations. Therefore, our findings indicate that maintaining balanced information propagation and refining positional encoding can mitigate attention collapse and positional information decay, both of which contribute to hallucinations.

Figure 3.

(a): Attention Collapse in MLLMs: Outlier tokens from different modalities are assigned disproportionately high attention scores, hindering interaction between relevant tokens. (b): Positional Information Decay: As text generation progresses, attention to visual information gradually diminishes. (c): Our FarSight, as a plug-in, mitigates these issues by effectively reducing attention interference from outlier tokens and improving response accuracy.

In this work, we propose FarSight, a versatile plug-and-play decoding strategy that reduces attention interference from outlier tokens by optimizing the causal mask. Specifically, we initialize a set of attention registers within the upper triangular matrix of the causal mask to capture attention diverted to outlier tokens. These attention registers retain the causal decoding properties, ensuring that information from future tokens is not accessed prematurely. Additionally, we design a dynamic register-attention distribution mechanism that explicitly optimizes attention allocation at each decoding step for robust in-context inference.

The core of our method is to optimize the effective propagation of tokens. We modulate attention distribution for tokens with multimodal informational content to improve token propagation. Furthermore, the relative positional limitations of RoPE encoding lead to insufficient transmission of vision-to-text token information during contextual interactions, which undermines positional awareness. Therefore, we introduce a progressively diminishing masking rate within the causal mask to encode absolute positional information, allowing the model to attend to further distant preceding tokens, especially for video sequence tasks.

With extensive experiments, FarSight demonstrates significant hallucination-mitigating performance across different MLLMs on both image and video benchmarks, proving its effectiveness. Our contributions are as follows:

• We analyze the self-attention token propagation patterns, revealing two main causes of hallucinations in MLLMs: attention collapse and positional information decay.

• We propose FarSight, a plug-and-play decoding strategy that effectively mitigates hallucinations stemming from these issues by merely adjusting the causal mask.

• Extensive evaluations on both image and video tasks demonstrate the superior performance of FarSight, offering an effective solution for mitigating hallucinations.

2. Related Work

Hallucinations in MLLMs.

Leveraging open-source large language models like LLaMA [66, 67] and Vicuna [6], MLLMs [4, 12, 23, 35, 38, 57, 65, 77, 81, 87, 90] can understand and generate a wide range of content more effectively by combining information from multiple modalities, such as text, images, and audio. Hallucination in MLLMs [16, 17, 21, 27, 42, 46, 47, 60, 64, 91] refers to the generation of text that is misaligned with the content of the provided images. Hallucination may originate from reliance on model priors [7, 19, 26, 33, 39, 43, 86, 93], limited knowledge comprehension [21, 42, 56, 80, 96], or an inability to effectively contextualize the given input [8, 22, 36, 48, 49, 69, 76]. According to the causes of hallucination, hallucinations can be classified into two types: initial hallucinations [58, 71] occur due to the model lacking necessary information; snowball hallucinations [88, 94] arise when the model generates a series of hallucinations to maintain consistency with previous ones, even when the required knowledge is available. In this paper, we primarily conducted experiments and analyses on image and video benchmarks.

Hallucination Mitigation for MLLMs.

Researchers have proposed various strategies, from data optimization to model adjustments, to improve the accuracy and consistency of generated content. To mitigate hallucination, solutions include robust instruction tuning [28, 82, 83, 85], post-hoc processing using auxiliary analysis networks [14, 72, 79, 92], and various decoding strategies [13, 24, 31, 34, 70, 98]. Recent studies have focused on outlier tokens, causing generated text to emphasize summarizing information from these tokens rather than utilizing dense and rich contextual cues. Additionally, some studies [53, 74, 78] have found that RoPE positional encoding is insufficient to support information propagation between multimodal tokens in contextual reasoning. This paper proposes an optimized causal masking approach to extract sufficient contextual information during token interactions, effectively mitigating hallucination without additional training, data, or inference time.

3. Preliminary and Motivation

3.1. Paradigm of MLLMs Generation

Vision and Language Inputs.

The inputs of MLLMs consist of both image and text. Generally, the raw images are commonly fed to the visual encoder. Then the cross-model projection module maps vision information into LLMs’ input space, which is denoted as vision tokens $x^v=\left\{x_0,x_1,\dots ,x_{N-1}\right\}$ where $N$ is the length of vision tokens. Similarly, text is processed by tokenizer and embedding modules, which is denoted as text tokens $x^t=\left\{x_N,x_{N+1},\dots ,x_{M+N-1}\right\}$ where $M$ is the length of text tokens. Then, the image and text tokens are concatenated as the final input and denoted as ${\left\{x\right\}}_{t=0}^{T-1}$ where $T=N+M$.

MLLMs Forward.

The backbone networks of MLLMs $M_{\theta }$ are pre-trained LLMs (e.g., Vicuna [6] and LLaMA 2 [6]), parameterized by $\theta$ that auto-regressively generates responses. Given a multimodal input sequence $x$, the model maps the logit distribution to the next token prediction output $y_t\in {ℝ}^{\left|𝒱\right|}$ at time step $t$ in the vocabulary set $𝒱$:

$$y_t~p_{\theta }\left(y_t|x,y_{<t}\right)\propto {\text{logit}}_{\theta }\left(y_t|x,y_{<t}\right),$$ (1)

where $y_{<t}$ denotes all previously generated tokens ${\left\{x_i\right\}}_{i=0}^{t-1}$.

Next Token Decoding.

After obtaining the next token probability $p\left(y_t|x,y_{<t}\right)$, different decoding strategies [8, 18, 24] are proposed to predict the next token. The decoded token is concatenated to the last of the original input text for the next-round generation, until the generation is ended.

3.2. What Causes Hallucinations

Attention Collapse in MLLMs.

We investigate the self-attention in the transformer block [68] of the auto-regressive decoder and leverage a column-wise product to calculate metric values. Denote the current generated sequence as ${\left\{x_i\right\}}_{i=0}^{t-1}$ and their causal self-attention weights as ${\left\{{\omega }_{t-1,j}\right\}}_{j=0}^{t-1}$ applied to the next token prediction. The weights $\omega \in {ℝ}^{n\times n}$ can be obtained from the softmax function as follows:

$$\begin{array}{cc}𝓞=\text{SoftMax}\left(\omega \right)\cdot V,& \omega =\frac{Q\cdot K^{丅}}{\sqrt{d_l}}+M,\end{array}$$ (2)

where $Q,K,V\in {ℝ}^{n\times d_l}$ are the Query, Key, and Value matrices. $n$ and $d_l$ are the sequence length and the hidden dimensions, $M\in {ℝ}^{n\times n}$ is the causal mask, and $𝓞$ is the output. The causal mask $M$ ensures that the model does not attend to future tokens, preserving causality in the sequence. The attention weights are structured as follows:

$${\omega }_i={\left[{\omega }_{i1},{\omega }_{i2},\cdots ,{\omega }_{ii},0,\cdots ,0\right]}_n.$$ (3)

Proposition 3.1 (Attention Collapse in MLLMs). Let inputs be sampled from a data distribution $q\left(x_1,x_2,\dots ,x_N\right)$ and processed by a contextual, layer-wise decoder with attention layers. Define the disproportionality in an attention layer as measured by the total probability of prefixes ${\sum }_{x_{<N}}q\left(x_{<N}\right)$, where the attention collapse after applying softmax for $i<n<N$ in the l-th layer satisfies:

$$\sum _{n=1}^N\sum _{j\le i}{\omega }_{n,j}^l>\frac{I\left(x_{\le i};x_{n+1}\right)}{I\left(x_{\le n};x_{n+1}\right)}\sum _{n=1}^N\sum _{j\le n}{\omega }_{n,j}^l+o\left(1\right),$$ (4)

Here, $I\left(A;B\right)$ denotes the mutual information between two variables $A$ and $B$, indicating the amount of shared information between them. $I\left(x_{\le n};x_{n+1}\right)>0$ represents that the token $x_{n+1}$ is informationally dependent on the preceding sequence $x_{\le n}$, quantifying how much information about $x_{n+1}$ is contained within $x_{\le n}$.

Remark:

Proposition 3.1 indicates that Attention Collapse refers to the phenomenon where the attention weights for certain tokens far exceed the informational contribution of those tokens. This often occurs with semantically irrelevant tokens, such as non-functional words (i,e., punctuation marks) and background vision tokens. As a result, the focus of the model diffuses across these irrelevant tokens, increasing perplexity during length extrapolation and hindering interaction among semantic tokens, as illustrated in Fig. 3 (a).

Positional Information Decay.

The vanilla attention model lacks positional awareness, as it does not encode relative distance between tokens. In contrast, RoPE [63] addresses this by encoding the positional data of tokens using a rotation matrix, which inherently includes an explicit relative position dependency. Within each attention $\omega$, RoPE is applied across all projected query $Q$ and key $K$ inputs to compute the attention weights by leveraging relative distance between tokens. Consequently, the attention with relative position embedding is expressed as:

$${\tilde{\omega }}_{ij}=\frac{R_i\cdot q_i\cdot R_j^T\cdot k_j^T}{\sqrt{d_l}}=\frac{q_i\cdot R_{j-i}\cdot k_j^T}{\sqrt{d_l}},$$ (5)

where $R\in {ℝ}^{n\times n}$ denotes the rotary position embedding matrices applied to the query and key. $j-i$ stands for relative position between $q_i$ and $k_j$. The long-term decay refers to the decrease of ${\tilde{\omega }}_{ij}$ as the relative distance $j-i$ increases.

Remark:

RoPE integrates relative position data by multiplying rotation matrices rather than appending positional embeddings to the input. The relative proximity between two tokens effectively determines their influence, as closer tokens should impact each other more than distant ones. However, using the same attention mechanism for both vision and text tokens results in unintentional text generation in MLLMs, as illustrated in Fig. 3 (b). Consequently, we argue that RoPE long-term decay limits multimodal tokens’ information propagation, which contributes to hallucination. In contrast, maintaining absolute positional focus in generated text could allow the model to achieve precise positional awareness and improve response accuracy.

4. Methodology

Fig. 4 provides an overview of the proposed strategy, built upon an LLM decoding paradigm in Section 3.1. Attention registers, detailed in Section 4.1, are introduced to absorb outlier tokens’ attention scores, dynamically guiding the model toward contextually rich semantic information. Meanwhile, a progressively diminishing masking rate is introduced to capture absolute positional focus with rigorous theoretical justification, which is described in Section 4.2. For ease of comprehension of how FarSight works, Algorithm 1 exhibits the pseudo-code in the decoder layer.

Figure 4.

The scheme of the proposed FarSight strategy, which integrates with the softmax operation, replacing the traditional causal mask. Specifically, the attention score matrix $\omega$ is cleared of attention values in the upper triangular part, then register-attention scores are added using the matrix $𝓟$ followed by the softmax computation. $𝓟$ has linear decay in the upper triangular part and zeros in the lower triangular part. After the softmax operation, the remaining attention probabilities in the upper triangular part are cleared to ensure the causal decoding property is preserved.

4.1. Upper Triangular Metric as Attention Registers

To alleviate attention collapse issues, we propose the dedicated attention register to allocate excess attention scores. For each $\omega$, we construct an upper triangular score matrix $𝓟\in {ℝ}^{n\times n}$ as attention register, defined as follows:

$${𝓟}_i={[\underset{i}{\underbrace{0,0,\cdots ,0,}}\underset{n-i}{\underbrace{{𝓟}_{i,i+1},{𝓟}_{i,i+2},\cdots ,{𝓟}_{i,n}}}]}_n,$$ (6)

where ${𝓟}_i$ allocates $n-i$ register-attention scores in each row to handle excess attention values while maintaining zero values for positions up to $i$. To integrate $𝓟$ with $\omega$, we adjust $\omega$ by adding the register-attention scores from $𝓟$ as follows:

$$\begin{array}{ccc}W=\omega \cdot C+𝓟,& \text{where}& C=\text{tril}\left({\mathrm{𝟙}}_{n\times n}\right),\end{array}$$ (7)

where $C\in {ℝ}^{n\times n}$ denotes a lower-triangular matrix filled with ones to ensure causal masking by allowing attention only to preceding or current tokens, as illustrated in Fig. 4.

Since the model is training-free, the attention-registers $𝓟$ should not interfere with the original attention score distribution $\omega$ during inference and align with the relative positional encoding $R$ in Eq. 5 to maintain coherence in generated text. For ${𝓟}_{i,j}$, the values are defined as:

$$\begin{array}{cc}{𝓟}_{i,j}=-\left(j-i\right)\cdot \sigma ,& \forall j>i,\end{array}$$ (8)

where $\sigma$ is a decay rate hyperparameter. This setup ensures that $𝓟$ conforms to the gradual attenuation pattern in attention. Thus, the final attention score matrix with FarSight is defined as:

$$W_i={\left[{𝓟}_{i,1},{𝓟}_{i,2},\cdots ,{𝓟}_{i,i},-\sigma ,-2\sigma ,\cdots ,-\left(n-i\right)\sigma \right]}_n,$$

where ${𝓟}_{i,j}$ denotes the original attention score at $\left(i,j\right)$, with ${𝓟}_{i,i}$ capturing the self-attention along the diagonal. The decay factor $\sigma \cdot \left(j-i\right)$ applied for future tokens $i<j$, which enforces causal masking. The standard causal mask operation in Eq. 2 is then modified as:

$$\tilde{W}=\text{SoftMax(}\underset{w}{\underbrace{\omega \cdot C+𝓟}})\cdot C.$$ (9)

Remark:

The $W=\omega \cdot C+𝓟$ within the SoftMax function incorporates register-attention scores by masking the attention matrix, while the $C$ outside SoftMax ensures that any masked scores are reset to zero. This design enables FarSight to retain causal decoding properties, preventing information from future tokens is not accessed prematurely. The register-attention matrix $𝓟$ effectively captures and buffers excess attention by providing dedicated slots for surplus values, ensuring that the main attention mechanism remains focused on relevant tokens without being distracted by irrelevant or future positions.

$$\overline{\underset{¯}{\begin{array}{l}\underset{¯}{\mathrm{Algorithm}1\text{Pseudocode}\text{of}\text{FarSight}\text{in}\text{PyTorch}\text{Style}.}\\ \#\text{x:}\text{hidden}\text{input}\text{in}\text{each}\text{attention}\text{layer}\\ \#\text{C:}\text{upper-triangular}\text{matrix}\text{filled}\text{with}\text{1}\\ \#\text{Sigma}:\text{decay}\text{factor,}\text{n\_head}:\text{attention}\text{head}\\ \text{def}\text{register\_score}\text{(self,}\text{seq\_len:}\text{int):}\\ \#\text{Create}\text{a}\text{register}\left(\text{upper-triangular}\text{matrix}\text{with}\text{0}\right)\\ \text{register}\text{=}\text{1}-\text{torch.triu}\text{(torch.full}\text{(}\left(\text{seq\_len,}\text{seq\_len}\right)\text{,}\\ \text{1),}\text{diagonal=1)}\\ \#\text{Generate}\text{register}\text{alibi}\text{biases}\\ \text{register\_score}\text{=}\text{get\_alibi\_biases}\text{(n\_heads,}-\text{register.flip}\text{(}\\ \text{dims=}\left[\text{1}\right]\text{))}.\text{flip}\left(\text{dims=}\left[\text{1}\right]\right)\\ \#\text{Final}\text{register}\text{score}\text{adjustment}\\ \text{return}\text{register\_score.contiguous()}\text{*}\text{(1}-\text{mask)}\\ \text{def}\text{FarSightAttention}\left(\text{self,}\text{x:}\text{torch.Tensor}\right):\\ \#\text{query,}\text{key,}\text{value}\text{projection}\\ \text{xq,}\text{xk,}\text{xv}\text{=}\text{qkv\_proj}\left(\text{x}\right)\\ \text{\#query,}\text{key,}\text{value}\text{projection}\text{and}\text{get}\text{QKˆT/sqrt}\left(\text{d}\right)\\ \text{scores}\text{=}\text{torch.matmul}\left(\text{xq,}\text{xk.transpose}\left(\text{2,3}\right)\right)\text{/}\text{math.sqrt(}\\ \text{self.hid\_dim)}\\ \#\text{add}\text{register}\text{scores}\text{and}\text{introduce}\text{decay}\text{factor}\\ \text{scores}\text{=}\text{scores}\text{*}\text{C}\text{*}\text{sigma}\text{+}\text{register\_score}\\ \#\text{remove}\text{register}\text{score}\text{to}\text{keep}\text{causal}\text{decoding}\\ \text{scores}\text{=}\text{torch.softmax}\left(\text{scores,}\text{dim=}-\text{1}\right)\text{*}\text{C}\\ \#\text{final}\text{projection}\text{and}\text{output}\\ \text{return}\text{self.wo(torch.matmul}\text{(scores,}\text{xv))}\end{array}}}$$

4.2. Positional Awareness Encoding

The core idea of absolute position encoding is to modify the attention matrix so that the sum of actual attention scores (located in the lower triangular part of the attention matrix $\omega$) is not constrained to equal 1, as illustrated in Fig. 4. Specifically, we introduce a progressively diminishing masking rate in the causal mask, allowing attention distributions to vary across positions, thereby effectively incorporating absolute positional information. Specifically, let ${\omega }_i$ denote the actual attention scores of the $i\text{-th}$ row and ${𝓟}_i$ the register-attention scores in the $i\text{-th}$ row. The overall attention distribution for the row is given by:

$$\sum _{j\le i}{\text{SoftMax}}_{j=1}^n\left({\omega }_{i,j}\right)+\sum _{j>i}{\text{SoftMax}}_{j=1}^n\left({𝓟}_{i,j}\right)=1,$$

where ${\text{SoftMax}}_{j=1}^n\left({\omega }_{i,j}\right)$ denotes the normalized attention score on position $j\le i$ in the $i\text{-th}$ row, while ${\text{SoftMax}}_{j=1}^n\left({𝓟}_{i,j}\right)$ denotes the normalized register-attention score for $j>i$ in the same row.

The model encodes positional information for a sequence of identical input tokens, $x={\left\{x_i\right\}}_{i=1}^n\in {ℝ}^n$, by leveraging both attention score accumulation and decay. Specifically, the actual attention scores ${\omega }_{i,j}$ are uniform across each row. Consequently, the cumulative sum of their exponentiated values progressively increases with the row index $i$. This cumulative increase emphasizes information before the current position, contributing to the encoding of absolute positional information. Simultaneously, the cumulative sum of the exponentiated register-attention scores decreases as $i$ increases due to the applied decay in ${𝓟}_{i,j}$. This decay constrains attention on content after the current position, ensuring that attention primarily emphasizes preceding information. Consequently, we obtain:

$$\sum _{j\le i}{\text{SoftMax}}_{j=1}^n\left({\omega }_{i,j}\right)<\sum _{j\le i+1}{\text{SoftMax}}_{j=1}^n\left({\omega }_{i+1,j}\right),$$

indicating that, after applying Eq. 9, the accumulated attention over valid tokens exhibits a monotonically increasing trend with respect to the row index $i$, i.e. $\tilde{W}\cdot V={\sum }_{i=1}^n{\beta }_i{\text{v}}_i$, satisfying ${\beta }_1<{\beta }_2<\cdots <{\beta }_n=1$, progressively encoding the absolute positional context. This progressive allocation allows the model to maintain an ordered information flow across positions, where tokens at later positions aggregate increasingly more historical context from preceding tokens. As $i$ grows, the model sharpens its focus on earlier tokens, reinforcing long-range dependencies and enhancing positional awareness in the generated sequence.

5. Experiments

5.1. Experimental Setup

Baseline.

We select six representative MLLMs to evaluate performance across image and video tasks, including InstructBLIP [10], LLaVA-1.5 [44], VILA [41], Video-LLaMA2 [5], Chat-UniVi [29], and Video-LLaVA [40]. InstructBLIP and LLaVA-1.5 primarily focus on image tasks, while VILA and Video-LLaMA2 specialize in video tasks. Chat-UniVi and Video-LLaVA are capable of processing both image and video data, allowing for a comprehensive evaluation across both modalities. More detailed descriptions are provided in Appendix A.

Evaluation Benchmarks.

We conduct evaluations on both image and video benchmarks. For image benchmarks, we assess three categories: (1) Comprehensive benchmarks (MMBench [50], LLaVA^W [45], MM-Vet [84]); (2) General VQA benchmarks (VizWiz [20], SQA [51]); (3) Hallucination benchmarks (POPE [37], CHAIR [59]). For video, we evaluate three zero-shot video understanding datasets: MSRVTT-QA [30], MSVD-QA [75], and ActivityNet-QA [95], along with the Video-Based Text Generation Benchmark for quantitative analysis [54].

Implementation Details.

FarSight supports Greedy, Sampling, and Beam Search decoding strategies, with Greedy decoding used for illustration. Details of the other methods are in Appendix E. For the Decay Factor, we set the sequence length $\left(seq\right)$ to 256 and define the decay rate $\sigma$ in Eq. 8 as $lo{g}_{\alpha }\left(seq\right)$, with $\alpha$ is 1024, the typical maximum token limit. Extensive experiments confirm $seq=256$ ensure stable and consistent generation.

5.2. Abalation Study

Effect of Attention Registers.

We experiment with various register-attention values to assess their impact on attention register performance. As shown in Fig. 5, our FarSight method improves performance by +6.4% and +5.4% on ${\text{CHAIR}}_S$ for LLaVA-1.5 and Video-LLaVA, respectively, significantly outperforming other attention values. In contrast, the causal masking with $-\infty$ restricts attention allocation in the upper triangular matrix, leading to instability in long-distance dependencies and reduced accuracy. Zero-padding fails to absorb excess attention effectively, increasing the risk of hallucinations during text generation. Although a fixed value of 10⁻³ introduces moderate attention absorption, which prevents excessive focus on irrelevant tokens, it still underperforms compared to our method.

Figure 5.

Comparison of different Upper Triangular Attention Values in Attention Registers. (a) and (b) show model performance with varying upper triangular attention values on the CHAIR and POPE-P datasets.

Effect of Positional Awareness Encoding.

We adopt various positional embedding strategies in the attention layer to assess their impact on the hallucination performance. As shown in Table 1, the baseline RoPE [63], FixVPE [52] and EDVT [52] strategies in LLaVA-1.5 and Video-LLaVA result in high hallucination rates. Specifically, RoPE introduces relative positional encoding between visual and text tokens, reducing attention to visual tokens during text generation. Although FixVPE’s fixed positional embeddings enhance the consistency of visual information, they are less effective than EDVT’s equidistant attention strategy. In contrast, our FarSight significantly improves CHAIR performance by using a progressively diminishing causal mask, retaining attention on earlier tokens (e.g., visual tokens).

Table 1.

Comparison of our Positional Awareness Encoding with other methods on the CHAIR [59] and POPE [37] datasets. RoPE: rotary positional embedding for both visual and text tokens, as used in the original MLLMs. FixVPE: fixed rotary embedding for visual tokens only. EDVT: rotary embedding for text tokens only.

Method	$CHAI{R}_S\downarrow$	$CHAI{R}_I\downarrow$	POPE-R ↑	POPE-P ↑
LLaVA-1.5 (RoPE)	48.0	13.9	87.0	82.8
+ FixVPE	47.3	13.4	87.5	84.7
+ EDVT	46.8	14.5	87.8	85.4
+ FarSight (Ours)	41.6 (+6.4)	13.2 (+0.7)	90.5 (+3.5)	86.1 (+3.3)
Video-LLaVA (RoPE)	50.2	15.6	81.6	85.3
+ FixVPE	48.5	14.9	81.9	85.2
+ EDVT	46.8	13.7	82.5	84.7
+ FarSight (Ours)	44.8 (+5.4)	12.9 (+2.7)	83.2 (+1.6)	85.8 (+0.5)

Effect of Decay Factor in Attention Registers.

We investigate the effect of query sequence lengths on attention decay, as shown in Fig. 6. In both the POPE-R and MSRVTT-QA datasets, MLLMs achieve peak accuracy at a sequence length of 256, with performance starting to decline as the sequence length continues to increase. This can be attributed to the decay factor, which is closely linked to the sequence length. Specifically, as defined in Section 5.1 (Implementation Details), the decay factor is influenced by the sequence length and directly affects the rate of attention decay. For shorter sequences, the decay factor rises rapidly, limiting the model’s ability to capture distant context. Conversely, for longer sequences, the decay factor may initially have a less pronounced effect, but as sequence length increases, attention distribution becomes diluted, increasing decay and information redundancy. A moderate sequence length (e.g., 256) effectively balances the decay factor, maintaining optimal focus on key information and preventing dispersion.

Figure 6.

The impact of sequence length on attention decay and the performance of MLLMs on the POPE-R and MSRVTT-QA datasets integrated with our FarSight.

Quantitative Analysis.

We visualize the responses and performance of LLaVA-1.5 across different methods and scenarios. Fig. 7 (a) shows that FarSight achieves higher accuracy in identifying query-relevant key regions than Baseline and EDVT. This improvement results from its dynamic attention register, which reallocates attention to task-related visual information and reduces attention to irrelevant tokens. The long-term decay curves in Fig. 7 (b) show that FarSight maintains strong attention on image tokens in later generation stages, enabled by progressive positional encoding that balances attention between visual and textual tokens throughout the sequence. Fig. 7 (c) shows attention distribution under varying decay rates. As the decay rate increases, the model’s attention becomes progressively more concentrated, reaching optimal focus at a decay rate of 0.8. However, when the decay rate further increases to 0.9, attention starts to disperse. This indicates the importance of a moderate decay rate for balanced attention.

Figure 7.

Qualitative Visualization of FarSight in Image Understanding Task on LLaVA-1.5. (a) Comparison of the average attention allocation to images during text generation among Base (Vanilla MLLMs), EDVT and our FarSight; (b) Visual attention decay across different methods within the generation of 60 text tokens; (c) FarSight’s attention distribution on images under varying decay rat $\sigma$. More detailed visualizations of images and videos are provided in Appendix F.

5.3. Comparison to State-of-the-Arts

GPT-4o Assisted Evaluation.

To comprehensively evaluate the overall quality of generated text, we employ the PPL (Perplexity) metric and utilize GPT-4o to assess the grammar, fluency, and naturalness of the text. We randomly select 600 images from the MSCOCO dataset and perform validation using the LLaVA-1.5 and Video-LLaVA. As demonstrated in Fig. 8, FarSight consistently preserves the quality of the generated text across multiple dimensions.

Figure 8.

The average performance is evaluated on a randomly selected set of 600 images from the MSCOCO dataset. PPL₁ and PPL₂ are calculated using GPT-3.5 Turbo, while the ratings for Grammar, Fluency and Naturalness are provided by GPT-4o.

Image Benchmarks Evaluation.

To evaluate the image understanding, we compare models with the FarSight extension against several decoding methods, including ICD [70], VCD [34] and OPERA [24], as shown in Table 2. Integrating FarSight as a plugin into LLaVA-1.5 results in an average improvement of +2% in the Comprehensive and General VQA tasks. It also achieves significant gains in hallucination metrics, with ${\text{CHAIR}}_S$ and POPE-P scores increasing by +6.4% and +3.3%, respectively. These results indicate that FarSight is effective at reducing hallucinations in both structured and unstructured environments. Furthermore, the benefits of FarSight extend beyond the LLaVA-1.5 model, as other models also experience considerable enhancements, especially in hallucination evaluation tasks, with the ${\text{CHAIR}}_S$ metric increasing.

Table 2.

Comparison of different MLLMs and FarSight across all image benchmarks. Notably, in the Hallucination Benchmark, lower scores on ${\text{CHAIR}}_I$ and ${\text{CHAIR}}_S$ indicate better performance, while higher scores are preferable for other metrics.

Method	Comprehensive Benchmark			General VQA		Hallucination Benchmark
	MMBench ↑	LLaVAW	MM-Vet↑	VizWiz↑	SQA↑	$CHAI{R}_S\downarrow$	$CHAI{R}_I\downarrow$	POPE-R↑	POPE-P↑	POPE-A↑
LLaVA-1.5	64.3	72.5	30.5	48.5	64.5	48.0	13.9	87.0	82.8	76.6
+ICD	63.1	69.7	30.4	46.9	62.8	47.7	13.6	87.9	84.0	80.2
+VCD	63.9	70.9	29.5	43.4	63.3	46.8	13.2	87.0	83.5	78.1
+OPERA	64.4	72.0	31.4	50.0	64.9	45.2	12.7	88.8	82.8	79.2
+ FarSight (Ours)	66.0 (+1.7)	74.7 (+2.2)	32.5 (+2.0)	50.8 (+2.3)	67.4 (+2.9)	41.6 (+6.4)	13.2 (+0.7)	90.5 (+3.5)	86.1 (+3.3)	80.4 (+3.8)
InstructBLIP	43.4	58.2	25.6	33.4	62.1	55.6	24.2	88.7	81.3	74.4
+ FarSight (Ours)	46.5 (+3.1)	61.0 (+2.8)	27.8 (+2.2)	36.0 (+2.6)	63.4 (+1.3)	51.8 (+3.8)	23.0 (+1.2)	89.5 (+0.8)	85.8 (+4.5)	76.7 (+2.3)
Video-LLaVA	60.9	73.1	32.0	48.1	64.6	50.2	15.6	81.6	85.3	86.2
+ FarSight (Ours)	62.8 (+1.9)	74.5 (+1.4)	32.8 (+0.8)	50.3 (+2.2)	66.2 (+1.6)	44.8 (+5.4)	12.9 (+2.7)	83.2 (+1.6)	85.8 (+0.5)	87.1 (+0.9)
Chat-UniVi	56.3	70.4	28.3	46.9	59.9	52.3	16.7	85.1	69.5	64.4
+ FarSight (Ours)	59.8 (+3.5)	72.6 (+2.2)	30.7 (+2.4)	48.2 (+1.3)	62.4 (+2.5)	48.9 (+3.4)	15.2 (+1.5)	87.4 (+2.3)	69.7 (+0.2)	65.3 (+0.9)

Video Benchmarks Evaluation.

In Zero-Shot Video Question Answering Tasks, FarSight achieves significant improvements over video MLLMs across three key benchmark datasets. As shown in Table 3, on the MSRVTT-QA dataset, our method delivers an average accuracy gain of +3% across multiple models, reaching a peak accuracy of 68.9%. On MSVD-QA and ActivityNet-QA datasets, FarSight improves accuracy by +2% and +0.7%, respectively, demonstrating consistent enhancements across different video contexts and question types. Moreover, in Video-Based Text Generation, the integrated model outperforms the baseline MLLMs across five critical dimensions.

Table 3.

Comparison of different Video MLLMs and FarSight across all video benchmarks. In the Video-Based Text Generation Benchmark, five scores are assessed: Cr. (Correctness of Information), Cs. (Consistency), De. (Detail Orientation), Ct. (Contextual Understanding) and Te. (Temporal Understanding). Following Maaz et al. [55], we use the GPT-3.5 Turbo model to assign a relative score to the model outputs, with scores ranging from 0 to 5. See Appendix E for further details.

Method	MSVD-QA		ActivityNet-QA		Video-Based Text Generation
	Accuracy↑	Score↑	Accuracy↑	Score↑	Cr.↑	Cs.↑	De.↑	Ct.↑	Te.↑
Chat-UniVi	64.6	3.6	43.1	3.2	2.84	2.93	2.55	3.16	2.43
+ FarSight (Ours)	66.4 (+1.8)	3.5	43.7 (+0.6)	3.2	2.86	2.94	2.56	3.19	2.48
Video-LLaVA	64.8	3.7	41.5	3.3	2.32	2.34	2.65	2.75	2.09
+ FarSight (Ours)	66.2 (+1.4)	3.6	42.0 (+0.5)	3.5	2.43	2.38	2.93	2.84	2.14
VILA	72.6	4.0	50.2	3.3	3.14	3.40	2.71	3.43	2.58
+ FarSight (Ours)	74.5 (+1.9)	4.2	51.4 (+1.2)	3.6	3.18	3.52	2.73	3.45	2.60
Video-LLaMA2	70.9	3.8	49.9	3.3	3.13	3.23	2.70	3.42	2.45
+ FarSight (Ours)	73.8 (+2.9)	3.9	50.4 (+0.5)	3.6	3.26	3.32	3.21	3.50	2.47

6. Conclusion

In this work, we analyze the self-attention token propagation patterns, revealing two main causes of hallucinations in MLLMs: attention collapse and positional information decay. To mitigate them, we present FarSight, a plug-and-play decoding strategy that reduces interference from outlier tokens and enhances in-context inference. The core of our method is effective token propagation, which is achieved by optimizing the causal mask with attention registers and a diminishing masking rate. Extensive experiments on both image and video tasks have shown that the proposed method outperforms existing state-of-the-art methods, and the ablation study has revealed the effectiveness of our FarSight.