ChartQA-X: Generating Explanations for Visual Chart Reasoning

Abstract

The ability to explain complex information from chart images is vital for effective data-driven decision-making. In this work, we address the challenge of generating detailed explanations alongside answering questions about charts. We present ChartQA-X, a comprehensive dataset comprising 30,799 chart samples across four chart types, each paired with contextually relevant questions, answers, and explanations. Explanations are generated and selected based on metrics such as faithfulness, informativeness, coherence, and perplexity. Our human evaluation with 245 participants shows that model-generated explanations in ChartQA-X surpass human-written explanations in accuracy and logic and are comparable in terms of clarity and overall quality. Moreover, models fine-tuned on ChartQA-X show substantial improvements across various metrics, including absolute gains of up to 24.57 points in explanation quality, 18.96 percentage points in question-answering accuracy, and 14.75 percentage points on unseen benchmarks for the same task. By integrating explanatory narratives with answers, our approach enables agents to convey complex visual information more effectively, improving comprehension and greater trust in the generated responses.

1. Introduction

Vision language models (VLMs) have made significant progress in addressing complex reasoning tasks [7, 9, 13, 26, 43, 47]. As their capabilities continue to grow, ensuring alignment with human values becomes increasingly critical. However, these models are often viewed as black boxes, producing answers without revealing the reasoning or evidence behind them. Providing explanations helps make their decision-making processes more transparent and interpretable to users. Prior work on visual reasoning for images [11, 49] emphasizes the value of generating answers with explicit rationales to support model predictions. Others [50] leverage large language models for commonsense reasoning, demonstrating improved performance in visual question answering. While recent work has highlighted the value of explanations in vision-language reasoning tasks involving general images [31, 34, 41, 48], little research has explored explanation generation for data visualizations. Chart question answering (ChartQA) is an emerging area focused on developing models that can interpret chart data and answer user queries [18, 19, 29, 30]. However, existing ChartQA models produce answers without the explanatory context needed to support user understanding and build trust. This limitation reduces their practical utility, as users often seek not only accurate answers but also clear and interpretable reasoning behind them.

Prior research often treats question answering and explanation generation as separate tasks [15, 17, 21, 41]. Some approaches use one model to generate answers and a different model to produce explanations. Others focus solely on generating explanations without grounding them in the reasoning process that leads to an answer [20, 24, 46]. This separation can disrupt the chain-of-thought (CoT) reasoning and result in inconsistencies between answers and their explanations. For example, a model might produce a correct explanation but an incorrect answer, or vice versa. These misalignments underscore the need for models that generate contextually relevant explanations in tandem with answering questions about chart data. Inspired by prior work [8, 39], we eliminate the need for separate models for answer prediction and explanation generation by using a single, general-purpose vision-language model to handle both tasks. This unified approach ensures that explanations are closely aligned with the predicted answers while also reducing memory usage and inference time, as shown in [39].

Motivated by this goal, we introduce ChartQA-X, a comprehensive dataset comprising 30,799 chart samples (28,299 train and 2,500 test), each paired with contextually relevant questions, answers, and explanations. Explanations are selected from the outputs of six state-of-the-art vision-language models. Figure 1 illustrates the ChartQA-X task setup, demonstrating how the dataset enables a model to generate both the answer and an explanation for a given chart-based question. While recent VLMs can generate explanations, their quality varies widely across questions and chart types. Without additional supervision, outputs are often verbose, ungrounded, or inconsistent. To ensure high-quality explanations in ChartQA-X, we simplify the task during dataset creation by supplying full context (e.g., answer, data table) and generating diverse candidate explanations with multiple VLMs. These are then filtered using automatic reasoning metrics to retain only high-quality explanations. ChartQA-X includes a diverse range of chart types, including (1) horizontal bar charts, (2) vertical bar charts, (3) line charts, and (4) pie charts, all derived from the original ChartQA dataset [29]. We assess ChartQA-X through an online human-subject study with 245 participants who rate ChartQA-X explanations as comparable to or better than human-written ones in terms of accuracy, clarity, logic, and overall quality.

Figure 1.

ChartQA-X dataset enables training VLMs capable of generating both answers and explanations in response to user questions about charts.

By releasing a validated dataset, ChartQA-X enables benchmarking and facilitates progress in VLM explanation generation for chart questions. To establish performance baselines, we fine-tune three state-of-the-art visionlanguage models, i.e., LLaVA 1.6 [26], Qwen2-VL [43], and InternVL-2.5 [6], on the ChartQA-X dataset and evaluate them in terms of the quality of generating explanations and accuracy of question answering. We further assess their question-answering capabilities on unseen benchmark chart datasets, including DVQA [18], PlotQA [30], and FigureQA [19]. In summary, our contributions are as follows:

• We introduce ChartQA-X, the largest dataset containing 30,799 chart samples with detailed explanations spanning diverse chart types.

• We provide empirical validation of the dataset through an online study involving 245 human participants, showing that ChartQA-X explanations are comparable to or surpass human-written explanations across four metrics.

• Models fine-tuned on ChartQA-X demonstrate significant improvements across various metrics, including an absolute gain of up to 24.57 points in explanation quality, 18.96 percentage points in question-answering accuracy, and 14.75 percentage points on unseen benchmark datasets for the same task.

2. Related Work

2.1. Explainable AI for Visual Data

Early work on natural language explanations focuses on vision tasks such as image classification [15]. This line of research is later extended to language tasks [4, 17, 32, 38] and vision-language tasks, including visual question answering (VQA) [20, 24, 34, 46]. For example, Park et al. [34] introduce two datasets: VQA-X and ACT-X. VQA-X augments the VQAv2 dataset [2] with natural language explanations that justify the answers provided by VQA models, while ACT-X provides explanations for activity recognition tasks. Marasović et al. [28] use separate models to extract visual features from images, including object bounding boxes, role boxes with coordinates, and VisualCOMET embeddings [35]. These features, along with the question and ground truth answer, are fed into GPT-2 [36] to generate an explanation. However, this approach treats question answering and explanation generation as separate processes. NLX-GPT [39] addresses this limitation by using ResNet-101 [14] and ViT [10] to extract image features, which are then combined with the question and answer into a single input sequence. This sequence is passed to a distilled version of the GPT-2 to generate both the answer and explanation. Similarly, we also train a single model to perform both tasks in a unified manner, but we focus on chart data.

2.2. Chart-Based Question Answering

Chart-based question answering has received increasing attention, leading to the development of several benchmark datasets, including FigureQA [19], DVQA [18], LEAF-QA [5], and LEAF-QA++[40]. FigureQA and DVQA feature charts generated from synthetic data, whereas LEAF-QA and LEAF-QA++ are based on real-world data. These datasets typically rely on a small set of question templates, and answers are limited to a fixed vocabulary. In contrast, PlotQA[30] introduces open-vocabulary questions that often require performing mathematical operations on chart data. ChartQA [29] expands the scope of chart QA by providing a large-scale dataset with 9,608 human-written questions and 23,111 automatically generated ones derived from chart summaries using the T5 model [37]. In terms of explanation, Kim et al. [21] present a small, manually constructed dataset comprising 52 charts and 629 question–answer pairs to examine how humans reason and explain when answering chart-related questions. This remains one of the few publicly available explanation datasets in this domain. Building on this prior work, we introduce the largest explanation dataset to date for chart question answering.

3. ChartQA-X

Figure 2 illustrates the process of constructing the ChartQA-X dataset, which involves four main steps: (1) preparing the input data, (2) generating explanations using six VLMs, (3) evaluating and selecting the best explanation for each question-answer based on various metrics, and (4) verifying the correctness of the selected explanations. We describe each of these steps in detail below.

Figure 2.

ChartQA-X is constructed in four stages: (1) preparing input data, (2) generating explanations using six VLMs, (3) selecting high-quality explanations based on ROSCOE scores, and (4) verifying explanation correctness. The dataset is evaluated through three types of experiments: (a) human-subject studies, (b) benchmark evaluations using accuracy and text generation metrics, and (c) generalizability tests on unseen datasets.

3.1. Input Data

ChartQA-X is built upon the ChartQA dataset [29], which consists of 18,317 chart images and 30,799 question–answer pairs. It spans four chart types and features a diverse range of descriptive and reasoning questions (Figure 3) with varying levels of complexity. Reasoning questions require calculation using chart data, while descriptive questions involve directly reading values from the chart. Each data point in the ChartQA-X dataset consists of four components: (1) Chart Image (I): A visual representation of the data, which may take the form of horizontal bar charts, vertical bar charts, line charts, or pie charts. The image provides visual context for the model, enabling it to interpret the data and incorporate visual attributes, such as color and shape, into its explanations. (2) Data Table (D): A structured, tabular representation of the chart’s underlying data, offering precise numerical values and categorical labels. This component complements the visual chart by providing the chart data in a machine-readable format. (3) Question (Q): A natural language query about the information presented in the chart. Questions span a variety of types, including trend analysis, comparisons, arithmetic reasoning, and specific data retrieval, serving as the primary prompt for the model. (4) Answer (A): The correct response to the question, derived from the chart and its associated table. This is the expected output the model must produce, and the explanation is evaluated based on how well it supports this answer.

Figure 3.

Distribution of (a) chart types, and (b) question types in the dataset.

3.2. Explanation Generation

We generate explanations for each question-answer pair using six different VLMs: LLaVA 1.6 [26], Phi-3 [1], CogVLM [44], Deepseek-VL [27], Qwen2-VL [43], and GPT-4o [16]. Each model is prompted to generate explanations in a chain-of-thought reasoning format [45], which breaks down the reasoning process into a sequence of logical steps, making the model’s thought process transparent. This involves analyzing the chart to identify key data points, trends, and relationships, aligning the relevant information with the question, and constructing a coherent narrative that links the data to the final answer. Our final prompt is based on iterative experimentation with several prompt designs to elicit high-quality explanations from VLMs. Our early prompt versions include a fixed set of sub-questions to guide step-wise reasoning, such as “What values are shown?” or “What comparisons are relevant?”. While structured, this approach lacks flexibility and performs poorly across diverse question types. We next attempt an open-ended prompt that instructs the model to generate its own sub-questions before answering them. This produces longer outputs, but often includes irrelevant or redundant steps and conversational phrasing. To improve clarity and relevance, our final prompt is designed following HCI literature to emphasize concise reasoning with visual references like human explanations [21], limit responses to four sentences, and avoid conversational language. Specifically, we design the following prompt:

Think like a human to arrive at the given answer. Generate an explanation (no more than four sentences) that outlines the steps taken to derive the answer using chain-of-thought reasoning. Focus on visual elements in the chart, such as color, height, position, and labels, and avoid conversational language or unnecessary commentary.

We note that explanation generation during dataset construction is not equivalent to explanation generation at inference time. During this phase, we provide models with full context (the chart, question, answer, and data table), which significantly lowers task complexity. Furthermore, to reduce model-specific bias, we sample explanations from six different VLMs. As a result, the final dataset is composed of explanations not tied to a single model’s reasoning style. This strategy mitigates circularity and enables models to learn from superior or complementary reasoning patterns present across models.

3.3. Evaluating and Selecting Explanations

To assess the quality and effectiveness of the explanations, we use the ROSCOE evaluation suite [12], which is specifically designed to score step-by-step reasoning. ROSCOE offers a comprehensive framework that evaluates reasoning quality across nine critical dimensions. In our setup, we define the source context as $s=\left\{s_1,\dots ,s_T\right\}$, a sequence of T sentences consisting of the instruction followed by the question. The hypothesis is denoted as $h=\left\{h_1,\dots ,h_N\right\}$, a sequence of N reasoning steps, including the final explanation. Reasoning Alignment (r–align) measures the degree to which each step in a multi-step explanation aligns with the source context (i.e., question). Alignment for each step is computed as:

$$r\text{-align}\left(h_i\to s\right)=\frac{\left[1+{\text{max}}_{j=1}^T(\text{cos}\left(h_i,s_j\right)\right]}{2}$$ (1)

yielding a vector $\in {\left[\mathrm{0,1}\right]}^N$. This reflects the semantic closeness of each reasoning step to the source. Contradiction Probability (p_contradicts) is estimated using a Natural Language Inference (NLI) model, in this case a fine-tuned DeBERTa-v3 [23], that classifies explanation-source pairs into entailment, neutral, or contradiction. The p_contradicts score is the model’s confidence that the explanation contradicts the source, serving as a proxy for logical inconsistency. English grammatical acceptability p_gram is scored by a classifier model [22]. Table 1 provides the equations for each of the nine ROSCOE metrics.

Table 1.

Equations for calculating ROSCOE metrics [12].

Metric	Equation
FS	$\frac{1}{N}{\sum }_{i=1}^{N}r\text{-align}\left(h_i\to s\right)$
FT	$\frac{1}{N+M}{\sum }_{i=1}^N\left[r\right.\text{-align}\left.\left(h_i\to s\right)\right]+{\sum }_{j=1}^{M_i}r{\text{-align}}^{\text{token}}\left(h_{i,j}\to s\right)$
IS	$\left(\frac{1}{T}{\sum }_{t=1}^{T}r\right.\text{-align}\left(s_t\to h\right)+\frac{1}{N}{\sum }_{i=1}^{N}r\text{-align}\left.\left(h_i\to s\right)\right)/2$
IC	$(1+\text{cos}(h,s\left)\right)/2$
SRC	$1-{\text{max}}_{i=1\dots N}{\text{max}}_{j=1\dots T}{p}_{\text{contradicts}}\left(h_i,s_j\right)$
SFC	$1-{\text{max}}_{i=2\dots N}{\text{max}}_{j<i}{p}_{\text{contradicts}}\left(h_i,h_j\right)$
PS	$\frac{1}{\frac{1}{N}{\sum }_{i=1}^N\text{PPL}\left(h_i\right)}$
PC	$\frac{1}{\text{PPL}(h)}$
GS	$\frac{1}{N}{\sum }_{i=1}^N{p}_{\text{gram}}\left(h_i\right)$

1. Faithfulness Step (FS) evaluates whether the model misinterprets the problem statement or if the reasoning chain is vague, irrelevant, or misuses information. It measures the alignment (r–align) between each step of the hypothesis (h_i) and source context (s) and is computed as the mean alignment score across all steps.

2. Faithfulness Token (FT) measures token-level similarity based on embedding alignment. Given M_i tokens in step h_i, where h_i,j is the jth token in the ith step, r–align^token denotes the alignment vector from tokens in h_i to all tokens in the source context s.

3. Informativeness Step (IS) measures how effectively each reasoning step uses information from the source context.

4. Informativeness Chain (IC) quantifies how well the overall reasoning chain (h) aligns with the source context (s).

5. Source-Consistency (SRC) measures logical entailment errors between the generated reasoning h and the source context s.

6. Self-Consistency (SFC) measures logical entailment errors within a reasoning step, capturing its internal coherence.

7. Perplexity Step (PS) is the average perplexity (PPL) of all tokens in the generated reasoning steps, where each token is scored using only the preceding tokens within the same step. Language coherence is evaluated using the GPT2-Large model [36] to compute PPL.

8. Perplexity Chain (PC) is the average perplexity (PPL) of all tokens in the generated reasoning steps.

9. Grammar Step (GS) evaluates the grammatical correctness of each individual step in an explanation, focusing on sentence structure, verb tense, punctuation, and other grammatical aspects within each reasoning unit.

Each explanation generated by the six VLMs is evaluated using the nine ROSCOE metrics. Since different models perform best on different subsets of the dataset, we select, for each sample, the explanation with the highest overall score across all metrics. To ensure fair comparison, all metric scores are normalized to a 0–1 scale, with 1 indicating the best performance and 0 indicating the worst. For perplexity-based metrics (PS and PC), where lower is better, scores are inverted by subtracting them from 1. We then apply weights between 0 and 1 to each metric to compute an aggregate score. To prioritize alignment between the explanation, question, and answer, we assign higher weights to faithfulness and informativeness metrics and lower weights to others. We set the weights as follows: 0.2 for the faithfulness step, 0.15 each for the faithfulness token and the informativeness chain, 0.1 for self-consistency and source-consistency, and 0.05 each for the perplexity step, perplexity chain, and grammar step. This weighting scheme emphasizes explanation quality and relevance while also considering coherence, fluency, and grammatical accuracy. To evaluate the stability of this approach, we conduct a sensitivity analysis using a variety of alternative weight configurations, including uniform, random, and metric-focused schemes. The results, presented in Appendix A.3, show that while weighting affects selection, our choices align well with other meaningful configurations.

We compute an Aggregate Score (AS) for each explanation as a weighted sum of all normalized metric scores. For each sample, we select the explanation with the highest Aggregate Score (AS) among the six models, resulting in a high-quality collection of explanations. Table 2 presents the average ROSCOE scores across all ChartQA samples for each of the six VLMs.

Table 2.

ROSCOE scores for six VLMs on the ChartQA-X dataset. Faithfulness Step (FS), Faithfulness Token (FT), Informativeness Step (IS), Informativeness Chain (IC), Source-Consistency (SRC), Self-Consistency (SFC), Perplexity Step (PS), Perplexity Chain (PC), Grammar Step (GS), and Aggregate Score (AS).

Models	FS	FT	IS	IC	SRC	SFC	PS	PC	GS	AS
Phi-3 [1]	0.845	0.933	0.860	0.943	0.673	0.665	0.993	0.916	0.943	0.856
CogVLM [44]	0.850	0.929	0.852	0.937	0.607	0.587	0.988	0.937	0.937	0.840
Deepseek-VL [27]	0.845	0.933	0.860	0.943	0.673	0.665	0.993	0.916	0.943	0.856
LLaVA 1.6 [26]	0.863	0.935	0.860	0.956	0.737	0.552	0.989	0.882	0.956	0.855
Qwen2-VL [43]	0.863	0.936	0.867	0.952	0.737	0.655	0.984	0.897	0.960	0.867
GPT-4o [16]	0.840	0.937	0.883	0.958	0.791	0.773	0.993	0.948	0.921	0.884

3.4. Verifying Correctness

To verify the correctness of the top-scoring explanations, we leverage the same six state-of-the-art vision-language models: LLaVA 1.6, Phi-3, CogVLM, Deepseek-VL, Qwen2-VL, and GPT-4o. For each explanation generated by a given model, the remaining five models serve as independent verifiers. Each verifier receives the input chart, question, ground-truth answer, data table, and candidate explanation, and produces a binary judgment, Correct or Incorrect. An explanation is accepted into the ChartQA-X dataset if a majority (i.e., at least three out of five) of the verifying models label it as correct. If the explanation fails to reach majority approval, we move to the next highest-scoring explanation based on its ROSCOE score and repeat the same verification process. This iterative evaluation continues until an explanation is accepted. If none of the initial explanations are accepted, we regenerate the candidate explanations from all six models and restart the process. Figure 4(a) shows the contribution of each model to the ChartQA-X dataset, while Figure 4(b) presents the word count distribution of the accepted explanations. See Appendix A.2 for the distribution of explanation word counts across the six VLMs.

Figure 4.

Distribution of explanations in ChartQA-X including (a) percentage of explanations obtained from each VLM, (b) lengths of explanations, and (c) Box plot comparing Human and ChartQA-X performance across four evaluation metrics: Accuracy, Clarity, Logic, and Overall Quality. Each pair of boxes represents the distribution of scores for a specific metric, highlighting differences in performance and variance.

4. Comparison to Human Explanations

To assess the quality of ChartQA-X explanations in a realistic setting, we conducted a human-subject study involving 245 participants that compared ChartQA-X and human-written explanations. We selected 180 samples from the ChartQA-X test set by sampling 30 instances from each of the six categories, defined by chart type (horizontal bar, vertical bar, pie, and line charts) and question type (descriptive and reasoning). This ensured a balanced coverage across formats and question types.

To collect human-written explanations, we recruited 10 graduate students outside the author team, each of whom was assigned 18 unique samples. These samples were carefully curated to include 3 examples from each of the 6 categories, ensuring comprehensive exposure to the dataset’s diversity. The instructions closely mirrored the VLM’s input prompt: “Write an explanation (no more than four sentences) that outlines the steps taken to derive the answer. Focus on visual elements in the chart, such as color, height, position, and labels, and avoid conversational language or unnecessary commentary.” Each human-written explanation was then paired with the corresponding ChartQA-X explanation, resulting in 180 paired comparison instances.

Each explanation pair was evaluated on Amazon Mechanical Turk by two different workers, with a total of 245 unique workers completing 360 tasks. Each task included a chart image, a data table, a question, an answer, and two unlabeled explanations (human, ChartQA-X), shown in random order. Workers rated each explanation using 1–7 Likert scales (1: poor, 4: neutral, 7: excellent) on four criteria: (a) Accurate: The explanation correctly reflects the chart data and supports the given answer, (b) Clear: The explanation is easy to understand and free from ambiguity, (c) Logical: The reasoning follows a coherent and sensible progression, and (d) Overall Quality: A holistic score accounting for usefulness, clarity, accuracy, and coherence.

Figure 4(c) shows the overall results of our user study. As seen in the figure, ChartQA-X explanations perform on par with human-written ones and slightly outperform them on average across all four criteria. In particular, they score significantly higher in accuracy (p = 0.035) and logical coherence (p = 0.043). These results suggest that ChartQA-X can generate high-quality explanations that closely match and sometimes surpass human reasoning. Example explanations and user ratings are provided in Appendix A.5.

5. Experiments and Evaluation

5.1. Baselines and Metrics

We evaluate six open-source models, including LLaVA 1.6 [26], Qwen2-VL [43], InternVL-2.5 [6], Deepseek-VL [27], Phi-3 [1], and CogVLM [44], and a proprietary model, GPT-4o [16], on the ChartQA-X test set. We also fine-tune LLaVA 1.6, Qwen2-VL, and InternVL-2.5 on the ChartQA-X training set to jointly generate both the answer and an accompanying explanation in a single output sequence, establishing three baseline models. Each training instance is formatted as: “Answer: <answer>. Explanation: <explanation>”. The model is trained using a standard causal language modeling (LM) loss applied uniformly across the entire output sequence, without assigning distinct weights to the answer and explanation components. This joint generation approach promotes logical consistency between the answer and its explanation, enhancing interpretability, and reducing the risk of error propagation commonly associated with two-stage (answer-then-explanation) methods. At inference time, the model is prompted with:

Please answer the following question using the format below: The answer is: <answer>. Explanation: <concise, logical explanation justifying the answer>.

This prompt is followed by the chart image and the question. All experiments are conducted with a batch size of 1 on 4 NVIDIA A100 (80GB) GPUs. Fine-tuning is performed on 8 NVIDIA H100 (80GB) GPUs and takes approximately six hours. Inference time on the test set is about one hour for all models, except CogVLM, which requires roughly 2.5 hours. For consistency, we use 7B-parameter versions of all models, except InternVL, which has 8B. We assess model performance using three criteria: (1) accuracy on the question-answering task, (2) explanation quality based on ROSCOE and n-gram metrics, including BLEU-4 [33], METEOR [3], ROUGE-L [25], and CIDEr [42], and (3) generalizability to unseen benchmarks after fine-tuning on ChartQA-X.

5.2. Results

Table 3 demonstrates the impact of fine-tuning on ChartQA-X across various chart and question types without data table in input. Fine-tuned models significantly outperform their base counterparts. InternVL-2.5 fine-tuned on ChartQA-X achieves the highest overall accuracy of 79.78%. Substantial gains are observed across all chart types, with improvements of up to 18.96 percentage points (Qwen2-VL on line charts). On the reasoning subset, which is typically more challenging, ChartQA-X fine-tuning leads to notable boosts. For instance, Qwen2-VL improves from 34.33% to 48.01%. Even models with strong initial performance (e.g., InternVL-2.5) benefit, indicating the broad effectiveness of ChartQA-X. These results highlight ChartQA-X’s value in enhancing both factual and inferential understanding of visual charts. While prior papers report higher accuracy on the original ChartQA benchmark, our evaluation shows lower performance due to a key difference in our setup: we do not provide the underlying data tables as input, which makes the task significantly more challenging. As shown in Table 4, incorporating the data table in the input substantially improves performance, with Qwen2-VL and LLaVA-1.6 achieving nearly twice their previous accuracy.

Table 3.

Question answering accuracy (%), without data table in the input, calculated for different chart and question types (test set only). Best scores are in bold, and second-best scores are underlined.

Models	Chart Types				Question Types		Overall
	Horizontal Bar	Vertical Bar	Pie	Line	Descriptive	Reasoning
State-of-the-Art Models
GPT-4o [16]	75.03	71.23	82.56	66.33	70.15	75.11	73.40
Deepseek-VL [27]	52.01	50.09	46.51	45.14	55.52	42.54	48.64
CogVLM [44]	56.53	54.59	54.07	48.63	56.69	51.19	53.62
Phi-3 [1]	74.51	79.90	77.33	69.58	81.13	70.70	75.53
Models Fine-tuned on ChartQA-X
LLaVA-1.6 [26]	23.93	17.50	34.30	23.44	22.46	20.56	23.53
+ ChartQA-X	35.32 +11.39	29.55 +12.05	47.67 +13.37	28.43 +4.99	34.75 +12.29	29.57 +9.01	34.22 +10.69
Qwen2-VL [43]	42.82	36.57	50.58	34.16	43.01	34.33	40.25
+ ChartQA-X	53.56 +10.74	55.46 +18.89	54.07 +3.49	53.12 +18.96	59.69 +16.68	48.01 +13.68	53.99 +13.74
InternVL-2.5 [7]	75.81	76.43	73.26	68.83	80.25	68.23	73.80
+ ChartQA-X	80.98 +5.17	82.76 +6.33	84.30 +11.04	70.57 +1.74	83.69 +3.44	76.35 +8.12	79.78 +5.98

Table 4.

Question answering accuracy (%), with data table included in the input, calculated for different chart and question types (test set only). Best scores are in bold, and second-best scores are underlined.

Models	Chart Types				Question Types		Overall
	Horizontal Bar	Vertical Bar	Pie	Line	Descriptive	Reasoning
State-of-the-Art Models
GPT-4o [16]	76.45	76.43	84.88	70.07	75.74	76.31	76.64
Deepseek-VL [27]	59.07	62.26	56.98	52.37	68.20	48.87	57.96
CogVLM [44]	60.74	61.74	65.12	50.62	63.68	55.40	59.55
Phi-3 [1]	74.65	82.96	74.42	72.32	84.47	70.56	76.56
Models Fine-tuned on ChartQA-X
LLaVA-1.6 [26]	48.39	55.74	53.49	42.14	57.40	43.73	50.15
+ ChartQA-X	60.36 +11.97	62.70 +6.96	63.95 +10.46	52.87 +10.73	67.83 +10.43	51.83 +8.10	59.92 +9.77
Qwen2-VL [43]	72.90	77.61	72.49	65.83	82.02	62.94	72.30
+ ChartQA-X	74.90 +2.0	80.95 +3.34	77.91 +5.42	67.83 +2.0	85.72 +3.70	66.20 +3.26	75.59 +3.29
InternVL-2.5 [7]	75.42	80.35	76.16	71.57	85.65	67.07	76.04
+ ChartQA-X	79.28 +3.86	83.74 +3.39	83.72 +7.56	72.82 +1.25	87.35 +1.70	72.65 +5.58	79.93 +3.89

Table 5 further shows that fine-tuned models achieve significantly higher ROSCOE scores on the ChartQA-X test set. Table 6 shows that fine-tuning significantly improves explanation quality on the ChartQA-X test set, as measured by n-gram metrics. For example, Qwen2-VL shows significant absolute gains on CIDEr (+24.57), BLEU-4 (+14.01), and METEOR (+9.13), while InternVL-2.5 improves by +20.78, +10.76, and +4.75, respectively.

Table 5.

ROSCOE scores on the ChartQA-X test set without data table in the input. Best scores are in bold, and second-best scores are underlined. FS: Faithfulness Step, FT: Faithfulness Token, IS: Informativeness Step, IC: Informativeness Chain, SRC: Source-Consistency, SFC: Self-Consistency, PS: Perplexity Step, PC: Perplexity Chain, GS: Grammar Step, and AS: Aggregate Score.

Models	FS	FT	IS	IC	SRC	SFC	PS	PC	GS	AS
State-of-the-Art Models
Phi-3 [1]	0.865	0.939	0.873	0.925	0.678	0.792	0.995	0.947	0.946	0.875
CogVLM [44]	0.873	0.938	0.887	0.919	0.471	0.714	0.997	0.997	0.937	0.851
Deepseek-VL [27]	0.871	0.939	0.884	0.921	0.311	0.722	0.996	0.928	0.928	0.832
GPT-4o [16]	0.882	0.941	0.892	0.916	0.803	0.850	0.972	0.972	0.937	0.899
Models Fine-tuned on ChartQA-X
InternVL-2.5 [7]	0.862	0.939	0.868	0.918	0.715	0.722	0.995	0.959	0.951	0.870
+ ChartQA-X	0.894 +0.032	0.947 +0.008	0.877 +0.009	0.922 +0.004	0.797 +0.082	0.853 +0.131	0.994 −0.001	0.963 +0.004	0.958 +0.007	0.901 +0.031
LLaVA 1.6 [26]	0.884	0.945	0.877	0.901	0.515	0.858	0.989	0.991	0.941	0.869
+ ChartQA-X	0.928 +0.044	0.949 +0.004	0.894 +0.017	0.918 +0.017	0.609 +0.094	0.920 +0.062	0.997 +0.008	0.996 +0.005	0.980 +0.039	0.901 +0.032
Qwen2-VL [43]	0.868	0.941	0.868	0.918	0.674	0.871	0.983	0.995	0.964	0.884
+ ChartQA-X	0.907 +0.039	0.953 +0.012	0.884 +0.016	0.963 +0.045	0.902 +0.228	0.918 +0.047	0.997 +0.014	0.997 +0.002	0.970 +0.006	0.932 +0.048

Table 6.

Evaluation results on n-gram metrics for the Chart-QA test set. Best scores are in bold, and second-best scores are underlined.

Models	BLEU-4	METEOR	ROUGE-L	CIDEr
State-of-the-Art Models
GPT-4o [16]	6.98	17.81	28.56	10.17
Deepseek-VL [27]	1.18	6.58	8.02	7.16
CogVLM [44]	1.03	9.91	19.22	3.58
Phi-3 [1]	12.83	20.76	30.06	30.18
Models Fine-tuned on ChartQA-X
LLaVA 1.6 [26]	11.07	16.40	26.44	29.19
+ ChartQA-X	17.95 +6.88	23.60 +7.2	33.52 +7.08	33.83 +4.64
Qwen2-VL [43]	6.49	15.18	25.81	12.31
+ ChartQA-X	20.50 +14.01	24.31 +9.13	34.69 +8.88	36.88 +24.57
InternVL-2.5 [6]	9.61	18.96	32.03	18.94
+ ChartQA-X	20.37 +10.76	23.71 +4.75	34.53 +2.5	39.72 +20.78

Finally, we evaluate the models’ QA accuracy on a random subset of 20,000 samples from three other chart datasets, including DVQA [18], PlotQA [30], and FigureQA [19]. As shown in Table 7, models fine-tuned on ChartQA-X show significant improvements up to +14.75%. Qwen2-VL performs best, reaching 97.32% on DVQA and 95.47% on FigureQA. Notably, FigureQA and PlotQA include dot-line plots, a chart type not present in ChartQA-X, highlighting the models’ ability to generalize to unseen chart types. ROSCOE score improvements for these datasets are reported in Appendix A.4.

Table 7.

Question answering accuracy on three other chart datasets. Best scores are in bold, and second-best scores are underlined.

Models	DVQA	PlotQA	FigureQA
State-of-the-Art Models
Deepseek-VL [27]	51.10	57.59	53.76
CogVLM [44]	43.89	51.43	82.49
Phi-3 [1]	89.67	76.75	70.51
Models Fine-tuned on ChartQA-X
LLaVA 1.6 [26]	59.69	40.02	53.79
+ ChartQA-X	65.57 +5.88	42.85 +2.83	56.70 +2.91
InternVL-2.5 [6]	72.57	57.57	64.64
+ ChartQA-X	76.91 +4.34	68.44 +10.87	74.09 +9.45
Qwen2-VL [43]	82.57	68.13	85.70
+ ChartQA-X	97.32 +14.75	78.14 +10.01	95.47 +9.77

6. Conclusion

We introduce ChartQA-X, the largest dataset of visual chart questions paired with detailed explanations. Alongside this dataset, we propose a novel carefully-designed explanation generation pipeline that leverages multiple models and evaluates outputs using ROSCOE metrics. Human evaluation indicates that ChartQA-X explanations are rated comparable to or better than human-written explanations across key dimensions of accuracy, clarity, logic, and overall quality. Furthermore, models fine-tuned on ChartQA-X significantly outperform existing baselines in generating chart explanations. These models also achieve higher accuracy in answering questions on external chart datasets, demonstrating strong generalization and robustness.