CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning

Abstract

Chart question answering requires multimodal large language models to integrate visual comprehension with logical reasoning, yet current models struggle with accurate visual grounding and coherent reasoning chains. While extrinsic chain-of-thought prompting and visual cues significantly improve performance, current MLLMs lack intrinsic visual grounded reasoning capabilities, leading to inaccurate perception and reasoning disconnected from visual evidence. We propose CURV, a curriculum learning framework that develops intrinsic visual reasoning by reformulating chart question answering as multi-step visual grounded reasoning, where each step coordinates logical reasoning with dynamic visual grounding. To support model learning, we introduce CCQA, a three-level curriculum dataset with scalable synthetic generation across diverse chart types and reasoning patterns. Experiments show gains of up to ↑20.50% over baselines, with generalization to real-world benchmarks (↑12.30%) and out-of-domain multimodal reasoning (↑10.20%).

What Prohibits Chart Understanding?

We evaluate GPT-4o on 60 CQA samples from CharXiv under four generation modes to isolate where failures come from. Reasoning alone helps (↑10.00%), visual grounding alone helps more (↑18.33%), but combining both is transformative (↑43.33%) — and vision errors drop to zero.

Extrinsic assistance helps at every depth

Both models improve when given reasoning or visual grounding, and improve most when given both — even as reasoning depth rises from D = 1 to D = 3.

That consistency is what motivates internalizing these capabilities rather than supplying them from outside at inference time.

CCQA: Curriculum Chart Question Answering

CCQA is generated entirely from human-authored templates, with no model in the loop. Every question, reasoning step, grounding box, and answer is computed deterministically from the plotting data, which is what guarantees ground-truth correctness at scale. Each reasoning step R_t is paired with ground-truth grounding V*_t and a binary mask M*_t.

Three Curriculum Levels, Five Tiers

Foundational Operators

Twelve operators compose in atomic or nested form across chart components and subplots. Nesting depth is what controls curriculum difficulty.

Seven chart types, thirty domains

Bar, histogram, scatter, line, heatmap, pie, and radar. Sample counts differ by type because they depend on chart features — scatter plots key on both axes, heatmaps on cell values and labels — and on the source plotting data.

CURV: Curriculum Visual Grounded Reasoning

Rather than learning the direct mapping f_θ: (I, Q) → A, CURV decomposes chart question answering into a structured progressive chain in which every reasoning step is anchored to a visual region:

Two-Stage Curriculum Training

Visual Grounding Strategies

Three strategies dynamically shift the visual focus as reasoning progresses. All follow the same RVA process — the model emits reasoning with grounded box coordinates — but differ in how the grounded state is rendered back to the model.

Applied — Highlighting the Focus

Predicted regions are underlined with semi-transparent yellow overlays. As reasoning progresses the highlight shifts to mirror each step, preserving full visual context. This is CURV's main method.

I'_vis,t = I_orig ⊙ (1 − κ · M_focus,t) + κ · H_yellow ⊙ M_focus,t

Boxed — Explicit Box Guides

Red rectangular borders mark the focus regions, relocating and resizing as the reasoning chain evolves. Nothing is occluded, but the cue is weaker than a filled highlight.

I'_vis,t = BOX(I_orig, {B_t,i}, C_red, τ)

Cropped — Zooming the Focus

Focus regions are extracted and presented as zoomed sub-images alongside the full chart, maximizing effective resolution at the cost of computation and straightforward comprehension.

I'_vis,t = {I_orig, {CROP(I_orig, B_t,i)}}

Trade-offs Across Strategies

Explicit grounding wins

Explicit visual grounded reasoning consistently outperforms its implicit counterpart by ↑8.78%.

Grounding is most valuable as an intermediate vision–reasoning bridge, not as the ultimate learning objective.

Experiments

Performance on CCQA

Five accuracy metrics per curriculum level: acc@M uses an MLLM judge, and acc@0.0 through acc@0.2 apply progressively relaxed rule-based thresholds.

Ablations

Gains hold across chart types

Bar charts benefit most, followed by line plots and heatmaps. Radar gains least, reflecting its lower prevalence in the curriculum.

Applied, boxed, and cropped show distinct distributions, indicating the grounding strategies suit different chart structures.

Foundations drive balanced gains

Training on levels 1+2 gives the most consistent improvement across all difficulties. Relying on level 3 alone is markedly less effective.

Foundational learning is what makes visual reasoning adaptable rather than narrow.

Multi-chart understanding

On curriculum level 3, CURV improves subplot localization by ↑4.68% and cross-chart relation understanding by ↑2.42%.

Gains come from reasoning grounded in visual structure, not task-specific adaptation.

Resources

Citation

@inproceedings{guo2026curv,
  title     = {CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning},
  author    = {Guo, Xuehang and Zhang, Pingyue and Zhang, Ruiyi and Wang, Zhenhailong and Lyu, Hanrui and Ji, Heng and Sun, Tong and Wang, Qingyun and Li, Manling},
  booktitle = {In Proceedings of Conference on Language Modeling 2026 (COLM 2026)},
  year      = {2026}
}