Hello, I'm Ziqiang Shi from Fujitsu Research & Development Center (FRDC), belonging to Brain Intelligence Group of Physical AI Lab.. I attended the 28th International Conference on Pattern Recognition (ICPR 2026) in Lyon, France, from August 17 to 22, 2026. At the conference, I was honored to give an oral presentation of our paper, Eliminating Object Hallucination in MLLMs via Convex Potential Flow Intervention. In this article, I share what the conference was like, introduce our Convex Potential Flow (CPF) method, and highlight the results that drew the most attention from the audience.
ICPR 2026: A Brief Overview
The 28th International Conference on Pattern Recognition (ICPR 2026), the flagship conference of the International Association for Pattern Recognition (IAPR), was held from August 17–22, 2026, at the International Convention Center in Lyon, France. The conference received 1,432 paper submissions and ultimately accepted 715 papers for presentation (an acceptance rate of approximately 49.9%). Of these accepted papers, fewer than approximately 25% were selected for oral presentation, with the remainder assigned to poster sessions. We were proud that our paper, Eliminating Object Hallucination in MLLMs via Convex Potential Flow Intervention, was among those selected for an oral presentation.
Overall, ICPR 2026 reaffirmed its position as a leading international forum for advancing pattern recognition research and fostering cross-disciplinary collaboration.
Our Oral Presentation
In this work, we propose Convex Potential Flow (CPF), a training-free, plug-and-play inference-time intervention for eliminating object hallucination in MLLMs—the problem where models confidently describe objects that do not exist in the input image. CPF formulates hallucination correction as an optimal transport problem between hallucinatory and truth-aligned activation manifolds, and steers attention activations back to trustworthy states along token-specific optimal paths in a single forward pass—without any retraining. Standing at the podium in Lyon, I walked the audience through why MLLMs "see" what isn't there, and how a single convex potential function can steer their attention activations back to reality. After the talk, attendees raised many engaging questions—about how CPF selects the most discriminative attention heads, the overhead of training the convex potential, and whether CPF can be combined with decoding-time methods. These candid exchanges are one of the greatest benefits of presenting in person.
The Core Issue: When MLLMs "See" What Isn't There
MLLMs such as LLaVA and Qwen-VL have revolutionized how we interact with visual content. However, behind their remarkable capabilities lies a hidden vulnerability: they frequently generate descriptions of objects that simply do not exist in the input image. For example, a model may look at a bathroom containing only a toilet and confidently claim there is a "window," or assert that "fried chicken legs are low in calories," ignoring visual evidence in favor of flawed linguistic associations.
The root cause lies in the internal mechanics of Transformers. Strong language priors cause the decoder to over-rely on linguistic context, neglecting fine-grained visual features. As information propagates through the layers, factually consistent activations drift into "hallucinatory manifolds." In high-stakes scenarios like medical diagnosis and autonomous driving, such confident-but-wrong outputs pose serious safety and trust risks.
Why existing methods fall short 1. Training-based methods (e.g., RLHF, DPO) incur massive computational costs and can introduce likelihood shifts that degrade reasoning. 2. Decoding-time strategies (e.g., VCD, OPERA) add inference latency and use non-representative perturbations. 3. Activation engineering (e.g., ICT, VTI) avoids retraining but applies uniform intervention directions across all tokens—erroneously assuming hallucination manifolds are globally translatable, and ignoring token-specific positional dependencies.
Introducing CPF: Steering Activations via Convex Potential Flow
To overcome these limitations, we proposed Convex Potential Flow (CPF), a training-free, plug-and-play intervention that operates during inference. CPF frames hallucination mitigation as an Optimal Transport (OT) problem between manifolds of hallucinated and truth-aligned activations. By constructing a convex potential field, CPF guides attention activations along token-specific optimal paths from hallucination-prone to truth-aligned distributions in a single step.
But how do we actually obtain a "flow" that carries a hallucinatory activation to a trustworthy one—and how is it applied inside the model? The full recipe has four steps.
Step 1: Manufacture hallucination–trusted activation pairs—using blur.
The key trick is that blurring is a controlled way to destroy visual evidence: when the image is blurred, the model can no longer rely on pixels and falls back on language priors—exactly the condition that breeds hallucinations. So for each image, we run both the original (sharp) image and a blurred variant through the frozen MLLM and record the attention activations layer by layer, head by head. The sharp run yields trusted activations z₁, the blurred run yields hallucinatory activations z₀, and together they form a contrastive pair (z₀, z₁). We do this at two granularities: - Image level: blur the entire image, producing pairs that capture global hallucination behavior. - Object level: use Grounding DINO to localize each object and blur only those regions, producing fine-grained pairs that capture object-specific hallucinations (wrong counts, colors, attributes).
Step 2: Locate the culpable heads.
Not every attention head participates in hallucination. We train a lightweight binary classifier (hallucinatory vs. trusted) on the collected activations and use it to rank heads by how discriminative their activations are; only the top-ranked heads (the top 256 in our experiments) will later receive intervention. This answers where to correct.
Step 3: Learn the hallucination→trusted map as a convex potential flow.
With the paired samples in hand, we train a convex potential function Ψ—parametrized by an input-convex neural network (ICNN)—via optimal flow matching, so that its gradient maps each hallucinatory activation to its trusted partner: ∇Ψ(z₀) ≈ z₁. By optimal transport theory (the Brenier theorem), this gradient is exactly the optimal transport map between the two activation distributions; and because the potential is convex, the trajectories it induces are straight lines, so one single step suffices at inference. The correction vector is then simply the displacement prescribed by the learned flow at the current activation:
S(x) = ∇Ψ(x) − x
This is where the "precomputed correction vectors" come from: they are neither hand-crafted nor uniform—they are read off the learned flow at whatever activation the model currently produces.
Step 4: Steer the selected heads in a single forward pass.
At inference, when the MLLM processes a new image, we take the current activation x at each selected head, evaluate the small ICNN to obtain S(x), and inject it—scaled by factors α (image-level shift S_img) and β (object-level shift S_obj)—into that head's contribution to the layer's output. Because ∇Ψ(x) depends on x itself, every token receives its own correction direction, dynamically adapting to sample-specific hallucination patterns—unlike prior activation-engineering methods such as ICT, which add the same fixed vector to every token. The MLLM's weights are never touched: the only added cost is evaluating a tiny ICNN, so latency overhead is negligible and the model's beneficial language and knowledge priors are preserved.
In one sentence: blur manufactures the training signal, the classifier picks the heads, the convex potential learns the optimal hallucination→trusted map, and a single forward pass with the resulting shift vectors steers the model back to reality.
Quantifying the Impact: State-of-the-Art Results on POPE and MME
A quick primer on the two benchmarks. - POPE (Polling-based Object Probing Evaluation) probes object hallucination directly: it asks the model binary yes/no questions about whether a given object appears in the image (e.g., "Is there a toilet in this image?"). The nonexistent ("negative") objects are sampled with three strategies—Random, Popular, and Adversarial—where Adversarial is the most challenging, picking absent objects that frequently co-occur with the image content. Because it polls with yes/no answers instead of free-form generation, POPE yields stable, verbosity-independent accuracy and F1 scores. - MME takes a broader view: it evaluates 14 perception and cognition subtasks—from existence, count, position, and color recognition to commonsense and higher-level reasoning—and aggregates them into a single unified score. Gains on MME therefore show that a method improves not just object existence but also attributional, positional, and reasoning fidelity.
We evaluated CPF on both benchmarks using LLaVA-1.5-7B and Qwen2.5-VL-7B, comparing against five representative hallucination-mitigation methods: VCD, OPERA, M3ID, AVISC, and ICT (full references at the end of this article).
Table 1: Highlight of performance () gains on POPE (Adversarial split, A-OKVQA)
| Model | Method | Accuracy | F1 Score |
|---|---|---|---|
| LLaVA-1.5-7B | Base model | 74.04 | 75.15 |
| LLaVA-1.5-7B | ICT (previous SOTA) | 75.56 | 78.68 |
| LLaVA-1.5-7B | CPF (ours) | 79.03 | 80.23 |
| Qwen2.5-VL-7B | Base model | 81.5 | 80.78 |
| Qwen2.5-VL-7B | ICT (previous SOTA) | 83.6 | 83.02 |
| Qwen2.5-VL-7B | CPF (ours) | 84.93 | 85.95 |
What do these numbers tell us?
First, the base models struggle significantly under challenging adversarial splits, confirming that strong language priors easily trigger hallucinations when visual evidence is tricky. Previous state-of-the-art methods like ICT provide some relief, but their improvements remain incremental because their fixed intervention directions cannot adapt to diverse, sample-specific hallucinations.
CPF, by contrast, delivers a massive leap. For LLaVA-1.5-7B, our method boosts Accuracy by nearly 5 absolute points (from 74.04% to 79.03%) and F1 Score by over 5 points compared to the base model. More importantly, Accuracy and F1 Score increase in lockstep. In hallucination mitigation, it is a common trap for models to artificially inflate Accuracy by becoming overly conservative (e.g., simply answering "No" to every question to avoid hallucinating an object), which typically tanks the F1 score. CPF's simultaneous improvement in both metrics proves that the model is genuinely "seeing" the visual content more accurately, rather than adopting a conservative guessing strategy. We observe similarly robust gains on the stronger Qwen2.5-VL-7B, confirming CPF's effectiveness across different architectures. On the MME benchmark, CPF shows equally strong gains: a total score of 740.95 on LLaVA (+42.38 over the base model, +12.15 over ICT) and 847.85 on Qwen (+45.71 over the base model, +25.71 over ICT). These standout improvements span across Count, Color, Position, and Common Sense Reasoning (CSR), proving that CPF mitigates not only object hallucinations but also attributional, positional, and reasoning hallucinations.
But is the dual-scale design (image-level + object-level) truly necessary? Our ablation studies on MME provide a clear answer. When we remove either module, the model's capabilities degrade in very specific ways: --Ablating the Object-Level Module (CPF w/o object): Without fine-grained object parsing, the model struggles with precise local details. For instance, on LLaVA, the Count task score drops significantly (from 148.33 down to 136.66). On Qwen, removing it hurts Color reasoning (dropping from 195.0 to 190.0). This proves the object-level flow is essential for fine-grained, object-centric accuracy. --Ablating the Image-Level Module (CPF w/o image): Without global image context, the model loses its grasp of high-level semantics and spatial relationships. On LLaVA, Common Sense Reasoning (CSR) takes a disproportionate hit (dropping from 119.28 to 110.0). More dramatically, on the stronger Qwen backbone, removing the image-level module triggers a massive 22.85-point drop in the total score (down to 825.0), primarily because the Position accuracy collapses (from 160.0 down to 140.0).
In short, the object-level cues refine local details (what color is it? how many are there?), while the image-level cues preserve global coherence (where is it relative to the scene? does it make logical sense?). Only the full integration of both paths delivers the optimal, hallucination-free performance. But numbers only tell half the story. The qualitative comparison below shows why CPF matters in practice:
Fig. 3: Qualitative comparison on MME and POPE. Where the base model and ICT fabricate implausible claims (e.g., "fried chicken legs are low in calories") or miscount objects, CPF stays grounded in the visual evidence.
Key takeaways:
- Dominance on POPE's Hardest Challenges: As demonstrated in our adversarial A-OKVQA results (Table 1), CPF delivers a massive leap over both the base model (nearly +5 points in Accuracy for LLaVA) and the previous state-of-the-art (ICT), proving its robustness against the most difficult visual-language conflicts.
- Comprehensive Enhancement on MME: CPF doesn't just stop models from seeing fake objects; it comprehensively enhances their perception and reasoning. As detailed in our ablation studies, it drives significant gains across Count, Color, Position, and Common Sense Reasoning on both LLaVA and Qwen architectures.
- Synergy of the Dual-Scale Design: Our experiments prove that neither module is redundant. Object-level cues are crucial for refining local details (Count, Color), while image-level cues are essential for preserving global spatial and semantic coherence (Position, CSR). Only their combination yields optimal results.
Impressions from the conference
I was struck by how the conference focus is shifting from pure recognition accuracy toward real-world robustness and benchmark quality. The commemorative lecture on benchmarking in the era of MLLMs resonated deeply with me: as models grow more capable, the quality of evaluation data becomes as important as the models themselves—a point directly relevant to hallucination evaluation on POPE and MME.
Why CPF Matters for Real-World AI
Our work contributes:
✅ A token-specific intervention framework that adapts to sample-specific hallucination patterns, unlike previous uniform methods.
✅ A training-free, plug-and-play solution requiring only a single forward pass with negligible computational overhead.
✅ Extensive cross-model generalization, effective on diverse architectures (LLaVA, Qwen) without sacrificing linguistic coherence.
Resources
Contact: shiziqiang@fujitsu.com
This research was conducted at Fujitsu Research and Development Center (FRDC) and Fujitsu Research Japan. We sincerely thank our colleagues for their valuable discussions and feedback.
References
[1] (VCD) Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., Bing, L.: Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In: Proceedings of the IEEE/CVF Conference on CVPR. pp. 13872–13882 (2024)
[2] (OPERA) Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., Yu, N.: Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In: Proceedings of the IEEE/CVF Conference on CVPR. pp. 13418–13427 (2024)
[3] (M3ID) Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swami-nathan, A., Soatto, S.: Multi-modal hallucination control by visual information grounding. In: Proceedings of the IEEE/CVF Conference on CVPR. pp. 14303– 14312 (2024)
[4] (AVISC) Woo, S., Kim, D., Jang, J., Choi, Y., Kim, C.: Don’t miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820 (2024)
[5] (ICT) Chen, J., Zhang, T., Huang, S., Niu, Y., Zhang, L., Wen, L., Hu, X.: Ict: Image- object cross-level trusted intervention for mitigating object hallucination in large vision-language models. Proceedings of the IEEE/CVF Conference on CVPR (2025)
[6] (MME) Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023)
[7] (POPE) Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)
