List of Papers Browse by Subject Areas Author List
Abstract
Generating natural language captions from fMRI remains challenging due to the large modality gap between brain activity and linguistic representations. We propose a two-stage brain-to-caption framework that leverages image embeddings as an intermediary between fMRI and language. In the first stage, a vision-language pipeline learns image-to-caption generation via LLaMA3. In the second stage, fMRI activations are aligned to the image embedding space through three novel components: (1) hyperbolic embedding alignment in the Poincaré ball providing more accurate and stable fMRI-to-image matching than Euclidean alternatives; (2) Token Probability Distribution Alignment (TPDA), which distills next-token distributions from the image-to-caption branch to the fMRI-to-caption branch; and (3) visual pathway tokenization, which converts fine-grained vertex-level visual cortical inputs into query tokens to supplement whole-brain regional features. On the Natural Scenes Dataset, our method achieves a CIDEr score of 56.79, outperforming existing methods. Beyond quantitative gains, neuroscientific analysis reveals the model’s regional activations align with known functional brain specialization of visual pathway areas without explicit word type supervision, suggesting neurobiologically plausible brain-to-language mappings. Our code is at https://github.com/gudtls17/BrainToCaption.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2335_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/gudtls17/BrainToCaption
Link to the Dataset(s)
N/A
BibTex
@InProceedings{ChoHyo_Imagemediated_MICCAI2026,
author = { Choi, Hyoungshin AND Lee, Youkyoung AND Park, Bo-yong AND Park, Hyunjin},
title = { { Image-mediated fMRI-to-caption generation with visual pathway tokens and hyperbolic alignment } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16894},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
1.This paper proposes a hyperbolic embedding alignment strategy in the Poincaré ball to more accurately and stably match fMRI activations to the image embedding space than Euclidean alternatives. 2.This paper introduces Token Probability Distribution Alignment to distill next-token probability distributions from a teacher image-to-caption branch to the fMRI-to-caption branch. 3.The model incorporates a visual pathway tokenization mechanism that converts fine-grained vertex-level activations from the visual cortex into query tokens to supplement regional whole-brain features.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The research introduces a novel geometric formulation by performing alignment in the hyperbolic Poincaré ball, which more effectively captures the complex, non-uniform hierarchical structure of brain representations compared to traditional Euclidean metrics. 2.The model employs an original data utilization strategy through visual pathway tokenization, which bypasses the information loss of standard regional averaging by converting fine-grained, vertex-level activations into query tokens. 3.The work demonstrates strong neurobiological interpretability, as the model’s regional activations align with known functional brain specialization for specific parts of speech without requiring any explicit word-type supervision during training.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The proposed method relies on a frozen image-to-caption pipeline as a teacher, which potentially caps the fMRI decoding performance by the inherent biases and linguistic limitations of the pre-trained CLIP and LLaMA3 models. 2.The Visual Pathway Tokenization utilizes a high-dimensional vertex input (V=12535 vertices) that significantly increases the parameter count and computational overhead compared to standard regional parceling.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The technical integration of hyperbolic geometry and fine-grained visual pathway tokenization provides a novel and effective solution to the long-standing modality gap between brain activity and linguistic representation. This paper achieves state-of-the-art performance on the Natural Scenes Dataset by outperforming strong baselines like UMBRAE and MindLLM in nearly all captioning metrics. Extensive ablation studies and neuroscientific analysis demonstrate that the framework is both robustly designed and neurobiologically plausible, aligning regional activations with known functional brain specialization.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I have read the authors’ response and the other reviewers’ comments, and I decide to maintain my opinion.
Review #2
- Please describe the contribution of the paper
This paper presents a two-stage fMRI-to-caption framework that uses image embeddings as an intermediate space between brain activity and language. The main technical contributions are: (1) hyperbolic alignment in the Poincare ball for fMRI-to-image embedding matching, (2) Token Probability Distribution Alignment (TPDA) to distill next-token distributions from the image-conditioned branch to the fMRI-conditioned branch, and (3) visual pathway tokenization to inject fine-grained vertex-level visual cortical information into the fMRI branch. On the NSD benchmark, the method outperforms the reported baselines and is supported by ablation studies and neuroscientific attribution analysis.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper has several notable strengths. First, the overall problem is interesting and timely: generating captions directly from fMRI is challenging due to the large modality gap between brain activity and language, and the proposed image-mediated formulation is well motivated. Second, the method is technically coherent and combines three meaningful design choices: hyperbolic embedding alignment, TPDA-based teacher-student distillation, and visual pathway tokenization using fine-grained vertex-level features from visual cortex. Third, the empirical results are strong: on NSD, the method reports the best CIDEr score among the compared methods, and the gains are supported by ablation studies that isolate the impact of the alignment loss, TPDA, stage-1 pretraining, and visual pathway tokenization. Fourth, the paper is generally well organized and includes both quantitative and qualitative analyses, including neuroscientific attribution analysis that improves interpretability. Finally, the submission provides an anonymized code link, which is a positive signal for reproducibility.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The main weaknesses are as follows. First, the relevance to core MICCAI themes is somewhat limited: although the paper uses fMRI, the task is natural-scene brain captioning rather than a medical imaging or clinically grounded problem, and the clinical significance remains indirect. Second, the experiments are limited to NSD and appear to focus on only the first subject who completed all trials, which raises concerns about subject-level generalization and broader robustness. Third, while the combination is interesting, some components are incremental relative to the recent brain decoding and multimodal literature cited by the authors, including BrainCap, BrainChat, UMBRAE, and MindLLM; therefore, the paper would benefit from a clearer discussion of what is fundamentally new beyond a strong integration of existing ideas. Fourth, the evaluation is centered on automatic captioning metrics, and stronger evidence such as cross-subject validation, statistical significance testing, or a more explicit analysis of failure modes would further strengthen the claims. Finally, the interpretability analysis is interesting, but at present it remains more suggestive than conclusive.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
This is an interesting and technically solid paper on fMRI-to-caption generation. I especially appreciate the clear motivation for using image embeddings as an intermediate representation and the careful ablation studies. My main concerns are about broader relevance and generalization: the task is still somewhat peripheral to core MICCAI application areas, and the experiments appear limited to a single-subject setting on NSD. I encourage the authors to strengthen the discussion of scope and clinical significance, clarify the novelty relative to recent multimodal brain decoding works, and, if possible, provide stronger evidence of robustness beyond the current benchmark setting.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend a weak reject. The paper is technically competent, well written, and supported by strong benchmark results and ablations. The proposed combination of hyperbolic alignment, token-level distillation, and visual pathway tokenization is thoughtful and appears effective on NSD. However, I have reservations about its fit and impact for MICCAI: the task is natural-scene brain captioning rather than a clearly medical or clinically grounded imaging problem, and the evidence is limited to a narrow benchmark setting centered on a single subject. In my view, this makes the work interesting and promising, but still somewhat below the bar for acceptance at MICCAI in its current form.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper tackles generating captions from fMRI. The authors use a two-stage approach: first train an image-to-caption model, then map fMRI to the same image embedding space so the caption model can be reused. Key ideas include hyperbolic alignment for fMRI-image matching, TPDA for distillation, and visual pathway tokenization to bring in fine-grained cortical signals.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper tackles an ambitious problem across neuroimaging, representation learning, and language generation. The two-stage design is clear and easy to follow. Results on NSD are strong, beating several prior baselines. The ablation study is reasonable, and the authors also make an effort to link their model’s behavior to known brain organization, which is a nice addition beyond just reporting benchmark scores.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The evaluation is limited to caption generation. It remains unclear whether the proposed method generalizes to other tasks such as VQA, which have been explored in concurrent work like MindLLM. The advantages of this approach in broader task settings are therefore not established.
2.The evaluation relies primarily on automatic NLG metrics (CIDEr, BLEU, etc.). For a generation task of this nature, human evaluation or semantic analysis would provide stronger evidence of caption quality.
3.The claimed neuroscientific interpretability seems to be overstated. The observation that the model activates known visual regions without explicit word-type supervision is interesting, but it does not demonstrate that the model has learned neurobiologically meaningful representations. A more plausible explanation is that the model simply picks up on statistical regularities in the data. For example, nouns and adjectives may correlate with different types of visual features, which in turn map to distinct fMRI patterns. To support the claim, the authors would need to show cross-subject generalization of these activation patterns or test whether perturbing specific regions degrades corresponding word-type predictions. As it stands, the analysis remains correlational.
4.The motivation for using hyperbolic geometry is not sufficiently justified. Figure 3 only shows that hyperbolic alignment yields smaller embedding distances, which is an auxiliary metric. The paper would benefit from a direct qualitative comparison of captions generated with and without hyperbolic alignment. For instance, whether hyperbolic alignment leads to better preservation of hierarchical or fine-grained semantic relationships. Additionally, a visualization of the embedding space (e.g., using horoPCA) comparing hyperbolic vs. Euclidean representations could provide intuitive evidence of why the hyperbolic space is better suited for aligning fMRI and image embeddings.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting but needs more justification and discussion.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The author addressed my concerns. The persuasiveness of the paper has improved.
Author Feedback
We thank the reviewers for valuable comments. Below we address concerns.
Q1 (R1-1,2): Architectural design rationale. A: The frozen image-to-caption pipeline is an intentional anchor for stable Stage 2 alignment despite the known bias. The high-dimensional vertex input feeds only a single MLP, not a large computation burden by recent standards. We will clarify this.
Q2 (R2-1): Clinical significance & MICCAI relevance. A: Brain decoding has been published in MICCAI (Huang et al., MICCAI 2022; Shen et al., MICCAI 2025), placing the task within core MICCAI themes. Studying the healthy brain is important to gauge the diseased brain. Our method treats the healthy brain as a normative reference to interpret simulated regional impairments. We generate captions with each visual region’s input zeroed at inference (see Q6) and note that impairment varies by region and meaning. Fusiform face area lesion distorts the gender for an image of a woman standing on a tennis court, the lesion of early visual regions misinterprets type of food for an image of pizza (Fig 2). The region perturbation results are similar to the caption generation without visual pathway tokenization (Fig 2), which can be interpreted as how the brain would decode fMRI if some visual pathway regions are impaired. We will clarify this.
Q3 (R2-2, R3-3;Meta#1): Cross-subject validation. A: We had cross-subject results, but chose not to include them for fair comparison since existing studies only reported Subject1.We train a separate model for each subject and present cross-subject results. Vertex-level visual-pathway attribution maps (Fig 4b across four subjects in NSD) yield intra-class correlation of 0.607/0.578/0.653 (noun/verb/adjective, 0.6+ is “Good” range). All 6 (4 choose 2) pairwise correlations are significant (mean r=0.478, all FDR p<0.001). We will briefly mention these results.
Q4 (R2-3;Meta#2): Novelty relative to recent brain decoding works. A: Three contributions are fundamentally new among cited methods (BrainCap/BrainChat/UMBRAE/MindLLM): (1) non-Euclidean (hyperbolic) alignment for fMRI-image. (2) vertex-level visual pathway tokenization — UMBRAE uses subject-specific voxel-subset tokenizer, MindLLM region-level attention, neither at vertex granularity. (3) post-hoc neuroscientific verification: prior works report captioning metrics only. We will clarify this.
Q5 (R3-1): The possibility of task extension. A: Our method generates embeddings that can generate captions from fMRI. Thus, zero-shot extension to VQA or retrieval tasks is possible by interfacing the output embedding with off-the-shelf image-conditioned VLMs without any retraining. This will be added.
Q6 (R3-2,3;Meta#3,4): Region perturbation as semantic evaluation. A: To test whether perturbing specific regions degrades corresponding word-type predictions, we generate captions with each visual region’s input zeroed at inference. Disruptions are region-specific and differ across meanings. For an image of a woman standing on a tennis court, fusiform face area lesion distorts the gender. For pizza, early visual regions yield within-category changes (compare with examples in Fig 2). This provides both (i) model-level perturbation evidence beyond correlation and (ii) semantic analysis complementary to automatic NLG metrics. This will be added.
Q7 (R3-4;Meta #5): Hyperbolic alignment justification. A: We examine whether the learned geometry encodes interpretable semantic structure with horoPCA-style visualization comparing hyperbolic vs. Euclidean representations. A horoPCA-style projection using tangent-space approximation provides intuitive evidence: PC1 correlates with the average noun concreteness per caption for hyperbolic (|r|=0.153, p<0.001), not for Euclidean PCA (|r|=0.051, p_FDR=0.165). Thus, the radial coordinate encodes word concreteness only in the hyperbolic case. We will briefly mention these results.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a two-stage fMRI-to-caption framework using image embeddings as an intermediate representation, with hyperbolic alignment, TPDA, and visual pathway tokenization. Reviewers appreciated the strong NSD results, clear ablations, and the attempt to connect model behavior with functional brain specialization. However, the rebuttal should address the reviewers’ concerns through clarification and appropriate qualification of claims, including, but not limited to, the narrow single-subject NSD evaluation and lack of broader robustness evidence, the novelty relative to recent brain decoding and multimodal captioning methods, the reliance on automatic captioning metrics without stronger semantic evaluation, the correlational nature of the neuroscientific interpretability claims, and the need to better justify why hyperbolic alignment is appropriate beyond improved embedding distance.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The submitted manuscript presents a technically solid fMRI-to-caption framework with strong NSD results and meaningful ablation evidence, and the combination of hyperbolic alignment, TPDA, and visual pathway tokenization is sufficiently interesting. However, the evaluation remains narrow and the neuroscientific interpretation should be moderated, given the single-subject setting and largely correlational analysis.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper presents a technically coherent fMRI-to-caption framework, with strong benchmark results and well-designed ablation studies that isolate the contributions of each component. The initial reviews raised legitimate concerns about single-subject generalization, the neuroscientific interpretability claims, and the justification for hyperbolic alignment. The authors’ rebuttal provided substantive additional evidence that directly addressed these weaknesses, including cross-subject reliability of attribution maps, region perturbation analyses demonstrating semantic specificity, and a horoPCA-style comparison showing that hyperbolic space better encodes concreteness. Two reviewers ultimately recommended acceptance, recognizing the work’s novelty and the strength of the response. The paper’s contributions and the rebuttal’s effective resolution of major concerns justify acceptance.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
After carefully reviewing the authors’ rebuttal, I have revised my initial assessment to Accept. The authors directly addressed key concerns including cross-subject validation with significant inter-subject reliability, region perturbation analysis demonstrating semantic specificity, horoPCA visualization validating hyperbolic geometry, clarification of clinical relevance to brain decoding for normative reference, and articulation of novelty relative to prior methods.