Abstract

Recent advances in vision-language models have accelerated multimodal brain decoding, particularly brain captioning, which aims to translate neural activity into natural language descriptions. However, most existing methods align brain signals with either textual embeddings or single-stream visual representations, overlooking the hierarchical and context-dependent organization of the human visual cortex. In this work, we propose a multi-expert routing framework with progressive alignment optimization for brain captioning. We introduce task-specialized vision experts that encode complementary perceptual attributes across multiple abstraction levels. To reflect neural variability across cortical regions and subjects, we develop a brain-conditioned dynamic routing mechanism that adaptively modulates expert contributions based on neural activity. To address noise and inter-subject variability, we formulate a progressive dual-objective alignment strategy that combines optimal transport for global distribution matching with robust regression for feature-level refinement. Extensive experiments demonstrate consistent improvements in both single-subject and multi-subject settings. These results suggest that structured multi-expert supervision combined with dynamic routing provides a principled framework for multimodal brain decoding.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4241_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{GuoZhi_MultiExpert_MICCAI2026,
        author = { Guo, Zhilin AND Xia, Weihao AND Cao, Mingdeng AND Zhou, Chenliang AND Oztireli, Cengiz},
        title = { { Multi-Expert Representation Learning with Dynamic Routing for Brain Captioning } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16894},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The main contribution is MERPA, a multi-expert representation learning framework for brain captioning. It uses six frozen vision experts including depth estimation, surface normals, edges, semantic segmentation, object detection, and OCR to provide hierarchical visual supervision. It adaptively aggregates expert tokens via brain-conditioned dynamic routing and optimizes brain-vision alignment using a progressive dual objective composed of optimal transport and robust regression. On the NSD dataset under the BrainHub evaluation protocol, the method outperforms the baselines reported in the paper in both single-subject and multi-subject joint-training settings.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The method has a reasonable motivation. Combining low-level geometric experts with high-level semantic experts is broadly consistent with the hierarchical vision mechanism described in this paper.

    2.The empirical results are promising. As shown in Table 1, MERPA outperforms the listed baselines in both single-subject and multi-subject joint-training settings, with particularly clear improvements on semantic metrics such as CIDEr, SPICE, CLIP-S, and RefCLIP-S.

    3.The paper provides useful ablation studies. They at least preliminarily show that the routing strategy, alignment objective, and expert grouping all contribute to the final performance.

    4.The framework appears to be extensible. Using multiple pretrained vision experts as structured supervision sources can potentially be extended to other perceptual attributes or modalities.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.There is a mismatch between the main claim and the actual evaluation setup. The paper repeatedly emphasizes cross-subject generalization, but the reported “cross-subject” experiment jointly trains on S1/S2/S5/S7 and evaluates on held-out images. This is much closer to multi-subject joint training than to genuine generalization to unseen subjects. Therefore, the central claim on cross-subject generalization is not well supported.

    2.The novelty boundary remains insufficiently clear. Based on the paper’s own related work and references, MEVOX has already introduced multi-task vision experts into brain decoding and brain captioning. BrainOT already uses optimal transport for brain-image alignment, and VINDEX discusses the effects of different visual feature spaces in multimodal neural decoding. The current manuscript does not clearly identify whether the real novelty lies in brain-conditioned dynamic routing, the progressive OT+Fair scheduling, or only in the specific combination of these components.

    3.The technical description contains several critical ambiguities that affect reproducibility. For instance, the paper mentions that “RoBERTa serves as the frozen language decoder” but does not explain the concrete implementation of caption generation. In addition, the granularity of routing weights (sample-level / ROI-level / token-level), ROI feature processing, adapter structure, and module interfaces are not clearly specified. The paper also repeatedly uses terms such as “interpretable” and “brain-inspired” without providing direct evidence on whether the expert preferences learned by the routing mechanism are consistent with the functional specialization of the cerebral cortex.

    4.The experimental evidence remains limited. The paper uses only a single dataset and evaluation protocol, and does not provide the mean and standard deviation from multiple random seeds, significance testing, qualitative or failure-case analysis, nor training and inference cost analysis. For modest improvements in some metrics, these omissions severely weaken the reliability of the conclusions.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The method is reasonably motivated, and the results are competitive on the reported benchmark. However, the current manuscript is still not sufficiently solid in several crucial aspects (see Weaknesses).

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    Thank the authors for their rebuttal, in which the authors admit that the statement of “cross-subject generalization” is overstated and promise to revise it into “multi-subject joint training”. Additionally, several technical details still remain unclear, including how RoBERTa serves as a frozen language decoder for caption generation, as well as the working granularity of routing weights (sample-level, ROI-level and token-level). Accordingly, I will keep my original score and tend to recommend rejection.



Review #2

  • Please describe the contribution of the paper

    This paper proposes a multi-expert routing framework for brain captioning that aligns fMRI signals with structured visual representations derived from multiple frozen vision experts. The core idea is to model brain captioning by leveraging the hierarchical and context-dependent organization of the human visual cortex, using task-specialized experts and dynamic routing. The motivation is interesting, and the manuscript is generally well organized.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The idea of aligning brain activity with multiple structured visual experts rather than a single embedding is interesting and potentially meaningful for brain decoding. The hierarchical decomposition (e.g., depth, edges, segmentation, OCR) provides interpretable intermediate representations and may better reflect the multi-level processing of visual information.

    2.The combination of optimal transport for global distribution alignment and robust regression for local refinement is reasonable and well-motivated for handling noisy fMRI signals.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The visual tokens produced by different experts are not inherently in a shared representation space, and their distributions may vary significantly. Moreover, these expert features are not explicitly aligned with the text token space of the LLM, which may limit the effectiveness of cross-modal alignment and ultimately constrain the performance of the proposed framework.

    2.The selection of vision experts appears heuristic. The paper does not justify why these specific experts were chosen, nor does it analyze the impact of different expert combinations or the number of experts.

    3.The paper refers to ROI-conditioned routing but does not clearly define the ROI regions. Since the design is motivated by neuroscience, additional analysis would strengthen the claim, such as region-wise routing visualization, ROI-specific performance, or evidence demonstrating neuro-inspired hierarchical alignment.

    4.Despite the increased model complexity, the performance improvements over BrainOT appear relatively modest. More discussion is needed to clarify where the gains originate, as well as to acknowledge potential limitations and outline future directions.

    5.Figure 2 could be improved by explicitly separating the training and inference pipelines, which would help clarify the workflow and improve readability.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents an interesting multi-expert routing framework for brain captioning that leverages hierarchical visual experts and progressive alignment, which is reasonably motivated. The approach is novel and shows consistent, though modest, improvements over prior work. However, the heuristic expert selection, limited neuroscience validation, and relatively small performance gains weaken the overall contribution, leading to a weak accept recommendation.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The authors introduce MERPA, a brain captioning framework that leverages multi-vision taske experts (e.g., depth estimation, segmentation, optical character recognition, etc.) to address the limitations of past works that align textual embeddings with single-source, non-specialized, image information.

    In more detail, MERPA first extracts image tokens from multiple image experts. With these visual tokens, the authors then train a routing network to learn coefficients for each expert contribution, effectively weighing how much information from an expert output to take. Next, the authors formulate two losses, one to align modalities or distributions as closely as possible to maintain global geometric (visual) consistency, and the other to address inter-subject variation (a common problem in vision-brain captioning). Experiments with MERPA show the methodology outperforms current works, demonstrating the effectiveness of multiple expert networks and aligning vision streams and brain activity.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is overall well-written, and the motivation and visualization of the usage of multiple vision experts, and plugging them into the brain-captioning framework, I think, is well communicated. It is easy to see how the methodology in this aspect differs from other works, especially in the “vision brain understanding” problem.

    Both the distribution alignment of brain captioning and visual expert tokens, and the robust feature refinement are well-motivated with regards to the architectural choice made by the authors. This is shown to be a simple contribution but one that performs well compared to the current works.

    BrainHub captioning experiments (Table 1) show that across all metrics and compared against many past works, MERPA performs better or at least comparably. This shows that incorporating multiple streams of visual information does hold weight and is a research direction that could merit further investigation. Ablation experiments (Table 2) also demonstrate how all MERPA components combined contribute to performance gains.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Vision-brain understanding is not my main field, but I do not see blatant, major, flaws in this work. I do have several points I wish for the authors to clarify.

    1) While the alignment of brain tokens with heterogeneous vision tokens makes sense to me, can the authors clarify how MERPA uses the brain tokens or the weighted vision tokens to pass on to the LLM for caption generation? This step should be essential to clarify both in the formulation and in the figure, which does not explicitly show the data flow into the LLM/decoder block to generate captions. This is also not clarified in the manuscript it seems; apologies if I missed this.

    2) Can the authors elaborate on their choice of the Roberta LLM? Was this standard across comparable works, or is it a good compromise between low-cost and effectiveness? Just a little discussion on this would be helpful, as well as what other works (in the experiments shown) use to make the comparisons more transparent.

    3) Additionally, how sensitive is Roberta (or any applicable LLM for that matter) to the new tokens it is taking in? I ask this on the basis that the distribution of the tokens from MERPA vs what LLMs like Roberta expect may be different, so it would strengthen the paper if the authors can provide insight on this, if it is an issue at all.

    4) Can the authors clarify how well the actual generated/decoded captions via MERPA compare with respect to visual stimulus images? I assume since ablations were done, the authors would have the response-image stimulus pairs already. All to say, while Tables 1 and 2 may suggest quantitatively that MERPA is performant in comparison to other methods, how does it compare qualitatively? Showing this inference output along with input images would be beneficial since much of the scores check for overlap of n-grams, but readers are not able to see full image-generated caption pairs.

    5) For the mix of vision experts, since multiple feature extractor look at the same image, there may be an overlap of features across (b,w) pairs, i.e., there may be redundant features. Can the authors provide insight as to how this may affect downstream modules? This could potentially be an insight in future work.

    6) Minor comment: The best results in Table 2 should be in bold. Additionally, any abbreviations in Table 2 should be defined in the caption, increasing readability when a reader just looks at the table.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    This work naturally uses fMRI signals from real human subjects, including neurodivergent persons and children, and so data privacy, privacy enhancing technologies, and user conduct should be central to these kinds of works.

    Data should be kept private and confidential only to medical professionals and patients. Privacy protection should be an upmost priority in such medical applications. Finally, any persons involved with the usage of a tool like this, both in research and in medical applications, should uphold high ethical standards.

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    While the paper operates in a crowded research space, I think the multiple image stream and alignment ideas seem to hold weight and therefore could merit future research directions. If the authors can address the major points, I will be more confident in my accept rating.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    After the authors’ rebuttal response, I believe that this work should be accepted into the MICCAI conference. It presents an interesting approach to the vision brain captioning problem by providing high- and low-level image embeddings, each backed by by neuroscience literature. Given this different approach, and both the modest gains and more significant metric gains, I think it would serve as an interesting contribution to the field in terms of providing a different perspective.

    The reviewers agreed with and provided clarification on, e.g., the joint-training scenario instead of zero-shot, as well as their methodology flow, which needed significant clarification. Additionally, if they can deliver on clarification of the implementation as I and other reviewers have requested, then I would be much more confident in my accept rating.



Author Feedback

We sincerely thank the reviewers and meta-reviewer for the constructive feedback and positive assessment of our method.

1.Cross-subject terminology We agree that the wording “cross-subject generalization” may overstate the experimental setting. This terminology follows prior methods such as MindEye/MEVOX/BrainOT/VINDEX, which also describe similar multi-subject joint training. We agree that the current setup corresponds more precisely to multi-subject joint training rather than unseen-subject zero-shot generalization. We will replace “cross-subject generalization” with “multi-subject joint training” and remove any implication of zero-shot transfer. Importantly, our contribution is not unseen-subject transfer, but a structured representation learning framework that improves multimodal alignment through: (1) hierarchical expert supervision, (2) brain-conditioned dynamic routing, and (3) progressive OT-based alignment optimization.

2.Novelty and relation to prior work Unlike the static aggregation (uniform weights) in MEVOX, MERPA introduces brain-conditioned routing that adaptively modulates expert contributions using ROI-dependent neural representations (Tab 2 Routing row: static 49.89 -> dynamic 71.91, +44.1%). In addition, while BrainOT uses OT for structural alignment, MERPA addresses the dual challenge of distributional alignment and neural noise robustness through progressive combination of OT-based global alignment with Fair loss-based local refinement, achieving consistent improvements across metrics (Tab 1-2).

3.Implementation details During training, frozen vision experts produce structured visual tokens dynamically aggregated via brain-conditioned routing as alignment targets for brain encoder. During inference, only optimized brain tokens are projected to frozen RoBERTa’s input space and decoded for caption generation. Vision expert features may be partially redundant; however, the brain-conditioned router dynamically selects complementary information across experts (Tab 2: removing any expert degrades performance). The learned adapter aligns brain tokens with the language model input. We will revise Fig. 2 to explicitly separate training/inference pipelines, clarify routing granularity and module interfaces, and provide implementation details in the appendix. Code and models will be released.

4.Expert selection and neuroscience motivation The selected experts were not chosen arbitrarily but to reflect complementary visual hierarchy, covering low-level geometric (depth / surface normals / edges) and high-level semantic (segmentation / object detection / OCR) cues. This design is motivated by the hierarchical organization of the visual cortex and supported by ablations (Table 2), where low-level-only and high-level-only experts both underperform the combined configuration. We will include neuroscience-oriented analyses and studies on different expert combinations in the revision.

5.Evaluation Although improvements over BrainOT are modest on some metrics, MERPA consistently improves performance across metrics in both single-subject and multi-subject settings, especially on semantic-sensitive metrics (CIDEr +5.1%, SPICE +3.8%), which are particularly relevant for semantic consistency in brain captioning. Statistical analysis across 5 runs further demonstrates stable performance with statistically significant improvements (p<0.05) in both single-subject (BLEU-4 20.94±0.19, CIDEr 71.91±0.42, and SPICE 14.55±0.31) and multi-subject (BLEU-4 23.13±0.22, CIDEr 77.46±0.48, and SPICE 15.65±0.35) settings.

6.Ethics Our work uses the publicly available NSD dataset, which is anonymized and distributed under established research protocols. We do not collect new human-subject data, and all experiments are conducted on anonymized publicly available data for research purposes. We will also include an ethics and privacy discussion, qualitative caption comparisons, and failure cases in the revised manuscript.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The reviewers acknowledge the MERPA framework as a well-motivated approach that aligns hierarchical visual experts with neuroscientific mechanisms, providing competitive quantitative results and clear ablation studies; however, the authors should address several critical methodological and evaluative gaps in the rebuttal. Specifically, the authors must clarify the technical novelty relative to works like MEVOX and BrainOT, and resolve the mismatch between claims of “cross-subject generalization” and an experimental setup that currently reflects multi-subject joint training. The rebuttal must describe the data flow into the language decoder, and justify the selection of heuristic vision experts. Furthermore, there are some concerns regarding the ethics of data privacy. The authors should clarify whether the data is properly anonymized and passes necessary ethical considerations.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    All reviewers were satistfied with the rebuttal. The authors should address the remaining concerns in the camera-ready version.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    After rebuttal, two reviewers suggest accepting this paper and one reviewer suggest rejecting this paper. Based on the rebuttal as well as the recommendations of the reviewers, the paper can be accepted.

    It is suggested that the authors sufficiently consider the comments raised by reviewers and further improve the paper in the final version.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The proposed MERPA framework introduces a well-motivated approach to brain captioning by aligning fMRI signals with hierarchical visual experts. In their rebuttal, the authors effectively addressed the reviewers’ primary concerns by clarifying the multi-subject joint training setup, detailing the dynamic routing mechanism, and confirming the ethical use of the public anonymized NSD dataset. Although one reviewer remained concerned about the language decoder’s implementation details, the majority of reviewers agreed that the neuro-inspired methodology and the statistically significant performance gains warrant acceptance.



back to top