List of Papers Browse by Subject Areas Author List
Abstract
Segmenting vocal tract articulators in real-time MRI (rtMRI) is a challenging dynamic image segmentation problem characterized by low contrast, rapid motion, and limited spatial resolution. However, while rtMRI acquisitions may provide synchronized acoustic signals, existing methods discard this information, and the few multimodal approaches that incorporate audio cannot be deployed when audio is unavailable. We propose a three-stage framework that leverages acoustic and phonological supervision during training while requiring only the rtMRI image at inference: phonological representations are converted into spatial bounding-box priors for articulator localization, visual and acoustic encoders are aligned via dual-level cross-modal contrastive pretraining, and the learned representations are fused through a cross-attention decoder, effectively transferring multimodal knowledge into a single-modality inference pipeline. Evaluated on 75-Speaker Annot-16 and USC-TIMIT datasets, our method outperforms existing unimodal and multimodal methods, demonstrating that multimodal supervision provides transferable benefits for precise and clinically deployable vocal tract segmentation. Our code is available at https://github.com/daiqi76/SGVocSegMRI.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1811_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/daiqi76/SGVocSegMRI
Link to the Dataset(s)
USC-75 dataset: https://sail.usc.edu/span/usc-timit/
USC-75-Speaker Annot-16: https://sail.usc.edu/span/75speakers_annot/
USC-TIMIT: https://sail.usc.edu/span/usc-timit/
BibTex
@InProceedings{LiuDai_SpeechGuided_MICCAI2026,
author = { Liu, Daiqi AND Mulzer, Lukas AND Hasan, Md AND Alves de Castro, Nyvenn AND Xing, Fangxu AND Kang, Xingjian AND Ye, Chengze AND Mei, Siyuan AND Sun, Yipeng AND Arias-Vergara, Tomás AND Hutter, Jana AND Woo, Jonghye AND Maier, Andreas AND Pérez-Toro, Paula Andrea},
title = { { Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16896},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a three-stage multimodal framework for vocal tract segmentation in real-time MRI that uses audio and phonological information during training but requires only the MRI image at inference. The key idea is that multimodal knowledge gets encoded into model weights during training, enabling audio-free deployment in clinical settings where microphones are impractical or patients have speech disorders. Experiments on two datasets show consistent improvements over both unimodal and multimodal baselines.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main technical contribution is distilling multimodal knowledge into a vision-only inference pipeline through spatial prior generation, contrastive alignment, and cross-attention fusion. This is supported by a comprehensive evaluation across two datasets, multiple generalization settings, and a thorough ablation study that clearly quantifies the contribution of each stage. The three-stage pipeline is well-structured, with each stage serving a clear purpose. The ablation study in Table 2(A) convincingly demonstrates that each stage contributes incrementally, and the progression from naive concatenation to spatial priors to contrastive pretraining to cross-attention is logical and well-justified. The phonological-to-spatial prior (Stage 1) is a creative contribution — converting symbolic linguistic knowledge into spatial bounding boxes bridges the gap between discrete phonological labels and continuous image features in a principled way.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The framework processes frames independently without temporal modeling, which is a significant limitation for real-time MRI where articulator motion is inherently sequential. The authors acknowledge this but don’t explore even simple temporal baselines, leaving it unclear how much performance is left on the table. The bounding-box priors in Stage 1 are computed per-subject using ground-truth masks, which means they require annotated data for each new speaker. This limits scalability to unseen subjects in truly clinical deployment. The method removes audio dependency but still requires real-time MRI, which is expensive and not widely available. The paper doesn’t discuss potential extension to more accessible modalities like ultrasound, which would better support the clinical deployability claim.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a well-motivated and technically sound framework that addresses a real clinical need — enabling vocal tract segmentation without audio at inference. The three-stage design is principled, each component is justified through a thorough ablation study, and the consistent improvements across two datasets and multiple generalization settings are convincing. While limitations exist around temporal modeling, reliance on real-time MRI, and lack of pathological data, the authors are transparent about these gaps. The overall contribution is clear and the evaluation is strong enough to support acceptance.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes a three-stage framework that integrates visual, acoustic, and phonological information during training, while enabling image-only inference. It introduces a novel approach to convert phonological class descriptors into spatial bounding-box priors, effectively injecting structured linguistic knowledge into the segmentation process. The study shows that multimodal supervision can be effectively distilled into model parameters, maintaining performance even when auxiliary modalities are absent at inference.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper addresses vocal tract segmentation in real-time MRI (rtMRI), a challenging task due to low contrast, rapid motion, and limited spatial resolution. The motivation is well-grounded in clinically relevant applications such as speech disorder assessment and surgical planning. 2.The introduction of phonological descriptors and their transformation into spatial bounding-box priors is a novel and meaningful way to bridge symbolic linguistic knowledge with spatial image representations. This provides explicit inductive bias for localizing articulators. 3.The method demonstrates consistent improvements over both unimodal and multimodal baselines on multiple datasets and evaluation settings (including unseen speaker/task scenarios), indicating robustness and generalization ability.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.While the overall framework is well-designed and coherently integrates multiple components, most of its building blocks—such as contrastive learning, cross-attention mechanisms, and the use of pretrained encoders (e.g., ViT and WavLM)—are based on established techniques. As a result, the primary novelty appears to stem from the combination of these components rather than from fundamentally new methodological contributions. 2.In the ablation study, the Image-only variant achieves performance that surpasses most existing unimodal methods. This is somewhat unexpected and not sufficiently discussed. Additional analysis is needed to clarify whether this gain arises from architectural advantages, training strategies, or other factors, and to ensure that the comparison is fair and meaningful. 3.There appears to be a discrepancy in Table 2 regarding the reported number of parameters for the proposed method (e.g., 186.64M vs. 188.64M). This inconsistency should be clarified to avoid confusion and ensure reproducibility.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper addresses a clinically relevant and technically challenging problem in vocal tract segmentation from real-time MRI, and proposes a well-motivated multimodal framework that leverages acoustic and phonological supervision while maintaining image-only inference. The idea of transferring multimodal knowledge into a unimodal deployment setting is practical and valuable, particularly for real-world clinical scenarios. However, the work has several limitations. The methodological novelty is somewhat incremental, as most components are built upon existing techniques. Additionally, certain aspects of the experimental analysis require further clarification, such as the unexpectedly strong performance of the image-only baseline and minor inconsistencies in reported results. The lack of temporal modeling for sequential rtMRI data also leaves room for improvement. Overall, despite these limitations, the paper presents a solid and practically meaningful contribution. With minor clarifications and revisions, it would be suitable for acceptance.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
A novel three-stage multimodal learning framework is proposed for vocal tract speech organ (tongue, soft palate, lips) segmentation in real-time MRI (rtMRI). A pre-training strategy based on a two-layer cross-modal contrast loss and a decoder based on cross-attention are designed to achieve deep alignment and fusion of visual and acoustic features. A “multimodal during training, monomodal during inference” paradigm is realized: segmentation can be achieved relying solely on a single rtMRI visual modality during the inference phase.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper addresses a key challenge in clinical rtMRI applications: obtaining high-quality synchronized audio is often difficult due to MRI scanner noise, equipment limitations, or patient speech disorders. Its “inference-free” design makes the algorithm feasible. On the 75-Speaker Annot-16 and USC-TIMIT datasets, this method significantly outperforms current state-of-the-art (SOTA) models in both the Dice coefficient and ASD metrics under the challenging US-UT setting. The paper provides robust ablation studies demonstrating the effectiveness of each stage and provides a detailed evaluation of the trade-offs between different visual backbone networks and model performance and inference latency.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
As the authors state in the discussion section, real-time MRI data is a highly dynamic time series. Current frameworks process data frame-by-frame independently, lacking modeling of the context of adjacent frames. This can easily lead to jitter in the predicted segmentation mask along the timeline, especially during rapid speech or vigorous articulation. While the initial aim of this paper is to address the lack of audio availability for patients with tongue cancer or dysarthria, all validations were performed on healthy subject datasets. Pathological articulation organs may exhibit severe anatomical deformities or abnormal movement patterns, and evidence of generalization performance on real-world pathological data is currently lacking.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
This is an excellent paper with a clear purpose, effectively bridging medical image computation and clinical speech science. To further enhance the paper’s impact, it is recommended to provide a link to the open-source code after acceptance. While the methodological details are clearly described, they involve cross-modal alignment and pre-training; having open-source code would greatly promote replication and development within the community.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This research focuses on a highly relevant and underresolved clinical challenge—how to leverage multimodal knowledge for accurate rtMRI tract segmentation in the absence of audio data. The proposed solution is both original in its architectural design and practical in its engineering applications. The experiments are comprehensive, and the results significantly outperform state-of-the-art medical vision models and similar multimodal models.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank all reviewers for their constructive feedback. We propose a three-stage multimodal framework for vocal tract segmentation in real-time MRI that uses audio and phonological information during training but requires only the MRI image at inference. The key idea is that multimodal knowledge gets encoded into model weights during training, enabling audio-free deployment in clinical settings where microphones are impractical or patients have speech disorders. Experiments on two datasets show consistent improvements over both unimodal and multimodal baselines. We address the main concerns below.
To R1 & R3 (lacking modeling of the context of adjacent frames). We fully agree that this is a limitation; our framework is intentionally designed as a strong single-frame foundation upon which temporal modeling can be built. Extension to clip-based segmentation with temporal context is our active future direction. We have already explicitly acknowledged in Sec 4.Discussion.
To R1 & R3 (On real-world pathological data is currently lacking). We fully acknowledge that validation on healthy speakers is a limitation. Publicly available pathological rtMRI datasets (e.g., glossectomy, dysarthria with paired articulator annotations) are currently extremely scarce. We are currently collaborating with clinical partners to acquire pathological rtMRI data to validate the clinical utility of our framework in future work.
To R1 (require bounding-box priors for new speaker). We clarify that bounding-box priors are computed per phonological class per subject using training-set frames only. At inference, the model takes only a single rtMRI frame as input, the bounding-box prior is no longer required.
To R2 (The Image-only variant achieves performance that surpasses most existing unimodal methods). We respectfully clarify that our Image-only variant outperforms most unimodal SOTA methods, and we believe this comparison is fair. All unimodal SOTA methods in Table 1 are re-trained on our datasets under an identical protocol (same splits, 224×224 input…), with each method initialized following its original recipe. The Image-only variant in Table 2A follows the same protocol as Table 1; it is an ablation of our full pipeline that retains the ImageNet-pretrained ViT-B encoder and the decoder, while removing only the audio and phonological inputs. Its advantage over CNN-based methods (e.g., nnU-Net, ResUNet-a) likely stems from the stronger representational capacity of a large-scale ImageNet-pretrained transformer backbone. We deliberately keep the encoder at its ImageNet-pretrained state so that the “+Pretrain” row can cleanly isolate the contribution of our contrastive pretraining. Critically, the gap between Image-only (77.29 DSC) and our full method (86.37 DSC) is +9.08 DSC, demonstrating that multimodal supervision provides substantial and non-trivial gains.
To R2 (Type error). We thank R2 for spotting this. The parameter count for the “DINOv2-B” row has been corrected from “188.64M” to 186.64M; the discrepancy was a typographical error and has been corrected in the camera-ready version.
To R3 (Code release). We will release the code and trained model weights as suggested. The link will be provided in the camera-ready version.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Based on the three reviews, all reviewers recommend acceptance, with consistent evaluation for the well-motivated framework, novel phonological-to-spatial prior, comprehensive evaluation, and strong empirical results. Minor concerns include the lack of temporal modeling, reliance on rtMRI, validation only on healthy subjects, and open-source code after acceptance. Given the clear strengths and unanimous support, my conclusion is Early Accept.