List of Papers Browse by Subject Areas Author List
Abstract
Thyroid cancer is the ninth most common malignancy globally, with Papillary Thyroid Carcinoma (PTC) representing the most prevalent subtype. The BRAF V600E mutation has emerged as a decisive diagnostic and prognostic biomarker of PTC, essential for risk stratification and personalized therapy. However, its gold-standard detection relies on complex and expensive sequencing technology. Therefore, developing advanced deep learning methods to predict BRAF mutation by integrating multimodal clinical data offers a cost-effective solution to facilitate accurate diagnosis and individualized risk stratification. In this paper, we present the first comprehensive multimodal dataset for thyroid cancer BRAF V600E mutation prediction, MM-BRAF dataset, integrating 3120 high-quality ultrasound images, annotated tumor regions, ultrasound reports, clinicopathological records, and BRAF mutation status. Furthermore, we propose BRAF-MLLM, a novel and first Multimodal Large Language Model (MLLM) framework for thyroid cancer BRAF V600E mutation prediction. Given that mutation-relevant signatures in ultrasound imagery are often subtle and localized within minuscule tumor regions, we design the Tumor-centric Multimodal Dual-scale Alignment (TMDA) module. During the training stage, TMDA explicitly decouples visual features into intra-tumoral and extra-tumoral scales, aligning them with fine-grained clinical semantics to suppress background noise and enhance the perception of tumor-specific indicators. To validate the effectiveness of our approach, we establish a comprehensive benchmark for this novel task using traditional multimodal fusion models and mainstream MLLMs. Extensive experiments subsequently demonstrate that BRAF-MLLM consistently outperforms all baseline methods, showcasing its significant potential for precise clinical decision-making.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2492_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/FiFi-HAO467/BRAF-MLLM
Link to the Dataset(s)
https://github.com/FiFi-HAO467/BRAF-MLLM
BibTex
@InProceedings{HaoPen_Toward_MICCAI2026,
author = { Hao, Pengfei AND Li, Shuaibo AND Wang, Hongqiu AND He, Sixu AND Huang, Chunwang AND Zhu, Lei},
title = { { Toward Thyroid Cancer BRAF V600E Mutation Prediction via Multimodal Large Language Model: Dataset and Model Development } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
1.The manuscript introduced a multimodal dataset (MM-BRAF) for thyroid cancer BRAF V600E mutation prediction, comprising comprehensive multimodal data: ultrasound images, annotated tumor regions, ultrasound reports, BRAF mutation status, and clinicopathological records. 2.The authors developed an effective model, BRAF-MLLM, for predicting the BRAF V600E mutation in thyroid cancer. Meanwhile, the developed Tumor-centric Multimodal Dual-scale Alignment module could capture interactions and enhance alignment across modalities. 3.The authors implemented several experiments to demonstrate the performance of the proposed BRAF-MLLM on BRAF V600E mutation prediction.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The manuscript developed a valuable multimodal dataset for thyroid cancer BRAF V600E mutation prediction. The proposed dataset contains multimodal data, including ultrasound images, annotated tumor regions, ultrasound reports, BRAF mutation status, and clinicopathological records. 2.The dataset construction utilized LLM as a synthesis agent, which efficiently reduced human annotation effort.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Anywhere we can access the dataset? For the dataset articles, I strongly suggest that the authors include detailed information about dataset access in the article, or at least indicate whether the access link will be released in the future. 2.Missing single-modal baselines. I would suggest the author consider adding single-modal baselines in the future. It would be meaningful to first demonstrate that the multimodal method outperforms the unimodal baseline, then determine which type of multimodal method performs best. Otherwise, it is unclear whether the benefits come from multimodal data fusion or whether the unimodal baselines are good enough for BRAF prediction, while multimodal data fusion actually decreases the performance. 3.For dataset construction: (a) Could the author indicate the cost of using GPT-4o to synthesize and structure the raw data? It would be helpful for others to evaluate the cost of reproducing this dataset. (b) Are the LLM-synthesized ultrasound reports and clinicopathological information used directly for further model training? Could any data quality control and verification be considered to ensure the reliability of the data? 4.To better demonstrate the value of the dataset, I would expect a comprehensive evaluation across multiple VLMs rather than a single one. Ideally, the experiment in the article should be divided into two parts: (1) Methodology-wise comparison: providing multiple baseline models for the proposed method, which have been included in this article; (2) A comprehensive evaluation to show the performance of finetuning multiple VLMs using the proposed dataset, which is missing in the article.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The recommendation is made based on whether the article: 1.Demonstrated the reproducibility and accessibility of the dataset. 2.Provided a sufficient baseline comparison for the proposed method. 3.Provided a solid demonstration of the value of the developed dataset.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have resolved most of my concerns in the response.
Review #2
- Please describe the contribution of the paper
The paper introduces BRAF‑MLLM, the first multimodal large language model (MLLM) specifically designed for predicting the BRAF V600E mutation in thyroid cancer. To enable this task, the authors construct the MM‑BRAF dataset, a collection that combines high‑quality ultrasound images (with tumor‑region annotations), corresponding ultrasound reports, clinicopathological records, and confirmed BRAF mutation status. The core technical novelty lies in the Tumor‑centric Multimodal Dual‑scale Alignment (TMDA) module, which explicitly separates visual features into intra‑tumoral and extra‑tumoral scales and aligns each scale with fine‑grained clinical semantics, thereby suppressing background noise and enhancing the detection of subtle, mutation‑relevant signatures.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Multimodal dataset: The MM‑BRAF dataset is the first publicly‑oriented collection that integrates US imaging, textual reports, and pathology for BRAF V600E prediction, filling a clear gap in thyroid‑cancer AI research.
First MLLM for this task: BRAF‑MLLM is the inaugural attempt to apply a large language model to multimodal mutation prediction, moving beyond conventional fusion networks.
TMDA module: By decoupling features into intra‑tumoral and extra‑tumoral scales and performing dual‑scale cross‑modal alignment, TMDA directly addresses the challenge of subtle, localized ultrasound signatures, a design not seen in prior multimodal fusion or MLLM works.
Comprehensive benchmarking: The authors evaluate a wide spectrum of baselines, several state‑of‑the‑art multimodal LLMs and supervised fine‑tuning variants, providing a transparent performance landscape.
Open‑source intent: The authors intend to release both the MM‑BRAF dataset and the code, which would be a valuable resource for the community.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Marginal gains: Although BRAF MLLM achieves the highest numbers, the absolute improvements over the strongest baselines modest; the clinical relevance of such incremental gains is uncertain. Lack of statistical analysis: No confidence intervals or significance tests are reported, making it impossible to judge whether the observed differences are statistically robust. Clinical acceptability: The reported accuracy (≈77 %) and recall (≈71 %) fall short of typical clinical decision thresholds for mutation testing, which often demand high sensitivity. Training details insufficient: The choice of “2 epochs” during fine tuning is not justified; a technical validation (e.g., learning curve analysis) demonstrating that 2 epochs is optimal is missing. Limited exploration of LoRA: The paper relies solely on LoRA based fine tuning of the MLLM. Prior work suggests that sequential fine tuning on other medical multimodal datasets can substantially improve downstream performance; a discussion or experiment on this would strengthen the methodology. Figure quality: Figure 1 contains cluttered elements, ambiguous color gradients, and unnecessary details that hinder interpretability. A cleaner redesign (manual or “controlled” AI assisted) is needed. Minor textual issues: – The abbreviation “MLLM” is introduced without its full form in the introduction (“…the first MLLM based framework…”). – A typo appears: “first omprehensive multimodal dataset” should read “first comprehensive multimodal dataset”. Broader disease relevance omitted:BRAF V600E also drives malignancy in other tumors (e.g., pediatric low grade gliomas). Discussing this broader relevance would enrich the impact statement and suggest future research avenues. Metric presentation: Results are reported with four decimal places, which exceeds clinical significance; rounding to three decimals would improve readability. Reproducibility details: Please confirm the dataset, codes, and models will be publicly available.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The manuscript delivers a novel and practically relevant contribution: a first‑of‑its‑kind multimodal large language model for BRAF V600E mutation prediction, supported by a newly curated, richly annotated dataset. The TMDA alignment mechanism is an original technical advance that directly tackles the difficulty of subtle ultrasound signs. The authors’ commitment to open‑sourcing the data and code adds substantial community value.
While there are certain methodological gaps (absence of statistical analysis, limited justification for training hyper‑parameters, modest absolute performance gains) and presentation issues (figure clarity, minor typographical errors), these shortcomings are addressable through revisions and do not undermine the core scientific contribution. Consequently, the paper merits acceptance.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper introduces a multimodal dataset for thyroid cancer BRAF V600E mutation prediction that combines ultrasound images, tumor annotations, ultrasound reports, clinicopathological records, and mutation labels. It further proposes an MLLM-based framework, BRAF-MLLM, with a tumor-centric multimodal dual-scale alignment module that separately aligns intra-tumoral and extra-tumoral visual features with fine-grained clinical text semantics for mutation prediction.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper studies a clinically relevant problem. Predicting BRAF V600E mutation from non-invasive multimodal data is meaningful for thyroid cancer risk stratification and treatment planning. 2.The dataset construction effort is valuable. The proposed MM-BRAF dataset integrates ultrasound images, tumor region annotations, reports, clinicopathological information, and mutation labels, which is richer than prior single-modal settings. 3.The proposed method is coherent. The tumor-centric dual-scale alignment design is intuitively motivated by the fact that mutation-related cues may reside both within the tumor and in surrounding context. 4.The paper includes baseline comparisons and ablation studies, and the proposed method shows improved performance over the reported baselines on the internal dataset.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The methodological novelty is limited. The proposed framework is largely a straightforward adaptation of an existing MLLM with LoRA fine-tuning plus region-level/text-level contrastive alignment. The main technical component, TMDA, is essentially a dual-branch alignment strategy based on clinician-provided tumor masks and LLM-parsed text entities, which feels more like a task-specific engineering integration than a substantially new methodological advance. 2.The experimental validation is not yet strong enough for a top-tier venue. All experiments appear to be conducted on a single internally collected dataset, and the paper itself acknowledges future work on multi-center validation. Without external validation or cross-center testing, it is difficult to assess robustness and clinical generalizability. 3.The benchmark design is not fully convincing. Several compared MLLMs are evaluated in a zero-shot or off-the-shelf manner, whereas the proposed model is fine-tuned on the target dataset using Qwen2.5-VL-7B with LoRA. This makes the comparison less balanced, especially when the strongest baseline is also the same backbone after SFT. The improvement over the fine-tuned Qwen2.5-VL-7B baseline is relatively modest. 4.The dataset protocol and clinical details are insufficiently reported. The paper mentions 3120 samples from 1258 patients and patient-level splitting, but many important details remain unclear, such as class distribution in train/test, acquisition variability, inclusion criteria, annotation protocol and agreement, missing-value handling in clinicopathological records, and whether multiple views per patient may still introduce bias despite patient-level splitting. 5.The use of GPT-4o in dataset curation raises reproducibility and data-quality concerns. The paper states that GPT-4o is used to collate original records into ultrasound reports and clinicopathological information, but the prompting protocol, quality control, and possible error propagation are not described in enough detail. This is particularly important because these generated text fields are central inputs to the final model. 6.Clinical relevance is still somewhat preliminary. The reported gains are based on retrospective internal data only, and there is limited analysis of failure cases, calibration, subgroup robustness, or how the model would be used in realistic clinical workflows. For a clinically oriented MICCAI paper, this reduces the practical impact.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(2) Reject — should be rejected, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper tackles a relevant clinical task and presents a coherent multimodal framework, but I do not find the current submission strong enough for acceptance. The main reasons are: (1) the technical novelty is moderate, with the proposed method largely consisting of fine-tuning an existing MLLM backbone augmented by a relatively standard region/text alignment design; (2) the evaluation is limited to a single internal dataset without external or multi-center validation, which makes it difficult to judge robustness and real clinical value; (3) several important dataset and protocol details are insufficiently specified; and (4) the comparison setting is not fully balanced, since the proposed method is fine-tuned while several general MLLM baselines are not. Overall, while the direction is interesting, the current level of methodological innovation and experimental rigor is below the acceptance bar for MICCAI.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We sincerely thank the reviewers for their constructive suggestions. We are encouraged that they recognized the clinical importance of BRAF V600E prediction (R1, R2, R6), the value of MM-BRAF (R1, R2, R6), and the promise of TMDA (R1, R2). Below, we address key concerns:
Common Concerns:
Dataset Accessibility (R1,R2,R6): We will release the dataset, code, prompts, splits, and trained weights upon publication.
GPT-4o structuring (R1,R6): GPT-4o was used only to map clinician-recorded numeric/categorical fields into standardized textual descriptions, not to infer BRAF status or generate new clinical findings. Mutation labels were excluded from all prompts. Outputs were checked against the original records and clinically verified; inconsistent cases were manually corrected. The total cost for 3,120 records was ~$15, and prompts/protocols will be released for reproducibility.
Fine-tuned VLM comparisons (R1,R6): For a fairer benchmark, we additionally fine-tuned representative VLMs under the same patient-level split: InternVL2.5-8B: 0.7395 Acc, 0.6331 F1, 0.6571 Recall; Qwen2-VL-7B: 0.7268 Acc, 0.6294 F1, 0.6350 Recall; MiniCPM-2.6B: 0.7270 Acc, 0.6684 F1, 0.6728 Recall. BRAF-MLLM remains best on all metrics, showing the gain is not merely from VLM fine-tuning but from TMDA.
#R1
Q2: Thanks for the suggestion. Table 2 includes image-only M1, text-only M2, and multimodal M3; M3 outperforms both M1 and M2, confirming the value of multimodal fusion. We will highlight these unimodal baselines more clearly.
#R2
Q1: We appreciate this comment. The gain is moderate but consistent across Acc/F1/Recall, especially Recall, reducing missed mutation-positive cases. Our primary contribution is the MM-BRAF benchmark, with BRAF-MLLM as a strong reference model.
Q2: 95% CIs (1,000 bootstraps): Acc 0.774[0.757-0.796], F1 0.698[0.670-0.720], Rec 0.713[0.702-0.731]. McNemar vs. best baseline: p=0.026<0.05.
Q3: We agree BRAF-MLLM should not replace molecular testing. It is intended as non-invasive decision support for risk stratification and prioritizing confirmatory testing. We will clarify this positioning.
Q4: We selected 2 epochs by validation Acc, which peaked at epoch 2: 0.7521/0.7714/0.7617/0.7459 for epochs 1/2/3/4, suggesting overfitting afterwards.
Q5: Thanks for the guidance. LoRA was used for parameter-efficient adaptation under limited medical data. Sequential medical multimodal pretraining is complementary to TMDA and may further improve performance; we will discuss it.
Q6-9: We will revise Fig. 1, define MLLM, correct typos, round metrics, confirm data/code/model release, and discuss broader BRAF V600E relevance.
#R6
Q1: BRAF-MLLM introduces a clinically motivated TMDA module, not generic dual-branch fusion. By aligning intra-/extra-tumoral visual cues with corresponding clinical semantics, it targets subtle mutation-related ultrasound signatures.
Q2: We acknowledge that multi-center validation is important for clinical generalizability. Given the scarcity of paired imaging-clinical-genomic data, our key contribution is the first open multimodal benchmark enabling such future validation.
Q4: Dataset details: train/test contain 2,184/936 images, with 56.36%/66.99% positives. Inclusion required confirmed cancer, BRAF status, US images, and clinicopathological data; poor-quality images or missing labels were excluded. Tumor masks annotated by two clinicians were reconciled by a senior expert. Patient-level splitting prevents leakage across multiple views from the same patient.
Q6: As the first multimodal BRAF benchmark, our study establishes MLLM feasibility for this task. BRAF-MLLM is intended to support risk stratification and prioritizing confirmatory testing, not replace molecular testing. We will add failure-case and subgroup analyses to clarify utility and limitations.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a potentially valuable multimodal dataset for BRAF V600E mutation prediction in thyroid cancer, along with an associated multimodal LLM for mutation prediction. The rebuttal should address concerns about both the model and the dataset: (1) baseline comparison is not fully balanced, as only one external model underwent supervised fine-tuning (R1R3); (2) the reported accuracy of 77% and recall of 71% may not meet clinical acceptability for mutation testing (R2); (3) validation is limited to a single site, leaving generalizability unclear (R3). Regarding the benchmark itself, (4) the use of GPT-4o for data generation raises concerns about cost, reproducibility, and data quality control (R1R3); (5) important dataset protocol and clinical details are missing (R3). Overall, it is unclear whether the paper’s primary contribution is the benchmark or the model development, and either direction requires more details and justification.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper proposes a multimodal dataset for thyroid cancer BRAF V600E mutation prediction, along with a task-specific multimodal LLM. The reviewers’ concerns were largely addressed in the rebuttal. Further validation, especially multi-center validation, is still needed. However, the paper represents an important first step in demonstrating the value of this new dataset for mutation prediction and risk stratification. The authors should commit to releasing the dataset, prompts, splits, and model weights.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Important information: Despite my reminder, reviewer 6 failed to fill in the post-rebuttal evaluation. His/her opinion regarding the rebuttal remains unknown. I did read the paper in detail in order to compensate for this.
Justification: I follow here the opinion of reviewer 1 post-rebuttal which is confirmed by the positive views of reviewer 2.I think that the authors did a really work in the rebuttal, but at the same time missed significant relevant information in the original article. Given the rebuttal I agree with reviewer 1 that most of the concerns are removed. While reviewer 6 did not assess the rebuttal, I see that criticisms 3, 4, and 5 are well addressed. I do agree with reviewer 6 on the statement in 1, but the value of a public dataset of this type overcomes this limitation. Reviewer 6’s criticisms 2 and 6 I see far less critical. Having said that, I really hope the authors take the content of the rebuttal and manage to put it in the camera-ready version. Else, MICCAI would have an incomplete work.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Post-rebuttal, R1 and R2 recommend accepting the paper. Though R6 has not responded with a final decision, the rebuttal has addressed most of the concerns raised and should therefore be considered for publication.