Abstract

Medical Referring Image Segmentation (MRIS) aims to segment target regions in medical images based on natural language descriptions. While multimodal large language models (MLLMs) with next-token prediction offer strong cross-modal understanding, adapting them to MRIS remains challenging: fine-grained boundaries are difficult to preserve under compact representations, diverse clinical descriptions require robust linguistic reasoning, and token-based mask generation lacks inherent spatial coherence. We propose 32TokenMRISeg, an autoregressive (AR) framework that reformulates MRIS as next-token prediction and enables a multimodal LLM to generate segmentation masks directly using a highly compact representation—just 32 discrete tokens from a 1024-entry codebook. To maintain segmentation fidelity under this strict token budget, the framework integrates mask-level supervision to refine boundary details, chain-of-thought prompting to better align clinical descriptions with anatomical cues, and a token-order prediction signal to stabilize autoregressive decoding. Together, these lightweight mechanisms allow 32TokenMRISeg to produce accurate and spatially coherent masks with practical inference efficiency. Experiments on QaTa-COV19 and MosMedData+ demonstrate that our method achieves state-of-the-art performance despite its compact 32-token mask representation.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1432_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{CheXin_Just_MICCAI2026,
        author = { Chen, Xinyu AND Pang, Gaoyang AND Xu, Zixuan AND Wang, Geng AND Yu, Haiyao AND Zhou, Luping AND Li, Yonghui},
        title = { { Just 32 Tokens: Medical Referring Image Segmentation via Next-Token Mask Prediction } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16883},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    In this paper, the author introduces a multimodal auto-regression based framework named 32TokenMRISeg for reference medical image segmentation. Each segmentation mask is represented as 32 discrete tokens from a 1024-entry codebook using a TiTok-based mask tokenizer. To compensate for the information loss with 32 tokens, the authors introduce three auxiliary mechanisms: Mask-Level Supervision (MLS) using BCE + Dice losses on the decoded mask, Chain-of-Thought (CoT) prompting for clinical reasoning, and Token Order Prediction (TOP) to stabilize autoregressive decoding. Experiments on QaTa-COV19 and MosMedData+ report state-of-the-art Dice and mIoU scores.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper achieves SOTA empirical results on QaTa-COV19 (91.53% Dice) and MosMedData+ (79.50% Dice).
    2. The author tried to use MLLM for direct mask token generation.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The motivation is internally contradictory. Premise A: “We use 32 tokens for efficiency. “ Premise B: “32 tokens loses fine-grained information, which hurts segmentation. “ Solution: “So we add MLS + CoT + TOP to compensate for the information loss. “ If the authors need three auxiliary mechanisms to compensate for losses caused by their design choice, then the design choice is not self-evidently correct — it is a problem the authors created and then solved. The natural question is: what if they had just used 64 or 128 tokens and had not needed the compensations, or had needed fewer of them? The paper never asks this. A cleaner framing would be: “we find 32 is sufficient, and here are techniques that make it work” — but this requires actually showing that 32 is sufficient by comparing against larger budgets.

    2.Efficiency is rarely the primary concern in medical image analysis. Accuracy, calibration, robustness across scanners and institutions, and handling of small lesions dominate real-world evaluation — not inference throughput. The paper’s central motivation is therefore weakly grounded in the domain it targets.

    3.The “32 tokens” choice is under-justified. The authors explicitly brand the paper on 32 tokens, but the justification in Section 3.3 is thin: 8 tokens → 84.13% Dice, 16 tokens → 89.69% Dice, 32 tokens → 91.53% Dice. The authors do not test 64, 128, or higher. Without these results, the claim that 32 is the right token budget is not empirically supported.

    4.Novelty is relatively thin. The authors state that “the mask tokenizer is built upon TiTok [12]. “ TOP is taken from cited prior work (Reference [9]). CoT prompting is a standard technique. MLS is a minor adaptation — standard segmentation losses applied through a frozen decoder. Taken together, it is difficult to identify the novel methodological contribution of this paper beyond the integration of existing components.

    5.The original TiTok was designed for natural images. Did the authors retrain TiTok on masks? If so, they should report the quality of the resulting mask tokenizer. No reconstruction FID or Dice ceiling is reported for the mask tokenizer itself, making it impossible to determine whether segmentation errors originate from the tokenizer’s representational limits or from the MLLM’s prediction errors.

    6.Despite being an AR-based method, the baselines should include non-AR MLLM segmentation methods such as LISA and PixelLM. Their absence makes it difficult to assess whether the reported gains come from the proposed 32-token design or simply from using a strong MLLM backbone.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My recommendation rests on three factors. First, the motivation is internally contradictory: the paper compresses to 32 tokens for efficiency, acknowledges the resulting information loss, and then introduces three mechanisms (MLS, CoT, TOP) to compensate — without testing whether 64 or 128 tokens would reduce the need for these compensations. Second, inference efficiency is not a dominant concern in medical imaging, where accuracy, calibration, and robustness matter more than throughput, so the core trade-off is weakly grounded in the domain. Third, novelty is thin: the mask tokenizer is TiTok, TOP is from prior work, CoT is standard, and MLS is a minor adaptation, making the contribution primarily one of integration. The benchmark numbers are strong, but the paper’s “32 tokens” branding is not empirically validated and the motivational foundation does not support the claims being made.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    The rebuttal was not submitted by the authors. In addition, the paper’s motivation is internally inconsistent, and the remaining concerns raised in the review have not been adequately addressed. Overall, the submission does not meet the quality standards expected for a MICCAI paper.



Review #2

  • Please describe the contribution of the paper

    This paper proposes an efficient AR framework for Medical Referring Image Segmentation (MRIS) which aims to segment target regions dictated by natural language. The method generates a highly 32-token representation of discrete tokens from a 1024 codebook. The work utilizes chain-of-thought prompting to articulate the anatomical landmark. To mitigate the exposure bias issue, the paper uses the Token Order Prediction (TOP) mechanism.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.SOTA performance in two of the MRIS benchmarks. 2.The method exhibits high efficiency (33X faster) as it only decodes 32 tokens for binary mask generation. 3.Ablation study includes the individual and synergistic contribution of CoT, MLS, and TOP.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Limited evaluation: The paper has evaluated two datasets: QaTa-COV19 (chest X-rays) and MosMedData+ (lung CT scans). It leaves a room of doubt for the generalizability of the method across other medical datasets. 2.32-Token Constraint: The method might exhibit good results on simple tasks with smoother foregrounds. The robustness of the method is highly dependent on the token length. How can one be sure that 32 tokens are enough for images with complex patterns, such as multiple peppered foreground regions? If not, what should be the extent of the number of tokens to be used for such complex regions, e.g., 64, 128, or more tokens? 3.Reliance on mask tokenizer: The method heavily relies on the ability of the mask tokenizer. How are the authors testing the capability of the tokenizer so that it is meeting the expectation of the method?

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please refer to the weaknesses.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper reformulates MRIS as a next-token prediction task using a multimodal LLM, representing each segmentation mask with only 32 discrete tokens. Three auxiliary strategies (MLS, CoT, TOP) are proposed to compensate for detail loss. Built on InternVL2.5-8B, the method achieves SOTA on QaTa-COV19 and MosMedData+ with ~33x inference speedup.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    S1.The approach is practical and the system-level design has clear value: unifying MRIS with the MLLM next-token prediction paradigm and achieving ~33x inference speedup through 32-token representation, which has obvious clinical utility. S2.The three auxiliary strategies are cleverly designed. Ablation results are good overall — CoT contributes relatively less but the direction is reasonable. S3.SOTA performance is achieved on both MRIS benchmark datasets.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    W1.The evaluation is limited to two COVID-19 lung datasets. The diversity of lesion types and anatomical regions is not sufficient, making it hard to verify the generalizability of the method. W2.The proposed method uses an 8B-parameter model while the compared baselines are orders of magnitude smaller. There is no model-scale ablation to separate the contribution of the method design from the large model capacity. W3.How is the CoT training data constructed? This missing detail hurts reproducibility and credibility. W4.I found a closely related work “NTP-MRISeg” that is neither cited nor compared. Also, HiMTok mentioned in the paper has no experimental comparison either, so it is hard to judge the advantage over the most direct competitors. W5.The token length analysis only covers 8, 16, and 32 tokens. Results for larger token numbers (e.g., 64, 128, 256) are not reported, so it is unclear whether 32 is truly the optimal choice or whether performance continues to improve with more tokens.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The ~33x inference speedup from the 32-token design is the strongest point of this paper — it makes MLLM-based MRIS actually usable in practice, and I value this practical contribution. The SOTA results and solid ablation also support the method. However, I have two main concerns that hold me back from a higher score. First, the evaluation only covers two COVID-19 lung datasets with fixed text formats, which is too narrow to claim general MRIS applicability. Second, comparing an 8B model against baselines that are orders of magnitude smaller without any scale ablation leaves a key question unanswered: how much comes from the method and how much from the big backbone? The missing CoT construction details and the absent comparisons with NTP-MRISeg and HiMTok are also not negligible. Overall I think the paper is slightly above the bar, but if the rebuttal does not adequately address W1, W2, and W3, I may lower my score to 3.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank all reviewers for their thoughtful feedback and recognition of the practical value, 33× inference speedup, and SOTA performance of our approach. We address the concerns below.

Q1: Token Length and Tokenizer Capacity (R1-W3/W5, R2-W2/W3, R3-W5) Our goal is not to claim that 32 tokens are universally optimal, but to show that highly compact AR mask generation can already achieve strong MRIS performance with substantial efficiency gains. This is supported by: (1) new SOTA performance on both benchmarks using only 32 tokens; (2) the ablation from 8→16→32 tokens shows clear diminishing returns but with increased inference burden; and (3) the tokenizer, trained specifically for binary medical masks, shows sufficient reconstruction capacity (Dice > 95%) for this task. Furthermore, the concurrent NTP-MRISeg reports lower performance despite using 1024 tokens, suggesting that substantially longer token sequences do not necessarily improve MRIS performance while incurring much higher autoregressive decoding cost.

Q2: Dataset Diversity (R2-W1, R3-W1) QaTa-COV19 (chest X-ray) and MosMedData+ (CT) cover two distinct imaging modalities with different lesion patterns and spatial distributions, providing meaningful diversity. To our knowledge, nearly all existing MRIS methods, including the compared MICCAI works in Table 1, are evaluated only on these two public benchmarks, making our evaluation consistent with prior work.

Q3: Missing Baselines and Related Works (R1-W6, R3-W4) LISA and PixelLM use AR MLLM backbones (LLaVA-series), but rely on external mask decoders from segmentation-token hidden states rather than autoregressively generating discrete mask tokens within the MLLM vocabulary space as in our framework. Neither reports results on MRIS benchmarks. NTP-MRISeg is a concurrent arXiv work using 1024 tokens. According to its reported results, our method achieves higher performance despite using only 32 tokens (QaTa-COV19 mIoU: 91.53% vs 91.10%; MosMedData+: 79.50% vs 79.18%), while substantially reducing autoregressive decoding cost. HiMTok mainly studies hierarchical tokenization, whereas our work focuses on stabilizing highly compact AR-based MRIS through MLS, CoT, and TOP under an extreme 32-token setting. We will add discussion of these related works.

Q4: Motivation, Efficiency and Novelty (R1-W1, W2, W4) Our work targets to address the inference efficiency bottleneck of autoregressive (AR) MLLMs for practical MRIS deployment under SOTA performance guarantee. Recent AR-based segmentation frameworks, including concurrent NTP-MRISeg, highlight growing interest in unifying multimodal reasoning and segmentation within generative frameworks (see Q3). However, existing AR approaches typically rely on long token sequences or external mask decoders, leading to substantial decoding latency, while simply reducing token length degrades spatial fidelity and boundary quality. Our core contribution is enabling accurate AR-based MRIS using only 32 tokens, achieving over 33× faster inference with new SOTA accuracy. MLS, CoT, and TOP are designed to address the resulting challenges of boundary preservation, clinical reasoning alignment, and decoding stability under aggressive token compression.

Q5: Model-Scale Ablation (R3-W2) Table 2 shows that, using the same 8B backbone, MLS, CoT, and TOP improve performance from 88.51% to 91.53% Dice. Notably, under the highly compact 32-token setting, the baseline model without these components does not outperform smaller non-MLLM methods such as GuideDecoder and TGCAM in Table 1, suggesting that the gains are not solely from backbone scale.

Q6: CoT Training Data Construction (R3-W3) CoT annotations are derived from the structured clinical labels provided in both datasets, including laterality, lobe involvement, and lesion count. These structured attributes are converted into stepwise reasoning prompts that guide the model to localize pathological regions before mask generation.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    Please try to address the pointed weaknesses during the rebuttal.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The reviewers appreciate the significance of the problem and extensive experimental results, and the initial concerns have been mostly addressed during the rebuttal. I recommend accepting this submission.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Reject

  • Please justify your recommendation.

    The paper addresses an interesting and timely direction by formulating medical referring image segmentation as an autoregressive mask-token generation problem. The reported results on QaTa-COV19 and MosMedData+ are strong, and the 32-token representation suggests a potentially efficient way to integrate segmentation into an MLLM-style generation framework. However, after considering the reviews and the authors’ response, several important concerns remain unresolved. First, the main empirical comparison is not fully convincing: the proposed method uses a large multimodal language model as the core Transformer block, while many baselines are CNN- or Transformer-based segmentation models. As a result, it is difficult to determine whether the improvement comes from the proposed 32-token autoregressive formulation or simply from the stronger MLLM backbone. More direct comparisons with reasoning segmentation methods, such as LISA, PixelLM, NTP-MRISeg, or related MLLM-based segmentation frameworks, would be necessary to support the claimed advantage. Second, the 32-token design is still under-justified. The paper only compares 8, 16, and 32 tokens, without testing larger budgets such as 64 or 128, so it remains unclear whether 32 tokens are actually sufficient or just a chosen efficiency point. Third, the CoT component is not described in enough detail: it is unclear whether the reasoning chains are manually designed or automatically generated, and how they are used during training and inference, including whether the setting is zero-shot, one-shot, or supervised with constructed reasoning annotations. Finally, the TOP mechanism is also insufficiently explained, with limited analysis of how it improves spatial consistency and segmentation quality beyond stabilizing token order. Overall, while the paper reports promising numbers and explores a useful direction, the current evidence does not adequately separate backbone strength from the proposed method, nor does it sufficiently justify the key 32-token, CoT, and TOP design choices. I would therefore be hesitant to recommend acceptance in its present form.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Overall, the paper makes a credible and practically relevant contribution by showing strong MRIS performance with a very compact AR mask representation, and the rebuttal partially addresses the main technical concerns. Given the strong results and efficiency claim, balanced against moderate novelty and narrow validation, my recommendation is accept.



back to top