Abstract

Fine-grained representation learning is crucial for retrieval and phrase grounding in chest X-rays, where clinically relevant findings are often spatially confined. However, existing methods remain limited on these tasks, as contrastive models lack explicit region-level supervision and large vision language models often fail to generalize fine-grained representations under external validation. To address these limitations, we propose Location-aware Fine-grained representation learning (LoFi), which jointly optimizes sigmoid, captioning, and location-aware captioning losses using a lightweight large language model. The location-aware captioning loss enables region-level supervision through grounding and dense captioning objectives, thereby facilitating fine-grained representation learning. Building upon these representations, we integrate a fine-grained encoder into retrieval-based in-context learning to enhance chest X-ray grounding across diverse settings. Extensive experiments demonstrate that our method achieves superior retrieval and phrase grounding performance on MIMIC-CXR and PadChest-GR.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2716_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/myeongkyunkang/lofi

Link to the Dataset(s)

N/A

BibTex

@InProceedings{KanMye_LoFi_MICCAI2026,
        author = { Kang, Myeongkyun AND Yang, Yanting AND Li, Xiaoxiao},
        title = { { LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-Ray } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16878},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper introduces Location-aware Fine-grained representation learning with the purpose of training a vision-language model able to localize and describe fine-grained features of medical images better than previous approaches. Following previous work, the authors trained their model on MIMIC-CXR and additionally evaluated it on PadChest-GR. Results for retrieval-based in-context learning on both datasets show performance superior to prior work.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper merges findings from two papers (SigLip and LoCa) to propose one unified training framework and applies it to radiology. Comparison with previous methods is thorough and results are convincing.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The methodological novelty is limited to the combining of two previously introduced methods and its application to two radiology datasets.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The technical novelty of the paper is somewhat limited, but results are extensive and convincing, showcasing how methods previously proposed in non-medical literature can be successful when applied to radiology.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors made the suggested edits to improve clarity of exposition and addressed the reviewers concerns appropriately.



Review #2

  • Please describe the contribution of the paper

    This paper presents a location-aware method for fine-grained representation learning by optimizing sigmoid, captioning, and location-aware captioning objectives. This method enables learning from long-form text and region-level supervision. Their results indicate its superior performance on phrase grounding and retrieval tasks.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Fine-grained representation learning can be very important for disease localization and explainability purposes. 2.Both internal and external validation results were provided. 3.The proposed approach was compared against a high number of baselines. 4.Visual results can better indicate the problem. 5.The used dataset is oen-source, improving the reproducibility of the work.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Have the authors read the Gloria paper? (Huang, Shih-Cheng, et al. “Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.” Proceedings of the IEEE/CVF international conference on computer vision. 2021.). It proposed an approach for learning both global and local associations between images and texts. 2.Some parts of the paper are difficult to understand and follow. Also, some introduction sentences are repetitious. 3.I recommend moving “Implementation Details” into the “Experiments and Results” section. 4.Did the authors conduct statistical tests to evaluate the significance of the results? 5.It’s not clear how “5%” was calculated on Page 8, Line 3.6.I’m not familiar with Ro/L and Ro/s metrics and didn’t find clear definitions for them. 7.In Equations 1,2, and 3, Does E refer to the expected value? Please explain the equations more clearly.

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    1. Did the authors provide the results using only one run? 2.Did the authors make sure that the data related to one patient does not get split across the training, validation, and test sets?
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    poor clarity, insufficient referencing to prior work, not enough explanations for used metrics

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    Overall, I found the paper difficult to understand and follow and was not fully convinced by the provided rebuttal, e.g., regarding the comparison to prior work (GLORIA).



Review #3

  • Please describe the contribution of the paper

    The paper introduces LoFi, a location-aware fine-grained representation learning framework for chest X-rays that addresses the lack of region-level supervision in existing contrastive and vision-language models. The key contribution is a unified training objective that combines contrastive (sigmoid), captioning, and location-aware captioning losses, enabling the model to leverage both global report information and region-level grounding signals without requiring fully annotated datasets. By integrating these fine-grained representations into a retrieval-based in-context learning pipeline, the method improves both retrieval and phrase grounding performance, particularly in external validation settings where generalization is challenging.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Multi-objective formulation for fine-grained learning: The paper uses a unified training objective that jointly combines contrastive (sigmoid), captioning, and location-aware captioning losses. The key idea is in integrating grounding (box prediction) and dense captioning into an autoregressive LLM framework, enabling implicit region-level supervision even when bounding box annotations are sparse.

    • Effective use of heterogeneous supervision: The method leverages large-scale radiology reports together with partially available text-box pairs in a principled way. Instead of requiring fully annotated datasets, it uses location-aware captioning objectives to bridge weak (report-level) and strong (box-level) supervision.

    • Comprehensive evaluation across retrieval and grounding: The paper demonstrates consistent improvements over multiple baselines on both retrieval and phrase grounding, including external validation using retrieval-based in-context learning.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Limited methodological novelty relative to prior location-aware captioning work: While the paper claims a novel “location-aware captioning loss,” the core idea of jointly predicting captions and bounding boxes via an autoregressive framework closely resembles prior work such as LocCa (Wan et al., NeurIPS 2024), which already formulates visual pretraining as a combination of caption generation and location prediction tasks (e.g., grounded captioning and referring expression prediction) . LoFi mainly adapts this paradigm to the medical domain with a lightweight LLM, but does not introduce a fundamentally new learning formulation beyond combining existing objectives.

    • Incremental use of retrieval-based in-context learning (ICL): The integration of retrieval-based ICL for grounding is not novel and follows prior work showing that performance depends heavily on retrieved demonstrations (e.g., Yang et al., 2022 as cited in the paper). The paper does not introduce new retrieval strategies, scoring mechanisms, or new insights, only replaces the encoder with a fine-grained one.

    • Reliance on noisy or weak supervision without sufficient analysis: The model is trained using large-scale silver-standard annotations (e.g., machine-generated text-box pairs from MIMIC-Ext), which are known to contain noise and inconsistencies. However, the paper does not provide a detailed analysis of how annotation quality affects performance, nor does it compare against human-annotated subsets. This is important because prior work has highlighted challenges with spatial annotation quality and inter-observer variability in medical grounding tasks.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    NA

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    NA

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses an important problem in medical vision-language learning, fine-grained representation learning for chest X-rays, and proposes a practical framework that effectively combines contrastive learning with location-aware captioning objectives. The approach is well-motivated and demonstrates consistent empirical gains across both retrieval and grounding tasks, including strong external validation, which strengthens its impact. However, the methodological novelty is somewhat incremental, particularly given similarities to prior location-aware captioning and retrieval-based ICL frameworks, and the paper lacks deeper analysis of annotation quality and clinical relevance.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The main remaining concern is the limited novelty, as the framework primarily combines existing ideas from global-local medical VLP and location-aware captioning rather than introducing a fundamentally new formulation. However, the paper addresses an important problem, demonstrates good empirical performance, and the rebuttal adequately clarifies the practical contributions and experimental concerns, making the work overall suitable for acceptance.



Author Feedback

We thank the reviewers for their constructive feedback. We address the major concerns below.

Novelty beyond prior work: We respectfully disagree that our work is a straightforward extension of prior work. Notably, prior work leaves two concrete gaps. (G1) Contrastive CXR models, GLoRIA, CARZero [11], and RadIR [28], align local features with text IMPLICITLY via attention-weighted similarity; none consume explicit bbox coordinates even when available (e.g., Chest ImaGenome), thereby upper-bounding their fine-grained signal. We benchmark against CARZero/RadIR, which prior work reports surpass GLoRIA. (G2) LocCa [23] shows that explicit grounding/dense-captioning produces strong location-aware features, but only at web scale: 1B pairs from scratch with unreleased weights, which is infeasible at medical scale (227K reports). Filling G1+G2 jointly is non-trivial: (a) medical captioners are not region-aware; (b) region-aware encoders (e.g., SigLIP2) are not generative; (c) joint contrastive+autoregressive training on 227K data is brittle. LoFi’s four choices each address a specific obstacle: (i) SigLIP2 [21] encoder — the strongest open patch-localized semantic features without billion-scale pretraining; (ii) Gemma3-270M decoder — small enough for 227K without overfitting; (iii) attention pooling (64 tokens) + LoRA — bridging the encoder/LLM with few trainable parameters for medical-scale stability; and (iv) our newly designed loss enables retrieval (absent in LocCa) and injects explicit bbox supervision (absent in GLoRIA/CARZero/RadIR). Fig 4 confirms that each loss contributes across all metrics; §1 will be rewritten to lead with this framing. Empirically, we show that no prior family matches LoFi’s SOTA results on external grounding and retrieval.

Evaluation protocol and data splitting (R2): We clarify that, following a foundation-model pretraining protocol, Tables 1–3 report the performance of the pretrained checkpoint from the last training epoch, which is intended for release. For Table 4, we used the official PadChest-GR train/val/test split, ensuring patient-level separation. Although multi-seed reporting could further strengthen the results in Table 4, FT w/ Ours clearly improves over FT w/o Ours (F@0.5: 28 to 33; run-to-run variance was small) and is also comparable to the much larger, private-data-trained MAIRA-2.Therefore, we reported the single-run performance of the fine-tuned checkpoint intended for release.

Retrieval-based ICL for bbox prediction (R3): We clarify that retrieval-based ICL serves as an evaluation setting for testing whether the learned fine-grained encoder generalizes to practical chest X-ray grounding scenarios. However, our results provide several insights beyond this evaluation setting. First, to the best of our knowledge, LoFi is the first to demonstrate effective ICL for bbox prediction, whereas prior ICL studies largely focus on VQA. Second, retrieval-based ICL offers a practical solution to key medical challenges in external validation, such as inter-observer variability. Third, we show that fine-grained retrieval is critical for ICL-based grounding and that LoFi-based retrieval improves the quality of retrieved demonstrations.

Annotation noise (R3): We agree that tight abnormality-level boxes can suffer from inter-observer variability. However, LoFi is trained with loose anatomical-level boxes (e.g., left lower lung) rather than tight abnormality boxes, making the supervision less ambiguous. Furthermore, MIMIC-Ext [17] shows that its annotations are more reliable than alternatives, and we further used only its highest-quality A++-rated text-box pairs for training. Restricting training to a small human-annotated subset would make region-level supervision much less frequent during training; since LoFi learns from non-box samples via the captioning loss, we expect this setting to behave similarly to our ablation w/o Lg,d.

Notation (R2): We will clarify the notation and metric definitions.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper proposes LoFi, a location-aware fine-grained vision–language representation learning framework for chest X-rays, which jointly optimizes sigmoid, report captioning, and location-aware captioning losses, using a lightweight large language model. The learned encoder is used in a retrieval-based in-context learning pipeline, yielding improved retrieval and phrase grounding performance on MIMIC-CXR and PadChest-GR datasets.

    The paper addresses the timely and relevant problem of fine-grained representation learning and phrase grounding for disease localization and explainability in CXRs. It presents a well-motivated unified training approach that combines global and local objectives in a practical way. Empirical evaluation against multiple baselines on two datasets is generally convincing. However, the following concerns should be addressed: 1) Related work: Insufficient related work discussion of closely related global-local medical VLP methods (e.g., GLoRIA) and prior location-aware captioning work (e.g., LocCa). 2) Novelty: Incremental as primarily demonstrated in combining existing ideas (e.g., SigLIP-style contrastive learning with location-aware captioning/grounded captioning paradigms such as LocCa) and applying them to radiology. 3) Clarity and presentation: Paper parts can be repetitive and difficult to follow, and some metrics and equations require clearer definitions and explanations. 4) Statistical reporting: Significance testing or reporting variance across runs is required. 5) Data splitting/leakage: It is unclear whether results are from a single run and that patient-level separation is enforced across train/val/test splits. 6) Noise in weak supervision: silver-standard text–box pairs may be noisy; the paper would benefit from analysis of annotation quality sensitivity and/or comparison to a higher-quality subset.

    I invite the paper for rebuttal to address the issues highlighted by the reviewers.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    After considering the initial reviews, the authors’ rebuttal, and the reviewers’ post-rebuttal replies, I recommend acceptance. The rebuttal clarified the intended novelty relative to prior work, evaluation protocol, data splitting and annotation noise. While concerns about incremental novelty (compared to GLoRIA) and paper clarity remain valid, the overall evidence is positive and the contribution is practically meaningful.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper addresses phrasal grounding directly within LLM frameworks. Earlier work had bounding box prediction done separately. Since LLMs are not as good at regression tasks, it was interesting to see the method still works. Not all concerns of the meta-reviewer appears to have been addresses.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Despite some limitations regarding clarity and novelty, I believe localization is an important problem in medical vision-language pre-training, and the authors made significant performance gains in this direction.



back to top