Abstract

Speech sound disorder (SSD) affects many school-aged children and can lead to long-term academic, social, and mental health difficulties. Speech-language pathologists use ultrasound, a non-invasive imaging modality, to visualize tongue motion during speech and provide biofeedback treatment. The American English rhotic (r sound) is particularly challenging, as distinct articulatory strategies can produce the same perceived sound. Label-efficient self-supervised learning systems are increasingly being developed to automate scalable and objective SSD assessment. However, existing work has primarily focused on phoneme-level classification and provides limited insight into within-phoneme articulatory variations. In this paper, we present the first self-supervised learning framework that focuses on within-phoneme articulatory structure of American English rhotic production from lingual ultrasound video of children with residual SSD. We further introduce speaker adversarial regularization to reduce inter-speaker variability while preserving articulatory structure in the learned representations. Our results on the recently curated PERCEPT-US multimodal corpus demonstrate that self-supervised learning without task specific fine-tuning yields embeddings that accurately discriminate between perceptually accurate and inaccurate rhotic productions (mean ACC = 0.974; mean MCC = 0.941) and between articulatory subtypes of perceptually accurate rhotics, namely retroflex and bunched configurations (mean ACC = 0.961; mean MCC = 0.897). Code is available at https://github.com/kimtae55/BYOL-VS.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3947_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/kimtae55/BYOL-VS

Link to the Dataset(s)

N/A

BibTex

@InProceedings{KimTae_Selfsupervised_MICCAI2026,
        author = { Kim, Taehyo AND Eads, Amanda AND McFee, Brian AND McAllister, Tara AND Shu, Hai},
        title = { { Self-supervised Learning on Lingual Ultrasound Video Encodes Clinically Meaningful Articulatory Structure of Rhotic Production in Residual Speech Sound Disorder } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16880},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper addresses the challenge of learning articulatory representations from lingual ultrasound videos for children with speech sound disorders (SSD), specifically focusing on the complex American English rhotic sound. The authors propose BYOL-VS, a joint embedding self-supervised learning (SSL) framework that integrates a video vision transformer and a speaker adversarial network. The adversarial component uses a gradient reversal layer to suppress inter speaker variability, which is a major confounding factor in ultrasound analysis. The method is evaluated on the PERCEPT-US dataset without task specific finetuning. It demonstrates superior performance over existing SSL baselines in classifying both perceptual accuracy and within-rhotic tongue shapes.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The integration of an adversarial speaker classifier into a BYOL framework is a practical and highly effective strategy to explicitly handle the high inter speaker variability inherent in pediatric ultrasound data.

    2.The quantitative results are impressive. The proposed BYOL-VS significantly outperforms strong SOTA baselines like VideoMAE and RESA on both downstream classification tasks using a simple k-NN probe.

    3.The qualitative analysis is a strong point, particularly the extraction of mean tongue contours from the learned clusters (Figure 3). This provides excellent clinical interpretability, validating that the representations capture true, recognizable articulatory geometry.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The downstream evaluation relies entirely on k-NN probing of the frozen representations. Providing results with standard linear probing or end-to-end finetuning would give a more comprehensive picture of the embeddings utility for standard clinical pipelines.

    2.The ablation study primarily focuses on the adversarial weight (\lambda_{grl}). Additional ablations on the choice of the spatiotemporal encoder itself (e. g. , comparing ViViT against standard 3D CNNs) would help isolate the specific contributions of the transformer backbone.

    1. While the PERCEPT-US dataset is large for this specific domain, the paper does not extensively detail the demographic distribution of the 70 child speakers. It remains somewhat unclear how well the model might generalize to different age groups (e. g. , adults) or more diverse clinical populations.

    4.As the authors briefly acknowledge in the conclusion, speech production is inherently multimodal. The current framework relies solely on ultrasound video, missing out on synchronized acoustic signals that could provide strong supervisory signals for disambiguating articulatory shapes.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a clinically meaningful problem with a well-designed and effective method, achieving strong and interpretable results. While evaluation and ablation scope are somewhat limited, the core contribution is solid and relevant to MICCAI. With expanded validation, this would be a clear accept.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    As stated in the response, most weakness points proposed by are due to the page limitation of MICCAI. I keep my weak accept decision and hope the authors can release a full version journal points to address the weakness points.



Review #2

  • Please describe the contribution of the paper

    The authors introduce BYOL-VS, a joint-embedding self-supervised learning (SSL) framework applied to lingual ultrasound video for evaluating residual speech sound disorders. The framework combines Bootstrap Your Own Latent (BYOL), a Video Vision Transformer (ViViT) for spatiotemporal encoding, and an adversarial speaker classifier to suppress inter-speaker variability. The method evaluates within-phoneme articulatory structure (rhotic production) without relying on task-specific fine-tuning.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper targets a highly specific and valuable clinical application: evaluating distinct articulatory strategies (e.g., retroflex vs. bunched) that produce perceptually equivalent rhotic sounds, which is notoriously difficult in clinical practice. The use of the PERCEPT-US corpus represents a strong empirical foundation, as it provides a large-scale, expert-labeled resource for within-rhotic child ultrasound analysis. The proposed method achieved high classification accuracy (mean 0.974 for accuracy, 0.961 for rhotic subtype) purely through a k-nearest neighbor probe without task-specific encoder fine-tuning, demonstrating the quality of learned representations.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.From a methodological standpoint, the paper is primarily an application-driven integration of existing techniques. It concatenates BYOL, ViViT, and a standard gradient reversal layer for domain adaptation (speaker adversarial network). It lacks a fundamental algorithmic innovation specifically tailored to the physics or dynamics of tongue ultrasound imaging. 2.The abstract highlights the ability to perform downstream tasks without task-specific fine-tuning. However, the Method section (Section 2) completely omits how the downstream classification is technically executed. The reader only discovers in the Experiments section (Section 3) that a kNN probe is utilized. This disrupts the logical flow and clarity of the proposed framework. 3.The experimental setup reveals critical inconsistencies in hyperparameter tuning across the compared state-of-the-art (SOTA) models. For instance, VideoMAE uses a batch size of 32, while BYOL and BYOL-VS use 16.Learning rates also vary significantly ($1 \times 10^{-4}$ vs $2 \times 10^{-4}$). These discrepancies severely undermine the fairness of the benchmark comparisons. 4.The evaluation is restricted to only two SSL baselines (VideoMAE and RESA). A broader comparison with other contemporary temporal/video SSL frameworks would better substantiate the superiority of BYOL-VS.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    While the application domain is highly relevant and the empirical results on the PERCEPT-US dataset are impressive, the paper currently falls short on methodological novelty and experimental rigor. The core architecture is a direct pipeline of established methods (BYOL + ViViT + GRL) without domain-specific algorithmic modifications. More importantly, the unfair hyperparameter configurations across baselines cast doubt on the comparative superiority claimed by the authors. A clear justification for the baseline settings and a restructuring of the method section to include the kNN downstream protocol are required before acceptance.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The author’s response has largely addressed my concerns, yet further comparisons with state-of-the-art methods are still needed in future work.



Review #3

  • Please describe the contribution of the paper

    The authors propose a self-supervised approach to analyze tongue ultrasound videos to track articulatory movements during speech production, particularly for analyzing rhotic sounds produced by children with residual speech disorder. As I understood, the main contribution is including a speaker adversarial module in a self-supervised method.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well-written and technically solid. The authors clearly explain their methods and the experimental setup appears to be correct. The inclusion of the speaker adversarial approach seems to yield considerable performance gains on the downstream task and also demonstrates how it improves the embedding space.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The medical application is unclear or not properly motivated. I’m not an expert in linguistics, so I can’t comprehend how important rhotic analysis is, but it seems the proposed approach addresses a very specific component of the English language, even though speech development also concerns the correct production of plosives and even vowels. Although I believe the same architecture could be easily extended to other phonemic groups.

    • To me, it is still unclear why speaker-dependent information should be suppressed. If the idea is to build a system that ultimately would support speech therapy, so shouldn’t the model be adapted to the speaker? The ablation study shows that suppressing some of the information improves performance in the downstream tasks. I think some clarification is needed.

    • It’s not explained in the paper in what context the rothic sounds are analyzed. I suppose there is a specific speech elicitation task, and rhotics are extracted from specific words, but the paper doesn’t clarify where. I’m asking because the context of the frames is 12, so I wonder if you actually catch enough articulatory movement in such a short context to properly analyze articulatory problems.

    • Regarding the methods, I wonder why the authors did not try to incorporate other methods such as MoCo, SimCLR, or DINO? Also, I wonder if the speaker adversarial actually improves the performance so much, wouldn’t this also improve the performance of the other methods?

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I think the paper is missing a comparison with more SSL methods to show that including this speaker adversarial approach boosts the performance of the SSL model. There are also some minor details about the data used and how the articulatory movement is actually evaluated that are not properly described in the paper. It is of particular importance because not many people in the community have knowledge of linguistics.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers for their feedback and respond as follows. The addressed points will be clarified in the final version.

Meta C1, Novelty: Our core contribution is meaningful to MICCAI. Although BYOL, ViViT, and GRL are established components, our integration of them is tailored to lingual ultrasound via spatiotemporal modeling, anatomically constrained augmentations, speaker-adversarial regularization, and speaker-disjoint evaluation to assess generalization. The careful design outperforms existing methods and recovers clinically interpretable embeddings. This is the first SSL study for within-rhotic articulatory structure and for using the newly curated PERCEPT-US, the largest American English child speech corpus. Our work contributes to the translational goal of an accessible, take-home speech therapy system.

Meta C2, Benchmark: Based on our initial experiments, GRL is not BYOL-specific: adding GRL to VideoMAE and RESA improves mean ACC by 2–3% and mean MCC by 7–9%. The paper only reported BYOL-VS not VideoMAE/RESA + GRL, because it gives the best results, and its BYOL backbone has already beaten VideoMAE and RESA on our data. We only consider VideoMAE/RESA/BYOL as baselines rather than MoCo/SimCLR/DINO because pair construction for contrastive methods (MoCo/SimCLR) is hard for sensitive articulatory data (papers [4][11]), and RESA is already a SOTA baseline that outperforms MoCo, DINO, SimCLR, and 8 others in clustering and downstream classification (paper [21]).

R1 C1, Probing: We only reported kNN due to page limit. In our initial experiments, linear probing result is directionally consistent but smaller in values, where BYOL-VS is still the best in both tasks (mean ACC = 0.793 & 0.753, mean MCC = 0.657 & 0.558; cf. Table 1). Finetuning is excluded as our goal tests whether SSL yields clinically useful, annotation-free representations.

R1 C2, Ablation: Due to page limit, we only include the key ablation: whether adversarial regularization encourages interpretable embeddings. We use ViViT as vision transformers have shown better performance than CNN-based models in literature (paper [2]).

R1 C3, Scope: PERCEPT-US targets the children population central to RSSD therapy: ages 8–17, sampled from CT/NJ/NY and balanced by sex and age (see paper [5]). It is the largest child speech corpus with ultrasound data for this clinical aim. Thus, findings should generalize to the target children population, while extension to adults requires further validation.

R1 C4, Multimodal: Multimodal alignment with acoustics is the next step. An ultrasound-only model is the necessary first step to confirm that the learned representations encode tongue-shape structure needed for visual biofeedback.

R2 C1, Novelty: See Meta C1. R2 C2, Protocol: We will revise Section 2 to explicitly describe the downstream evaluation protocol.

R2 C3, Fairness: All models used the same 5-fold CV protocol and comprehensive search space (e.g. LR 0.0001–0.0005; BS 2–256). Reported settings are each model’s best validation configuration.

R2 C4, Benchmark: See Meta C2. R3 C1, Motivation: In the U.S., the American English rhotic sound is the highest prevalence therapy target in older children (Farquharson et al, 2026, CLP). This combined with SLP staffing shortages and heavy caseloads (ASHA 2024 Schools Survey) motivates our project.

R3 C2, Speaker: The motivation is to prevent models from exploiting speaker-specific artifacts over articulatory subtypes. The ablation study shows that moderate regularization improves performance and embedding separability, but excessive suppression leads to degradation.

R3 C3, Context: The task is syllable repetition: 15 syllables, repeated 3 times, spanning consonantal r (e.g. ‘ru’), coda/vocalic r (e.g. ‘ear’), syllabic r (e.g. ‘er’), and vowel contexts. Productions are 50–60ms; 12 frames capture suffice rhotic movement while limiting neighboring coarticulatory influence (paper [5]).

R3 C4, Benchmark: See Meta C2.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    I recommend rebuttal, broadly consistent with the reviewer signal (4, 4, 3 — two Weak Accepts and one Weak Reject). Per MICCAI guidelines, the strict “all ≥4” rule for early accept is not satisfied (R3 = 3), so rebuttal is the appropriate choice rather than early accept. The paper proposes BYOL-VS, a self-supervised framework for lingual ultrasound video built from BYOL + ViViT spatiotemporal encoder + a gradient-reversal speaker-adversarial branch, and evaluated on PERCEPT-US for two clinically meaningful tasks: rhotic perceptual accuracy (correct vs incorrect) and within-rhotic subtype (retroflex vs bunched). Speaker-disjoint 5-fold CV is correctly enforced. Paper-level inspection confirms two real strengths that justify reconsideration after rebuttal: (1) The empirical gains are unusually large. (2) The qualitative validation is genuinely interpretable. Spectral clustering of frozen embeddings yields cluster-mean tongue contours that match expert-traced ground truth (Fig. 3 right) with low RMSE. However, the paper has two issues that the rebuttal must address before acceptance: (a) Methodological novelty is integrative. BYOL + ViViT + GRL are all standard, off-the-shelf components; there is no domain-specific algorithmic modification. (b) Only VideoMAE and RESA are compared as video-SSL baselines. There is no MoCo/SimCLR/DINO comparison and no apples-to-apples test where the speaker-adversarial GRL branch is added to a baseline.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    I recommend accept. In essence, BYOL-VS (BYOL + ViViT + speaker-adversarial GRL) achieves unusually strong empirical results on lingual ultrasound video for residual speech sound disorder, with interpretable cluster-mean tongue contours that match expert tracings and a complete ablation study supporting the design, justifying acceptance over R3’s lowest-confidence concern on novelty/baseline breadth,



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Although the technical contribution of the paper is limited, it targets a clinically important problem of articulatory structure learning from lingual ultrasound videos for children with speech sound disorders. The paper presents a BYOL style SSL approach, making it label efficient and generalizable. The paper should be of interest to the MICCAI community.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    I recommend Accept. The paper studies a narrow but well-defined clinical problem: learning ultrasound video representations for rhotic production in children with residual speech sound disorder. The method is not a new SSL objective. It combines BYOL, a video transformer encoder, and a speaker-adversarial branch. The value of the paper is that this combination matches a real structure in the data: tongue motion should be retained, while speaker-specific ultrasound variation should be reduced.

    The empirical results support this design. BYOL-VS performs well with a frozen encoder and a simple kNN probe, which suggests that the representation itself carries useful articulatory information. The cluster analysis is also helpful. The recovered mean tongue contours are not just performance numbers; they show that the embedding separates recognizable rhotic tongue shapes. This makes the result easier to interpret clinically.

    The original concerns were fair. The method has limited algorithmic novelty, the first version did not describe the downstream kNN protocol clearly enough, and the baseline comparison needed more explanation because the training settings differed across methods. The rebuttal addresses these points sufficiently for me. Broader SSL baselines and linear-probe or finetuning results would still make the paper stronger, but they are not necessary for acceptance here.

    I therefore support acceptance. The paper is application-driven, but the application is specific, the nuisance factor is handled in a direct way, and the learned representations show both quantitative and anatomical evidence of usefulness.



back to top