Multimodal Deep Learning with Routine Clinical Data for Recurrence Risk Stratification in HR/HER2 Early Breast Cancer.

Hormone receptor-positive/human epidermal growth factor receptor 2-negative (HR/HER2) early breast cancer (EBC) patients face long-term recurrence risk despite standard treatment. Current prognostic tools relying on clinicopathological factors or multigene assays have limited accuracy or accessibility. In this study, we developed a multimodal recurrence risk prediction (MRRP) model integrating routinely available clinical data, including whole-slide images (WISs), ultrasound (US) imaging and diagnostic reports, and structured clinical parameters. The MRRP model employs a hierarchical transformer-based fusion framework with innovative intra- and intermodality cross-attention mechanisms to dynamically integrate diverse feature representations. Using a well-curated cohort of 768 HR/HER2 EBC patients with long-term follow-up, MRRP demonstrated superior prognostic performance (C-index = 0.840) compared to single-modality models, with robust time-dependent AUCs exceeding 0.85 at 3, 5, and 7 years. Ablation studies highlighted the central role of pathology features and the complementary value of US and clinical data. We further validated the optimal query selection strategies and evaluated different pretrained encoders, revealing complex modality interactions. To address real-world challenges of missing modality data, a learnable compensation mechanism was implemented, improving model robustness. Our study provides a clinically practical, AI-driven tool for precise risk stratification in HR/HER2 EBC patients, facilitating individualized treatment and surveillance decisions without reliance on costly multi-omics data.