Robust analysis of the tumor spectrum in a preclinical model of breast cancer reveals stable subtypes with distinct growth patterns.

[BACKGROUND AND OBJECTIVE] The tumor microenvironment plays a crucial role in influencing tumor progression and responses to therapy, shaped by both inherent tumor features and external factors. We aim to develop a pipeline that computes tumor subtypes and growth patterns based on nuclear shape, spatial arrangement, and protein measurements in preclinical models. Preclinical models enable the investigation of exogenous perturbations on tumor development. In this context, accurately segmenting and classifying nuclei is vital. The main challenges include: (i) the presence of densely packed nuclei, and (ii) the need to characterize tumor diversity across a large set of mouse-derived tumor samples.

[METHOD] The computational pipeline requires methods for nuclear segmentation and tumor heterogeneity characterization. For robust segmentation of nuclei, we developed LoG-based Saliency for Guided Encoding with Convolutional Block Attention Module (LoGSAGE-CBAM), a dual-encoder segmentation model that combines a Swin Transformer with a saliency encoder based on Laplacian of Gaussian (LoG) response. The outputs of these encoders are then fused through a CBAM module, and the model is trained with a curvature-aware loss function. Subsequently, the immune cells are classified, and their locations are recorded. To capture the tumor spectrum, cellular responses and localizations are binarized, and tumor subtypes are identified, which are then associated with preclinical variables using Cox regression.

[RESULTS] The integrated computational pipeline identified four stable tumor subtypes in 184 tumor-derived mice using computed indices from 2168,733 nuclei. At the same time, the LoGSAGE-CBAM achieved a segmentation performance with Dice 95.5 and RCE: 86.6. One of the subtypes is enriched in K14+ tumors and CD8+ lymphocytes and is associated with longer latency.

[CONCLUSION] The proposed computational pipeline can provide both novel insights and automation for biomarker discovery in preclinical studies and pharmaceutical research.