The advancements in data acquisition, storage, and processing techniques have resulted in the rapid growth of heterogeneous medical data. Integrating radiological scans, histopathology images, and molecular information with clinical data is essential for developing a holistic understanding of the disease and optimizing treatment. The need for integrating data from multiple sources is further pronounced in complex diseases such as cancer for enabling precision medicine and personalized treatments. This work proposes Multimodal Integration of Oncology Data System (MINDS)—a flexible, scalable, and cost-effective metadata framework for efficiently fusing disparate data from public sources such as the Cancer Research Data Commons (CRDC) into an interconnected, patient-centric framework. MINDS consolidates over 41,000 cases from across repositories while achieving a high compression ratio relative to the 3.78 PB source data size. It offers sub-5-s query response times for interactive exploration. MINDS offers an interface for exploring relationships across data types and building cohorts for developing large-scale multimodal machine learning models. By harmonizing multimodal data, MINDS aims to potentially empower researchers with greater analytical ability to uncover diagnostic and prognostic insights and enable evidence-based personalized care. MINDS tracks granular end-to-end data provenance, ensuring reproducibility and transparency. The cloud-native architecture of MINDS can handle exponential data growth in a secure, cost-optimized manner while ensuring substantial storage optimization, replication avoidance, and dynamic access capabilities. Auto-scaling, access controls, and other mechanisms guarantee pipelines’ scalability and security. MINDS overcomes the limitations of existing biomedical data silos via an interoperable metadata-driven approach that represents a pivotal step toward the future of oncology data integration.

Harmonized ONcologY Biomedical Embedding Encoder (HONeYBEE) is an open-source framework that integrates multimodal biomedical data for oncology applications. It processes clinical data (structured and unstructured), whole-slide images, radiology scans, and molecular profiles to generate unified patient-level embeddings using domain-specific foundation models and fusion strategies. These embeddings enable survival prediction, cancer-type classification, patient similarity retrieval, and cohort clustering. Evaluated on 11,400+ patients across 33 cancer types from The Cancer Genome Atlas (TCGA), clinical embeddings showed the strongest single-modality performance with 98.5% classification accuracy and 96.4% precision@10 in patient retrieval. They also achieved the highest survival prediction concordance indices across most cancer types. Multimodal fusion provided complementary benefits for specific cancers, improving overall survival prediction beyond clinical features alone. Comparative evaluation of four large language models revealed that general-purpose models like Qwen3 outperformed specialized medical models for clinical text representation, though task-specific fine-tuning improved performance on heterogeneous data such as pathology reports.

Prognostic markers such as overall survival (OS) and tertiary lymphoid structure (TLS) ratios, alongside diagnostic signatures like primary cancer-type classification, provide critical information for treatment selection, risk stratification, and longitudinal care planning across the oncology continuum. However, extracting these signals solely from sparse, high-dimensional multi-omics data remains a major challenge due to heterogeneity and frequent missingness in patient profiles. To address this challenge, we present SeNMo, a self-normalizing deep neural network trained on five heterogeneous omics layers—gene expression, DNA methylation, miRNA abundance, somatic mutations, and protein expression—along with the clinical variables, that learns a unified representation robust to missing modalities. Trained on more than 10,000 patient profiles across 32 tumor types from The Cancer Genome Atlas (TCGA), SeNMo provides a baseline that can be readily fine-tuned for diverse downstream tasks. On a held-out TCGA test set, the model achieved a concordance index of 0.758 for OS prediction, while external evaluation yielded 0.73 on the CPTAC lung squamous cell carcinoma cohort and 0.66 on an independent 108-patient Moffitt Cancer Center cohort. Furthermore, on Moffitt’s cohort, baseline SeNMo fine-tuned for TLS ratio prediction aligned with expert annotations (p < 0.05) and sharply separated high- versus low-TLS groups, reflecting distinct survival outcomes. Without altering the backbone, a single linear head classified primary cancer type with 99.8% accuracy across the 33 classes. By unifying diagnostic and prognostic predictions in a modality-robust architecture, SeNMo demonstrated strong performance across multiple clinically relevant tasks, including survival estimation, cancer classification, and TLS ratio prediction, highlighting its translational potential for multi-omics oncology applications.

A bstract: Surgical pathology reports provide essential diagnostic information critical for cancer staging, treatment planning, and cancer registry documentation. However, their writing styles and formats vary widely, reflecting each pathologist’s stylistic choices, institutional norms, and largely inherited practices from residency training. When performing large-scale data analysis, this unstructured nature and variability across tumor types and institutions pose significant hurdles for automated data extraction. To overcome these challenges, we present a consensus-driven, reasoning-based framework that adapts multiple locally deployed large language models (LLMs) to extract both standard diagnostic variables (site, laterality, histology, stage, grade, and behavior) and organ-specific biomarkers. Each LLM generates structured outputs, accompanied by justifications, which are subsequently evaluated for accuracy and coherence by three separate reasoning models (DeepSeek-R1-large, Qwen3-32B, and QWQ-32B). Final consensus values are determined through aggregation. Board-certified pathologists conducted expert validation. This framework was applied to over 6,100 pathology reports from The Cancer Genome Atlas (TCGA) spanning 10 organ systems and 510 reports from Moffitt Cancer Center. For the TCGA dataset, automated evaluation demonstrated mean accuracy of 84.9% ± 7.3%, with histology (89.0%), site (88.3%), and behavior (87%) showing the highest extraction accuracy averaged across all models. Expert review of randomly selected 138 reports confirmed high agreement for behavior (100.0%), histology (99%), grade (97%), and site (95%) in the TCGA dataset, with slightly lower performance for stage (88%) and laterality (87%). In Moffitt Cancer Center reports (brain, breast, and lung), accuracy remained high (88.2% ± 7.2%), with behavior (99%), histology and laterality (96%), grade (93%), and site (91%) achieving strong agreement. Biomarker extraction achieved 70.6% ± 8.1% overall accuracy, with TP53 (85%) on brain tumor, Ki-67 (68%) on breast cancer, and ROS1 (82%) on lung cancer showing highest accuracy. Inter-evaluator agreement analysis revealed high concordance (correlations > 0.89) across the three evaluation models. Statistical analyses revealed significant main effects of model type (F=1716.82, p<0.001), variable (F=3236.68, p<0.001), and organ system (F=1946.43, p<0.001), as well as model × variable × organ interactions (F=24.74, p<0.001), emphasizing the role of clinical context in model performance. These results highlight the potential of stratified, multi-organ evaluation frameworks with multi-evaluator consensus in LLM benchmarking for clinical applications. Overall, this consensus-based approach demonstrates that locally deployed LLMs can provide a transparent, accurate, and auditable solution for integration into real-world pathology workflows such as synoptic reporting and cancer registry abstraction.