Multimodal Artificial Intelligence for Histopathology & Genomics Fusion

Abstract

The field of medical diagnostics has witnessed a transformative convergence of artificial intelligence (AI) and healthcare data, offering promising avenues for enhancing patient care and disease comprehension. However, this integration of multimodal data, specifically histopathology images and genetic sequencing data, presents unique challenges due to modality disparities and the need for scalable computational solutions. This thesis endeavors to bridge this gap by introducing a comprehensive multimodal framework that harmonizes these indispensable data sources, with a focus on its potential in cancer diagnosis, particularly by leveraging immunogenomic data. The thesis addresses the scarcity of multimodal solutions, primarily centered around unimodal data solutions, thus limiting the realization of the rich insights that can be derived from integrating images and genomic data.

The first stage of this research introduces a novel patching algorithm called "Sequential Patching Lattice for Image Classification and Enquiry" (SPLICE). With the ever-growing scale of whole slide images (WSIs), computerized analysis becomes essential. SPLICE's approach of sequentially analyzing and selecting representative samples from WSIs results in a compact "collage" representation, enabling efficient processing and preserving diagnostically relevant regions. SPLICE addresses the dearth of holistic representation learning solutions for high-resolution WSIs and emphasizes the importance of unsupervised approaches in a landscape dominated by labeled data.

In the second stage, the thesis presents MarbliX "Multimodal Association and Retrieval with Binary Latent Indexed matriX," an innovative multimodal framework that integrates histopathology images with immunogenomic sequencing data, encapsulating them into a concise binary barcode, referred to as "monogram." This binary representation facilitates the establishment of a comprehensive archive, enabling clinicians and pathologists to retrieve and match similar cases. MarbliX empowers healthcare professionals with in-depth insights, leading to more precise diagnoses, reduced variability, and expanded personalized treatment options, particularly in the context of cancer.

This thesis stands at the intersection of AI, medical diagnostics, and multimodal data integration. It addresses the critical need for comprehensive solutions in this evolving field, offering a two-fold approach through SPLICE and MarbliX to unlock the potential of combining diverse data sources. By merging macroscopic tissue information with microscopic (molecular) insights, it aims to advance our understanding of complex diseases and elevate the standard of patient care in the ever-evolving landscape of healthcare, with a specific emphasis on harnessing the power of immunogenomic data in the diagnostic process.