PathoFusion: An Open-Source AI Framework for Recognition of Pathomorphological Features and Mapping of Immunohistochemical Data

Simple Summary We present an open-source AI framework for marking, training, and automated recognition of pathological features in whole-slide scans of diagnostic tissue sections. The integrated system permits high-resolution qualitative as well as quantitative morphological analyses of entire histological slides and harbors significant potential to facilitate the microscopic analysis of complex pathomorphological problems and the simultaneous mapping of immunohistochemical markers in routine slide diagnostics. Abstract We have developed a platform, termed PathoFusion, which is an integrated system for marking, training, and recognition of pathological features in whole-slide tissue sections. The platform uses a bifocal convolutional neural network (BCNN) which is designed to simultaneously capture both index and contextual feature information from shorter and longer image tiles, respectively. This is analogous to how a microscopist in pathology works, identifying a cancerous morphological feature in the tissue context using first a narrow and then a wider focus, hence bifocal. Adjacent tissue sections obtained from glioblastoma cases were processed for hematoxylin and eosin (H&E) and immunohistochemical (CD276) staining. Image tiles cropped from the digitized images based on markings made by a consultant neuropathologist were used to train the BCNN. PathoFusion demonstrated its ability to recognize malignant neuropathological features autonomously and map immunohistochemical data simultaneously. Our experiments show that PathoFusion achieved areas under the curve (AUCs) of 0.985 ± 0.011 and 0.988 ± 0.001 in patch-level recognition of six typical pathomorphological features and detection of associated immunoreactivity, respectively. On this basis, the system further correlated CD276 immunoreactivity to abnormal tumor vasculature. Corresponding feature distributions and overlaps were visualized by heatmaps, permitting high-resolution qualitative as well as quantitative morphological analyses for entire histological slides. Recognition of more user-defined pathomorphological features can be added to the system and included in future tissue analyses. Integration of PathoFusion with the day-to-day service workflow of a (neuro)pathology department is a goal. The software code for PathoFusion is made publicly available.

Neural network design. The bifocal convolutional neural network (BCNN) introduced here has two input paths that accept one narrow-focus image tile ( , ∈ ℝ ) and another paired wide-focus image tile ( ) simultaneously. The narrowfocus patch contains a view limited to the actual structure of interest, whereas the widefocus patch not only contains the relevant structure of interest as image content, but additional information on the tissue context of the structure of interest is included as well.
For a given narrow-focus input , where H and W are spatial dimensions, C and N are the input and output channel dimensions. The paired wide-focus input has the same centroid as , and with a spatial dimension of and * and * , where is the integer factor to explore. The output of the network is defined as the following: where, is the Global Average Pooling layer; and ′ are the feature maps that simultaneously generated by the two convolutional branches and defined as:

•
(2) In other words, is a product after applying ReLU activation function on , which is the final convolutional output of and in each network branch (output of last convolutional layer ); (also denoted as ) is a stack of convolutional output for each kernel ( N); represents the kernel weights, is the kernel size, and denotes biases.
Specifically, the neural network was implemented in the following way: The BCNN consists of two convolutional subnets, one feature concatenation module, and a classification layer. Each subnet is 26-layers deep, has a structure similar to ResNet-26, which consists of a convolutional layer with an output feature map size of 16, three convolutional blocks with the output feature map size ranging from 64 to 256, and one average pooling layer, but without skip connections. Each convolutional block has 8 consecutive convolution/batch normalization pairs. The first convolutional layer has a stride of 2, which downsamples the input image and greatly reduces the amount of computation for the subsequent layers. To make the network more computationally efficient, each convolutional block also introduced a stride of 2 in its first layer, which further reduces the image size from one block to another. The ReLU activation function is applied to each convolutional output for introducing non-linearities to the network. The Softmax function was not explicitly used in the last layer of the network since it is already incorporated in the PyTorch cross-entropy loss function which we adopted for this study. The network was trained to read the paired bifocal image tiles simultaneously. Figure S1. CD276 immunopositivity of glioblastoma vasculature. Adjacent paraffin sections stained for hematoxylin and eosin (left column) and immunohistochemically labeled for CD276 (right column), respectively. Corresponding tissue areas illustrating CD276 immunoreactivity of microvascular proliferation and specifically of abnormal endothelial cells are shown at increasing magnification. Figure S2. Pathology image labeling module. A website was designed and set up on the institutional intranet for marking pathomorphological features in scanned histopathological slides. The marking was performed by a consultant neuropathologist. A neuropathologist first defines the relevant feature categories for the regions of interest (ROIs) in the backend system, e.g., six types of H&E morphological features in this piece of work. Via the website, the consultant is able to zoom in on the whole-slide H&E and IHC images and mark the different ROIs with color dots. Different colored dots represent the categories that have been predefined in the backend system. The gigabytesized image scans are precached on a local computer and the website redirects its image content to the local cache in order to speed up image loading. Therefore, only labeling coordinates need to be uploaded. (1-6) P he : Area percentage of detected palisading necrosis (1), microvascular proliferation (2), histologically normalappearing blood vessels (3), geographic necrosis (4), brain tissue (5), and tumor background (extensive diffuse infiltration of brain tissue by glioma cells, 6), in the respective histological sections.
Video S1. Video demonstration of the AI framework for detection of cancerous features in while-slide tissue sections.