H&E Meets AI: Driving Quantitative Insights for Precision Oncology

The microscopic examination of hematoxylin and eosin (H&E)-stained tissue has been the cornerstone of cancer diagnosis for over a century, relying on the trained eye of pathologists to detect cellular abnormalities that signal malignancy. In recent years, however, cancer care has grown increasingly complex. Especially with the advent and advancement of targeted therapeutics, there remains a need for biomarkers that would best match patients with available treatments. Quantitative insights are essential to maximize the precision of these biomarkers.

At PathAI, our mission is to improve patient outcomes through AI-powered histology. To this end, we’ve developed solutions that allow investigators to gain comprehensive, quantitative insights from whole slide images of routine, H&E-stained cancer tissue.

PathExplore

While cancer composition and morphology can be appreciated at a qualitative level from visual review of an H&E-stained slide, these important aspects of cancer biology cannot be precisely measured under the microscope. Developed by PathAI, PathExplore* is an AI-powered digital pathology tool designed to quantify histopathological features from H&E-stained whole slide images (Figure 1). This suite of models provides detailed cell- and tissue-level insights, including the comprehensive computation of tumor microenvironment (TME) composition:

• Pixel-level prediction and classification of relevant tissue regions (e.g., tumor epithelium, cancer-associated stroma) and cell types (e.g., cancer cells, fibroblasts, lymphocytes) allow the quantification of the TME composition at a highly granular level across the entirety of a whole slide image [1,2].

• Nuclear morphology (size, shape, solidity, stain intensity) is further captured for each predicted cell type.

• Classification of specimens into the three main immune phenotypes (desert, excluded, inflamed) is performed by our PathExplore-IOP* solution [3].

• Integration of our iQMAI technology into PathExplore-Fibrosis* further allows the identification of collagen fibers and quantitative analysis of fiber morphology [4].

These insights are generated through over 600 human interpretable features (HIFs), which include relative tissue areas, relative cell counts, cell densities, and spatial relationships for cells in relation to other cells and tissue regions.

Figure 1. PathExplore overview

The quantitative insights gained from the use of AI-powered pathology models can yield key insights about how patients respond to therapy. For example, in a post hoc analysis of patients with head and neck squamous cell carcinoma enrolled in the innovaTV (part C) clinical trial (NCT03485209) [5], PathExplore features were compared in patients classified as responders and non-responders to tisotumab vedotin (TV), an antibody-drug conjugate targeting Tissue Factor [6]. Patients responding to TV had higher fractions of macrophages, plasma cells, and fibroblasts within 40 μm of cancer cells, as well as an elevated tumor-to-stroma ratio (Figure 2).

Figure 2. Association of PathExplore features with response to tisotumab vedotin in innovaTV (part C). Figure modified from [6].

This type of analysis can be extended to understand how the integration of AI-derived TME features can associate with or predict response to therapy. In work conducted by investigators at SUNY Upstate Medical Center [7], unsupervised clustering of PathExplore HIFs in a cohort of non-small cell lung cancer patients treated with immunotherapy revealed the presence of distinct TME phenotype clusters, which corresponded to response. Multivariate Cox analysis confirmed that the PathExplore-derived immune cold subtype was an independent predictor of overall survival in NSCLC (HR=2.12, p<0.001).

A recently reported retrospective analysis of the NEPTUNE clinical trial of durvalumab and tremelimumab compared to chemotherapy in metastatic non-small cell lung cancer [8] revealed a similar theme: AI-derived features derived from H&E slides were associated with and, in some cases, predictive of outcomes [9]. Specifically, elevated density of immune cells was predictive of response to the durvalimumab/tremelimumab combination (FDR p=0.04). Importantly, these AI-derived features improved the ability to stratify outcome after durvalimumab/tremelimumab treatment compared to clinical data, tumor mutation burden, and PD-L1 status.

Finally, retrospective analysis of the OAK (NCT02008227; 10) and IMpower150 (NCT02366143; 11) clinical trials of atezolizumab in non-small cell lung cancer using AI-powered TME analysis also allowed the identification of prognostic cell proportion signatures [12]. One of the identified signatures, termed CP1, was characterized by enrichment of plasma cells and lymphocytes in regions of cancer stroma. Patients with enrichment of this signature showed improved overall survival with atezolizumab, even in patients lacking detectable PD-L1 (Figure 3).

Figure 3. Association of an H&E-derived cell proportion signature with response to atezolizumab in OAK and IMpower150. Figure modified from [12].

Thus, the quantification of components of the TME that AI-powered pathology methods provide has the potential to identify key parameters associated with response to various treatment modalities, including IO agents and ADCs. By extracting standardized and reproducible data from routine H&E-stained specimens, PathExplore provides a more granular understanding of tumor biology, enabling investigators to potentially link tumor microenvironment composition with outcomes.

Additive Multiple Instance Learning (aMIL)

Beyond the comprehensive TME quantification allowed by PathExplore, additional visual cues may be inherent to H&E-stained cancer specimens that, when seen by a predictive model, can be integrated by the model in a way that further improves predictive power. This is the power of our aMIL framework [1,13].

Using visual information aggregated across individual patches from a whole slide image, aMIL workflows generate slide-level classification predictions. To date, we have successfully used this approach for the slide-level prediction of luminal status in urothelial carcinoma [14] and expression of the TGFꞵ-CAF gene expression signature in multiple cancer types [1]. Most recently, we used aMIL to predict microsatellite instability status in gastric cancer [15]. Using an aMIL classifier and PLUTO embeddings, this model predicted MSI status with a mean AUROC of 0.86 across five folds (Figure 4).

A major benefit to aMIL predictive approaches is the ability to integrate model predictions with additional models to provide biologic interpretability to model predictions. Through integration of our aMIL MSI prediction model with PathExplore, we demonstrated that, in slides predicted to be MSI, increased fractions of lymphocytes were detected in regions with excitatory model attention (Figure 4).

Figure 4. Prediction of MSI status from H&E-stained slides in gastric cancer, and association of lymphocyte proportion with model attention. Figure modified from [15].

Summary and a look forward

In conclusion, the ability to quantify features inherent to H&E-stained cancer specimens represents a pivotal shift in the landscape of precision oncology. At PathAI, we utilize two complementary approaches for the AI-based study of H&E-stained cancer specimens: PathExplore enables a detailed, human-interpretable quantification of the tumor microenvironment by analyzing cellular and tissue-level features across entire whole slide images, while aMIL harnesses aggregate visual information to provide slide-level predictive modeling, such as determining MSI status or gene expression signatures, while maintaining biological interpretability through its synergy with PathExplore.

Figure 5. Potential utility for quantitative, AI-based analysis of H&E-stained cancer specimens.

By transforming routine H&E-stained specimens into a rich source of standardized and reproducible data, these AI-powered tools empower investigators to look beyond simple tissue morphology of cancer specimens. This capability allows for innovative exploratory research to study mechanisms of action and the association between H&E features and outcome, as well as the development of biomarkers and companion diagnostics.

Looking ahead, quantitative H&E has the potential to become a foundational layer of precision medicine. As these tools continue to mature, routine diagnostic slides could help bring insights regarding therapy selection closer to the point of diagnosis, enabling a future in which a patient’s first tissue image is not only diagnostic, but directly informative for the treatment decisions that follow.

*PathExplore, PathExplore-Fibrosis, and PathExplore-IOP are For Research Use Only. Not for use in diagnostic procedures.

References:

1. Markey M and Kim J, et al. Spatial Mapping of Gene Signatures in Hematoxylin and Eosin-Stained Images: A Proof of Concept for Interpretable Predictions Using Additive Multiple Instance Learning. Mod Pathol. 2025; 38(8):100772.
2. Abel J, et al. AI powered quantification of nuclear morphology in cancers enables prediction of genome instability and prognosis. NPJ Precis Oncol. 2024; 8(1):134.
3. Le N, et al. Correlation of immune phenotypes derived from H&E-stained whole slide images with prognosis and response to checkpoint inhibitors in NSCLC. J Clin Oncol 2024; 42 (16 suppl):8539.
4. Nguyen TH and Zhang J, et al. Quantification of collagen and associated features from H&E-stained whole slide pathology images across cancer types using a physics-based deep learning model. bioRxiv 2025.03.17.643273.
5. Sun L, et al. Tisotumab vedotin in head and neck squamous cell carcinoma: Updated analysis from innovaTV 207 Part C. J Clin Oncol. 2024; 42:S6012.
6. Bieda MC, et al. 53 Assessing the mechanism of action (MOA) of tisotumab vedotin (TV) in head and neck squamous cell carcinoma (HNSCC) with digital pathology analysis of H&E images from innovaTV207 part C. J Immunother Cancer. 2025; 13(Suppl 2):A61.
7. Rad MR, et al. Validation of HistoTME-predicted immune subtypes and immunotherapy outcomes using human interpretable features (HIFs) from H&E images in non-small cell lung cancer. J Clin Oncol. 2025; 43: S8583
8. De Castro G, et al. NEPTUNE: Phase 3 Study of First-Line Durvalumab Plus Tremelimumab in Patients With Metastatic NSCLC. J Thorac Oncol. 2023; 18(1):106-119.
9. Parmar C, et al. Abstract 6725: AI-derived H&E biomarkers predict differential survival under immunotherapy vs chemotherapy in phase-3 metastatic NSCLC study NEPTUNE. Cancer Res. 2026; 86:(7_Supplement): 6725.
10. Rittmeyer A, Barlesi F, Waterkamp D, et al. Atezolizumab versus docetaxel in patients with previously treated non-small-cell lung cancer (OAK): a phase 3, open-label, multicentre randomised controlled trial. Lancet. 2017;389(10066):255-265.
11. Socinski MA, Jotte RM, Cappuzzo F, et al. Atezolizumab for First-Line Treatment of Metastatic Nonsquamous NSCLC. N Engl J Med. 2018;378(24):2280-2293.
12. Qamra A, et al. Abstract 5705: Digital pathology based prognostic & predictive biomarkers in metastatic non-small cell lung cancer. Cancer Res. 2023; 83(7_Supplement): 5705.
13. Javed SA, et al. Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathology. 2022; arXiv:2206.01794
14. Kirov S, et al. Abstract B016: AI analysis of histological images accurately identifies luminal subtype urothelial carcinomas characterized by high PPARG expression. Mol Cancer Ther. 2023; 22(12_Supplement): B016.
15. Nofallah S, et al. Abstract 73: Accurate prediction of microsatellite instability-high gastric cancer from H&E-stained whole slide images. Cancer Res. 2026; 86(7_Supplement):73.