In vivo tumor immune microenvironment phenotypes correlate with inflammation and vasculature to predict immunotherapy response

Sahu, Aditi; Kose, Kivanc; Kraehenbuehl, Lukas; Byers, Candice; Holland, Aliya; Tembo, Teguru; Santella, Anthony; Alfonso, Anabel; Li, Madison; Cordova, Miguel; Gill, Melissa; Fox, Christi; Gonzalez, Salvador; Kumar, Piyush; Wang, Amber Weiching; Kurtansky, Nicholas; Chandrani, Pratik; Yin, Shen; Mehta, Paras; Navarrete-Dechent, Cristian; Peterson, Gary; King, Kimeil; Dusza, Stephen; Yang, Ning; Liu, Shuaitong; Phillips, William; Guitera, Pascale; Rossi, Anthony; Halpern, Allan; Deng, Liang; Pulitzer, Melissa; Marghoob, Ashfaq; Chen, Chih-Shan Jason; Merghoub, Taha; Rajadhyaksha, Milind

doi:10.1038/s41467-022-32738-7

Download PDF

Article
Open access
Published: 09 September 2022

In vivo tumor immune microenvironment phenotypes correlate with inflammation and vasculature to predict immunotherapy response

Nature Communications volume 13, Article number: 5312 (2022) Cite this article

8546 Accesses
16 Citations
25 Altmetric
Metrics details

Subjects

Abstract

Response to immunotherapies can be variable and unpredictable. Pathology-based phenotyping of tumors into ‘hot’ and ‘cold’ is static, relying solely on T-cell infiltration in single-time single-site biopsies, resulting in suboptimal treatment response prediction. Dynamic vascular events (tumor angiogenesis, leukocyte trafficking) within tumor immune microenvironment (TiME) also influence anti-tumor immunity and treatment response. Here, we report dynamic cellular-level TiME phenotyping in vivo that combines inflammation profiles with vascular features through non-invasive reflectance confocal microscopic imaging. In skin cancer patients, we demonstrate three main TiME phenotypes that correlate with gene and protein expression, and response to toll-like receptor agonist immune-therapy. Notably, phenotypes with high inflammation associate with immunostimulatory signatures and those with high vasculature with angiogenic and endothelial anergy signatures. Moreover, phenotypes with high inflammation and low vasculature demonstrate the best treatment response. This non-invasive in vivo phenotyping approach integrating dynamic vasculature with inflammation serves as a reliable predictor of response to topical immune-therapy in patients.

A spatial architecture-embedding HLA signature to predict clinical response to immunotherapy in renal cell carcinoma

Article 21 May 2024

Best practices for single-cell analysis across modalities

Article 31 March 2023

Cell-intrinsic and microenvironmental determinants of metastatic colonization

Article 07 May 2024

Introduction

Immunotherapy, especially immune checkpoint blockade therapy, has revolutionized cancer management by providing durable responses in several cancers^1,2,3. However, only a subset of patients derives long-term benefit, highlighting a clinical need to develop effective biomarkers for patient stratification^4,5,6. Phenotyping of tumors into ‘hot’, ‘cold’ or ‘altered’ based on infiltration of CD3⁺ and CD8⁺ T-cells at tumor center and margin (Immunoscore)⁷, PD-1/PD-L1 expression, and tumor mutation burden are important determinants of response to immunotherapy in solid cancers^8,9. Although hot versus cold tumor phenotypes have shown association with treatment response and overall cancer outcomes, specific immune cell subsets modify this association, including regulatory T cells (Tregs), myeloid-derived suppressor cells (MDSCs), and tumor-associated macrophages (TAMs)¹⁰. Importantly, not all hot phenotypes respond to treatment, suggesting that immune cell infiltration is important, but not always sufficient, for inducing potent anti-tumor immunity to eradicate tumors^11,12. Evidently, tumors utilize additional mechanisms for evading immune response while establishing an immune-suppressive microenvironment, complicating patient stratification strategies because of dynamic tumor host/immune crosstalk and tumor-intrinsic biology^8,13,14.

The tumor vasculature is a key component of the microenvironment that can influence tumor behavior and treatment response. Tumor vasculature plays a central role in T-cell trafficking via regulation of endothelial adhesion molecule expression and creation of immunosuppressive microenvironments¹⁵. Angiogenesis promotes immune evasion through induction of a highly immunosuppressive TiME by inhibiting dendritic cell (DC) maturation, T-cell development and function, and most importantly, limiting access of effector immune cells to tumors by modulating leukocyte trafficking¹⁶. In addition, tumor vasculature can display decreased expression of adhesion molecules, and demonstrate non-responsiveness to inflammatory cytokines through development of vascular endothelial anergy. By downregulating trafficking of effector cells and upregulating trafficking of regulatory immune cells, endothelial anergy contributes to ineffective anti-tumor immune responses and immune evasion^17,18,19.

To address the complex and highly interdependent vascular-inflammation axis within the TiME, in vivo phenotyping that integrates dynamic vascular features with inflammation may be more advantageous as compared to ex vivo pathological phenotyping based mainly on infiltrating T-cells. High-resolution non-invasive in vivo imaging is fundamental to this integrative phenotyping, since static ex vivo analyses on patient tissue are limited in recapitulating dynamic vascular events and the continuous, evolving cellular-level crosstalk between the immune system and tumor^6,20. Reflectance confocal microscopy (RCM) is a high-speed (pixel times ~0.10 µs, frame rates 10–30 per second) cellular-level label-free imaging approach based on backscattered light and endogenous tissue contrast, capable of capturing dynamic phenomena inside patients in real-time^21,22. Further, large image mosaics (64 mm² in 50 s) imaged to a depth of ~0.25 mm can facilitate spatial resolution of features. RCM is routinely used for real-time skin cancer diagnosis and management at the bedside^23,24. Although a few studies have reported RCM imaging of vessels and leukocyte trafficking in humans^25,26,27, analysis of individual RCM TiME features, the vascular-inflammation axis and its role in treatment response prediction have not been studied.

In this work, we report TiME phenotypes detected in vivo in basal cell carcinoma (BCC) and melanoma. We define TiME phenotypes with key inflammation and vasculature features such as vessel diameter, vessel density, leukocyte trafficking, intratumoral inflammation, peritumoral inflammation and perivascular inflammation. We demonstrate feasibility of automated quantification of immune and vascular features. Further, we also perform in-depth histopathological validation and molecular correlation of TiME features and phenotypes with gene and protein expression. Finally, we investigate the utility of immune and vascular features and TiME phenotypes in predicting response to a toll-like receptor agonist (TLRA) topical immune-therapy in a prospective pilot study (Fig. 1).

**Fig. 1: TiME phenotypes were derived from RCM imaging and correlated with underlying biology and treatment response to topical toll-like receptor agonist (TLRA) imiquimod.**

Results

Immune and vascular features on RCM correlate with histopathology and demonstrate unique TiME phenotypes

TiME phenotyping was investigated by integrating inflammation and vasculature features. First, agreement between RCM and histopathological TiME features was explored. Manual evaluation of RCM TiME features (Fig. S1a–c, Supplementary Movies 1–3) by two independent readers resulted in substantial to almost perfect agreement (k = 0.62–1.0) for most RCM features. No TiME features were observed in the normal perilesional area (Fig. S1d); thus analysis was restricted to tumor lesion. RCM manual evaluation correlated well with corresponding histopathological features evaluated by a board-certified dermatopathologist (Table 1), demonstrating good to very good agreement (AC₁: 0.74–1.0). Unsupervised multivariate clustering of TiME features in BCC patients revealed three main phenotypes with variable prevalence of inflammation (Inflam) and vascular (Vasc) features (Fig. 2a). This feature heterogeneity was clearly apparent in the varying degrees of inflammation, trafficking and vessels in corresponding RCM and H&E images of these phenotypes. No correlation of TiME phenotypes with clinical factors such as BCC subtype, gender, age, sun exposure and ulceration were observed (Fig. 2a). Phenotypic classes were assigned based on individual TiME measurement contributions to the first two principal components, which together comprise 77% of the total variability (Fig. 2b). Number of vessels, trafficking and dilated vessels were major contributors to the variability captured by the PC1 or dimension 1 (Fig. 2c) whereas perivascular, peritumor and intratumor inflammation were major contributors to variation captured by PC2 or dimension 2 (Fig. 2d). TiME feature contributions to PC1 and PC2 therefore explain the distribution of BCC patient samples according to variability in inflammation and vasculature, where samples along PC1 cluster based on vasculature (Fig. 2e, f, low=purple, high=pink and black) and samples distributed along PC2 cluster based on inflammation (Fig. 2e, f low=black, high=pink and purple) thereby allowing for discrete phenotypes to emerge: Inflam^LOWVasc^HIGH (black), Inflam^HIGHVasc^LOW (purple) and Inflam^HIGHVasc^HIGH (pink).

Table 1 High agreement suggests the reproducibility and presence of TiME features on histopathology

Full size table

**Fig. 2: Unsupervised clustering identifies three main RCM TiME phenotypes and assigns groups based on inflammation and vasculature.**

Molecular landscape associates with Inflam^HIGH and Vasc^HIGH signatures

To determine transcriptional variation underlying observed phenotypic differences in inflammation and vasculature, we measured transcript abundance by bulk RNA sequencing from 14 BCC samples representing the two groups Inflam^HIGHVasc^HIGH and Inflam^LOWVasc^HIGH (Fig. S2a–d). To identify genes specific to functional gene regulatory networks relevant to BCC tumor and TiME, we performed modular co-expression analyses using CEMiTool (Fig. S2e–h)²⁸. Of the resulting 8 modules comprised of co-regulated genes, 2 modules (M2 and M5) showed statistically significant enrichment of gene activity in RCM TiME phenotypes, both M2 and M5 displayed higher activity in the Inflam^HIGH group (Fig. S3a–d and Fig. 3a–b). To understand differential control of biological processes and cell types, M2 and M5 were annotated using gene ontology (GO) enrichment analysis. We discovered distinct biological processes, pathways, and cell types for M2 compared to M5; M2 module genes exhibited significant enrichment in pathways such as pro-inflammatory signatures, allograft rejection, interferons, and myeloid cells while genes comprised in M5 displayed signatures related to leukocyte adhesion and migration along with cytokine receptor activity. M2 shows signatures enriched in cells comprising blood whereas genes expressed in M5 were associated with blood vessels (Fig. 3c and Fig. S3e). Given the shared concentration of genes participating in immune cell function, we generated a gene interaction network using gene connections defined in T-lymphocytes (curated gene pairs downloaded from TissueNexus²⁹, see methods). We discovered module hub genes participate in T-lymphocyte gene regulatory networks (GRNs) to a high degree and together with network hub genes are connected to genes enriched in GO terms involved in regulation of distinct inflammatory pathways for M2 and M5. For M2 T-lymphocyte network, these pathways include T-cell activation and differentiation, myeloid cell differentiation and leukocyte adhesion. In contrast, M5 T-lymphocyte networks were enriched in blood vessel proliferation and cell adhesion (Fig. 3d and Fig. S3f). We next assessed gene networks in tissues and cell types related to BCC and TiME albeit beyond T-lymphocytes. Interestingly, we find module hub genes persist in connectivity with gene networks in skin, macrophage and blood. Notably, network hub genes are largely shared across cell types, indicating M2 and M5 genes participate in shared pathways across functionally distinct tissues and contribute to observed differences in inflammatory phenotypes. Indeed, module hub genes (SLA, DOCK2, CD34, ABCA6/9) and shared network hub genes including ICAM1, VCAM1, TGFBR3, CXCL12 and PDGFD are known to be involved in immune and vascular signaling and function, immune cell migration and all of these are overexpressed in the Inflam^HIGH as compared to the Inflam^LOW phenotype (Fig. S3g and Fig. 3e).

**Fig. 3: Molecular signatures reveal inflammatory signatures predominantly correspond to Inflam^HIGH phenotype.**

Since a role for differential regulation of immune cell function among RCM phenotypes emerged from assessment of bulk RNA-seq, we hypothesized variations in populations of immune cells could contribute to differences in RCM phenotypes. To assign transcripts to immune cell types and estimate cell proportions, we deconvoluted bulk RNA-seq using CIBERSORTx³⁰ (Fig. 3f). Unsupervised k-means clustering on CIBERSORTx output distinguished Inflam^HIGH and Inflam^LOW phenotypes, which was not achieved using all variable transcripts from bulk RNA-seq or when assessing variability in genes comprising M2 and M5 (Fig. 3g, Figs. S2i–k, S3h). Relative differences in composition of major immune cell types were inferred and significant differences in cell proportion across phenotype groups were discovered for CD4⁺ T memory cells (resting) and M1 macrophages (Fig. 3h).

Inflam^HIGH phenotypes correlate with abundant CD8⁺ and CD8⁺ PD1⁺ cells

The gene expression results identified immune cell composition and function, especially T-cells and macrophages were important in distinguishing RCM phenotypes. Subsequently, differences in T-cells and macrophages along with PD-1/PDL-1 checkpoint expression were investigated across phenotypes in BCC patients. Assessing peritumoral areas of inflammation, median % (mean %, 95% CI) proportions of CD8⁺ cells were 6.35% (12.01%, 1.53–22.49%) in Inflam^LOWVasc^HIGH, 21.97% (21.24%, 15.68–26.79%) in Inflam^HIGHVasc^LOW and 11.74% (11.2%, 8.1–14.29%) in Inflam^HIGHVasc^HIGH groups. Furthermore, out of these CD8⁺ cells, 25.22% (27.33%, 13.11–41.55%), 46.86% (40.7%, 23.99–57.41%), 31.01% (31.04%, 20.01–42.06%) were PD1 positive in Inflam^LOWVasc^HIGH, Inflam^HIGHVasc^LOW and Inflam^HIGHVasc^HIGH, respectively (Fig. 4a). Similar trends were seen in FOXP3⁺ T-regulatory cells and CD68⁺ macrophages in peritumoral infiltrates (Fig S4a). High intratumoral infiltration by CD68⁺ macrophages was observed in Inflam^HIGHVasc^LOW (2.9%, 0.69–5.12%) as compared to Inflam^LOWVasc^HIGH (1.26%, −0.39–2.9%) and Inflam^HIGHVasc^HIGH (0.47%, 0.03–0.91%) groups (Fig. 4a). Additionally, flow-based immunophenotyping on 3 BCC tumors indicated higher activated CD8⁺GzmB⁺ and CD8⁺ Ki67⁺ cells in Inflam^HIGHVasc^HIGH as compared Inflam^LOWVasc^HIGH, also suggesting the prevalence of inflamed state in the Inflam^HIGH phenotype (Fig. S4b–d).

**Fig. 4: Immunophenotyping through multiplexed staining correlates with RCM phenotypes.**

Lymphocytes show strong association while tertiary lymphoid structures (TLS) show minimal association with TiME phenotypes

Tertiary lymphoid structures (TLS) are lymphoid formations that form in nonlymphoid tissues at sites of chronic inflammation and are associated with improved patient outcomes and response to immunotherapies^31,32. We investigated TLS as markers for immune states. As seen in multiplexed immunofluorescence, large variation was observed in the T-cell and B-cell immune infiltrates and tertiary lymphoid structures (TLS) across patients. Total CD3⁺ and CD20⁺ lymphocytes proportions were 9.2% (12%, 7.7–16%) in Inflam^LOWVasc^HIGH, 25% (25%, 15–36%) in Inflam^HIGHVasc^LOW and 14% (14%, 8.4-20%) in Inflam^HIGHVasc^HIGH groups. TLS area positivity did not differ significantly across the phenotypes (Fig. 4b).

Melanoma TiME phenotypes correlate with histopathology and T-cell infiltrates

Statistical clustering for 2 groups on the melanoma TiME features showed the presence of two main clusters: Inflam^LOWVasc^HIGH and Inflam^HIGHVasc^HIGH. The phenotypic classes were assigned based on PCA, similar to BCC samples (Fig. 5a, Fig. S5a–e). No correlation of TiME phenotypes with tumor stage, age, gender or sun exposure was observed, although most of the invasive melanomas (superficial spreading subtype) belonged to the Inflam^HIGHVasc^HIGH phenotype. The CD3⁺ T-cell proportion was found to be 7.2% (7.2%, 5.2–9.2%) in Inflam^HIGHVasc^HIGH while 2.1% (2.2%, 0.49–3.9%) in the Inflam^LOWVasc^HIGH groups. As seen in BCC, TLS did not show a significant association with TiME phenotypes (Fig. 5b).

**Fig. 5: Identical RCM TiME phenotypes in melanoma correlate with immune signatures.**

Automated quantification of TiME features is feasible

Next, we investigated automated quantification of RCM TiME features— immune cells, leukocyte trafficking and vasculature, using machine learning and image processing algorithms. Quantification of immune cell density was explored using a U-Net segmentation model, which resulted in a Dice coefficient³³ (a common measure of accuracy) of 0.72 (Table 2). Representative images and segmentations are shown in Fig. S6a. Image-processing algorithms were used for quantification of vascular features (vessel area, diameter and number). Using manual vessel segmentation as ground truth, the Dice coefficient ranged between 0.29–0.78 (Table 2). Leukocyte trafficking was quantified using a custom pipeline involving image registration, background subtraction and particle tracking (detailed in Methods). High agreement was observed between manual and automated counts at the optimization and final validation stages (Table 2). Parameters and representative examples for vessel and trafficking counts are shown in supplementary data (Fig. S6b–e). Subsequently, correlation of RCM TiME features with corresponding gene expression for inflammation, angiogenesis and trafficking suggested high correlation between total area % inflammation with myeloid cells, dendritic cells and T-cell markers (CSF1R, CD1E and CD3E), and total leukocytes-like area % with cytotoxic T-cell markers (CD8B and GZMA). Vascular features such as vessel diameter and trafficking were also correlated with angiogenesis and endothelial markers (VEGFD, VEGFA and PDGFD), and trafficking markers (CCL-18, CAV-1 and CCL28), respectively (Fig. 6a). These TiME features were additionally used as traits in module-trait analysis to determine relationships with gene co-expression modules shown to be enriched in the Inflam^HIGH groups (Fig. 3a). A significant correlation between M5 and total myeloid (dendritic cell, macrophages) inflammation area was discovered (Figs. 6b, S6f).

Table 2 Automated quantification of RCM features is feasible

Full size table

**Fig. 6: Quantified RCM TiME features correlate with gene expression.**

Treatment response to imiquimod can be predicted by combined immune and vascular features and TiME phenotyping

Treatment response to the TLRA immune-therapy in BCC patients was correlated with TiME features and phenotypes. Unsupervised statistical clustering yielded 2 groups: one group for responders (5 of the 7 responders) and the other for non-responders (4 of the 6 non-responders). The responder group was characterized mainly by high inflammation, while the non-responder group showed both high inflammation and vasculature (Fig. 7a). Phenotype-prediction of responders and non-responders was performed by overlaying on the original BCC phenotyping scatter plot (Fig. 2f). Most responders (5 of 7) belonged to the Inflam^HIGHVasc^LOW phenotype (Fig. 7b). Further, evaluation of differences in TiME features between responders and non-responders demonstrate increased number of vessels and stromal macrophages/dendritic cells in non-responders. Higher leukocyte trafficking, vessels, and stromal macrophages were present in 50%, 100%, and 86% of the non-responders, respectively. Although intratumoral inflammation and tumor infiltrating lymphocyte (TILs)-like features were similar, vessel density was found to be significantly different between responders and non-responders (Fig. 7c). Linear separability plots confirmed addition of vessels to inflammation enhanced separation between responders and non-responders (Fig. S7a). Linear regression models for response prediction (Fig. S7b) demonstrate low predictive power of inflammation as a variable, either as “TIL-like cells” or “intratumoral inflammation” with accuracy of 46% and 61%, respectively. Addition of stromal vessels to intratumoral inflammation or TIL-like cells as features in the linear regression model resulted in best model performance (71% sensitivity, 83% specificity and 76% accuracy) (Table 3).

**Fig. 7: Inflam^HIGHVasc^LOW phenotype corresponds to highest response in TLRA treated patients.**

Table 3 Modeling of vascular features with immune cells improves response prediction accuracy

Full size table

Discussion

Phenotyping the tumor microenvironment beyond simply T-cell infiltration is crucial to develop robust predictive platforms for patient stratification during immunotherapies. Since not all T-cell inflamed or hot phenotypes respond to treatment, tumors seemingly utilize additional mechanisms for evading immune response and establishing an immune-suppressive microenvironment. The tumor vasculature plays an important role in mediating this immune suppression and immune exclusion. To comprehensively evaluate the inflammation-vascular axis, we integrated features from inflammation and vasculature to investigate the presence of distinct in vivo TiME phenotypes in skin cancers that are predictive of patient response to therapy. The TiME features imaged non-invasively on RCM show high agreement with the gold standard histopathology, confirming the feasibility of detecting these features in vivo in patients (Table 1). Using a combination of vessel and immune features, we derived phenotypes using unsupervised clustering to minimize subjective bias. Within the BCC dataset, phenotyping using unsupervised clustering yielded three main phenotypes (Fig. 2a). In the smaller melanoma dataset, the clustering was programmed for two unsupervised clusters (Fig. 5a). While the Inflam^LOWVasc^LOW phenotype is a clinical possibility, skin cancers are typically highly immunogenic and vascular tumors, therefore we did not expect to encounter this phenotype in this smaller dataset^34,35,36. Assessment of large cohorts would be needed to identify, and study the effect of this phenotype on treatment response in skin and other cancers. The TiME phenotypes in BCC and melanoma strongly correlated with inflammatory molecular signatures, along with T-cell and macrophage abundance (Figs. 3, 4a, b, 5b). Furthermore, the phenotypes and combined vascular/immune features better correlated with treatment response than inflammation as a singular feature and best treatment response was observed in Inflam^HIGHVasc^LOW phenotype (Fig. 7a–c, Table 3), highlighting the importance and potential clinical utility of this approach in integrating both inflammation and vasculature to characterize and phenotype TiME.

Specifically, analysis of gene co-expression modules generated from all variable genes expressed across 14 BCC samples successfully identified eight modules of co-regulated genes, of which 2 modules were significantly enriched in RCM phenotypes (Fig. 3a, b). The modules show distinct regulation of immune cell and vascular function where M2 was enriched for genes associated with T-cell activation and differentiation, myeloid cell differentiation and leukocyte adhesion, in contrast to M5 enrichment of genes involved in blood vessel proliferation and cell adhesion (Fig. 3c). Given the shared concentration of genes participating in immune cell and vascular functions for both M2 and M5, albeit potentially from distinct cell origins, we generated a gene interaction network using gene connections in T-lymphocytes, macrophages, blood and skin. Together, M2 and M5 T-lymphocyte networks appear to engage in cell type specific pathways in agreement with module enrichment in biological processes similarly describing contents of blood, i.e. immune cells, for M2 whereas cell types comprising the blood vessel were enriched for M5.

Investigations into specific genes and pathways within the module and network hub genes demonstrate higher prevalence of genes crucial for immune cell migration (including leukocyte trafficking) and vascular function such as ICAM1, VCAM1, CXCR4, CXCL12, NCKAPL1, DOCK2, PDGFD and TGFBR3 in Inflam^HIGHVasc^HIGH phenotype (Fig. 3d, e). This suggests this phenotype likely had comparatively ‘normal’ vasculature³⁷. Conversely, Inflam^LOWVasc^HIGH show downregulation of major adhesion molecules (ICAM1, VCAM1) along with significantly lower immune cell signatures (Fig. 3d, e), suggesting features of endothelial anergy and anergy-induced immune excluded state^16,37. This immune-exclusion and endothelial anergy state may have been potentiated by immunosuppressive tumor-intrinsic factors (CTNNB1, PTEN, COX11)^38,39 in the Inflam^LOWVasc^HIGH phenotype, which needs to be studied in context of tumor genetic signatures, mutational burden and immune exclusion^40,41,42.

Furthermore, the majority of module hub genes (80% of the top 5 genes in M2 and M5) were found to have known immune functions (Fig. 3d-e). Notably, network hub genes are largely shared across cell types, regardless of tissue origin of gene network. This exemplifies the robustness of the M2 and M5 network hub genes, through participation in shared pathways across functionally distinct tissues and contributing to observed differences in RCM phenotypes. Most of the network hub genes common across gene networks are also involved in inflammatory or vascular pathways. Few module and network hub genes (SCN9A, PLCB4, RASSF2) have minimal or no direct immunological function, although they show connectivity within gene networks enriched in skin, macrophage, blood and T-lymphocyte pathways. While SCN9A encodes the voltage-gated ion channel Na_v 1.7 and is primarily associated with pain disorders, it can also influence chemokine-induced migration of the CD1a⁺ dendritic cells⁴³. Similarly, PLCB4 or phospholipase C beta-4 enzyme has established neurological roles, and was recently shown to play a role in selective promotion of CD8-T cell-dependent adaptive immunity^44,45. Further, RASSF2, a known tumor suppressor gene that promotes apoptosis and cell-cycle arrest is also implicated in suppression of immune responses, angiogenesis, and metastasis⁴⁶. Enrichment of these genes in the Inflam^HIGHVasc^HIGH phenotype draws parallels to these studies and strongly suggests previously unknown roles of these relatively non-immune genes in regulating inflammation and/or vasculature within the tumor microenvironment. Although the DEG analysis on bulk RNA-seq data did not identify other major immunological or vascular genes, attributed to the low sample size and high variability within patient data, our analysis into co-expression of variable genes enabled detection of known and previously unknown genes important in resolving phenotypes.

Significantly, clustering on the CIBERSORTx data which enriches for immune cells genes from 22 cell types explained assignment of samples to RCM phenotypes (Fig. 3g). CIBERSORTx also identified major differences in T-cells and macrophages across the two groups, which were subsequently validated using immunostaining (Figs. 3f, h and 4). While heterogeneity in immune cell populations through CIBERSORTx does resolve RCM phenotypes, it only captures one component i.e. inflammation, within the phenotype. We anticipate that similar heterogeneity in cell populations comprising the stroma such as fibroblasts and endothelial cells will also capture distinguishing features between these phenotypes, these were not addressed in this study. Since genes involved in immune response regulation and function yet possibly originating from non-immune cells featured prominently in the module and network hub genes, future studies should also address cellular heterogeneity in these stromal cell populations using unbiased approaches such as single-cell RNA-sequencing that may comprehensively address the source of gene expression differences across RCM phenotypes.

Using immunophenotyping through dual IHC and multiplexed IF, the RCM phenotypes correlated with peritumoral abundance of CD3⁺, CD20⁺, CD8⁺ and CD8⁺ PD-1⁺ T-cells in BCC, and CD3⁺ T-cells in melanoma (Fig. 4a, b and 5b). Similar trends for abundance of myeloid cells (macrophages) and regulatory T-cells were found in the Inflam^HIGH as compared to the Inflam^LOW. While the strongest difference in CD8⁺ cells was found between Inflam^HIGHVasc^HIGH and Inflam^HIGHVasc^LOW groups, the overall CD8⁺ cells in the Inflam^LOWVasc^HIGH group were lowest. One outlier within this group had intense immune infiltrates in the deeper dermis, explaining very high lymphocytic density on IHC which was not found in RCM. Furthermore, tentatively excluding this outlier patient from Inflam^LOWVasc^HIGH led to a significant difference in CD8⁺ T-cells between Inflam^HIGH and Inflam^LOW groups. No association was observed with the area of tertiary lymphoid structures, (TLS) a hallmark of an inflamed micro-environment, and positive cancer outcomes³², in both melanoma and BCC. We suspect this discrepancy to be due to most of the TLS arising much deeper in the tissue, outside the field-of-view of RCM (depth-limited to 0.25 mm). In conclusion, a comprehensive immunohistochemical assessment using multiplexed IHC supports the described distinct phenotypes as identified through RCM. These results were also corroborated on the flow-based immunophenotyping on 3 patients that showed higher activated CD8⁺ cells in the Inflam^HIGHVasc^HIGH phenotype (Fig. S4b–d).

Although immune-based therapies have shown unprecedented and durable responses, the response rates have been variable. Topical TLRA imiquimod immune-therapy has shown response rates between 60-84% in superficial melanoma and basal cell carcinoma^47,48, highlighting the need to identify the responders and non-responders early to minimize treatment side-effects and patient morbidity, and streamline management. Ulceration, LINC and PD-L1 expression, and perifollicular infiltration of melanocytes have shown some association with response to imiquimod in BCC and superficial melanoma^49,50,51, but are not often used for informing treatment decisions. There is a need for more comprehensive and quantitative analysis of major determinants of anti-tumor immunity within TiME that enable patient stratification at the bedside. Because of the dynamic nature of the interactions within TiME, in vivo imaging is crucial for studying dynamic processes, including active vascular processes such as leukocyte trafficking which is optimally studied as live dynamic events, as opposed to ex vivo tissue studies on vasculature which can show inconsistent vessel measurements²⁰. As skin cancers are accessible for such non-invasive imaging, we evaluated thirteen patients undergoing prospective treatment with topical TLRA. Seven patients responded to treatment (complete tumor clearance) while 6 did not respond (partial or no tumor clearance). On this small cohort, the Inflam^HIGHVasc^LOW phenotype correlated with maximal response, and inclusion of vascular features in predictive models led to improved response prediction accuracy (Fig. 7c, Table 3). On RCM, this phenotype demonstrated lower immunosuppressive angiogenic features with higher intratumoral inflammation. This phenotype also presented as the inflamed exhausted phenotype on multiplexed IF with highest abundance of CD8⁺ and CD8⁺ PD-1⁺ T-cells (Fig. 4a). Notably, only Inflam^HIGHVasc^LOW showed highest response to the TLRA immune-therapy (Fig. 7b). Therefore, we speculate that the Vasc^HIGH phenotypes could benefit from adjunct vascular-targeted therapies in combination with immune-modulating therapies. However, specific mechanisms of treatment resistance attributed to “vasculature” will first need to be resolved for the two groups within Vasc^HIGH by studying TLRA response on a large patient cohort. For example, vessel normalization through pharmacological targeting of the Wnt/β-catenin pathway and anti-angiogenic topical treatments (COX-2, basic fibroblast growth factor or bFGF inhibitors) can overcome endothelial cell anergy and, reinduce/enhance adhesion molecule expression for increased leukocyte infiltration of effector T cells into tumors¹⁵, especially benefiting Inflam^LOWVasc^HIGH ^52,53. The predictive accuracy of 76% achieved by incorporating both inflammation and vascular features in our models (Table 3) can potentially be improved by incorporating adjunct stromal features like collagen and mucin in future studies, since these can also influence immune infiltration into tumors. Thus, this study confirms the feasibility of a predictive platform that will ultimately introduce a more individualized or personalized immune-therapy treatment approach, similar to the IMPACT^TM Panel⁵⁴.

Using RCM and skin cancer as a model, we demonstrate the proof-of-concept for unperturbed characterization and phenotyping of TiME inside patients and the feasibility to predict response/no-response before treatment. However, manual evaluation of TiME features would require an effort that is unrealistic in clinical practice. Manual approaches would also inevitably involve extensive training of readers in feature recognition. The current era of machine learning and artificial intelligence allows extraction of quantitative biomarkers from images to inform on disease characterization, monitoring and assessment of response to treatment, also a mission of the quantitative imaging network for radiological imaging^55,56. Quantitation also has the potential to minimize intra- and inter-observer variability of feature evaluation and provide objective decision-support tools in patient management. Thus, towards real-world clinical implementation, we produced preliminary algorithmic pipelines for automated quantification of immune cell density, leukocyte trafficking frequency, and vessel thickness and density. The preliminary machine learning model trained on 2 subclasses of immune cells (leukocyte-like, and myeloid cells that included macrophages and dendritic cells) showed a 72% classification accuracy for estimating areas covered by immune cells (Table 2). Subsequent efforts will be directed towards instance segmentation (e.g., object detection) to estimate the occurrence statistics of cells, and the training will be extended to individual morphologies as shown in Fig. S1a. Through RCM imaging of blood vessels in humans, we uncovered important limitations that affect the accuracy, including difficulty in acquiring good quality data at specific anatomical locations and axial motion during data acquisition by the hand-held probe due to lack of stability. These limitations resulted in a more variable Dice coefficient range (Table 2). We have focused our efforts on overcoming these limitations by employing tissue coupling during video acquisition and using advanced frame stabilization tools to further improve the processing of vascular features. The quantified TiME features also demonstrate strong quantitative correlations with the amount of gene expression, suggesting the quantifications truly represented underlying immune or vascular features (Fig. 6a). M5 module genes which were enriched in the Inflam^HIGHVasc^HIGH group also correlated with density of myeloid cells (macrophages, dendritic cells) (Fig. 6b), similar patterns were also seen in the multiplexed IF results (Fig. 4a). In this study, the quantified trafficking included all major trafficking events (rolling, adhesion, and crawling) (Supplementary Movies 2, 3) that correlated with CCL18 and CCL28 expression (Fig. 6a). Since specific genes orchestrate discrete steps within the trafficking cascade⁵⁷, better gene correlation with individual trafficking steps (e.g. ICAM1/VCAM1 with rolling cells) can be expected. Thus, subsequent efforts will be focused on quantifying individual trafficking events.

Critically, through automated analysis, it is also possible to quantify longitudinal changes to monitor treatment-induced alterations e.g. normalized vasculature after vascular normalizing therapies could be analyzed in real-time to help assess response and uncover mechanisms of treatment resistance. This real-time longitudinal evaluation and quantification will contribute to treatment optimization and personalization of immune-therapy regimens. Similar to our recent efforts in automating image acquisition and diagnosis^58,59, in future, the quantification pipelines would be implemented on clinical devices to enable real-time quantification and TiME phenotyping for response prediction using validated quantitative predictive platforms. Thus, this may enable more precise and real-time clinical decision-making for patient stratification at the bedside.

Our study demonstrates a combination of high-resolution spatially resolved and dynamic imaging to advance current limitation in static detection of TiME features. Outstanding limitations of the approach include grayscale-limited specificity tissue contrast and imaging depth to 0.2–0.25 mm. The label-free approach enables visualization of all TiME features, but is limited in specificity for functional phenotyping. One major advantage is the feasibility of longitudinally monitoring spatio-temporal changes in immune cells and vasculature during treatment, akin to on-treatment biopsies. With the present state of RCM devices and technology, this approach is currently restricted to accessible diseases and cancers on the skin and mucosa. Complementary multimodal approaches⁶⁰ such as dynamic optical coherence tomography or optical frequency domain imaging for imaging vasculature, lymphatics and tissue viability^61,62, multiphoton microscopy for better contrast and collagen delineation^63,64,65, photoacoustic microscopy for functional vascular imaging⁶⁶ and fluorescence lifetime imaging for probing immune cell specificity and activation states⁶⁷ will further enhance in vivo TiME visualization and enhance current TiME phenotyping in the future.

In subsequent studies, extensive validation with targeted molecular analyses on precision biopsies⁶⁸ will enable better correlations. Since only the most prominent signals can be detected in bulk-sequencing, other minor or rarer cellular differences in immune and stromal cells, especially in non-infiltrated samples may be detected using single cell or spatial transcriptomic analysis, to further our understanding of RCM phenotyping^69,70. Through robust prospective studies on large cohorts, fundamental basis of phenotyping and their correlation with variable treatment responses in cancer immunotherapy systems can be explored for better patient stratification. This research could impact several oncological settings, including but not limited to cutaneous and mucosal, primary and metastatic keratinocyte and melanocyte cancers (melanoma, squamous cell carcinoma, basal cell carcinoma), and cutaneous lymphomas. The advances in TiME phenotyping presented here can enable hypothesis-driven research for developing new druggable targets and response predictive platforms, and gaining mechanistic insights on host anti-tumor immune response in cancers.

Methods

Study design

The study was approved by the Institutional Review Board at Memorial Sloan Kettering Cancer Center (MSKCC-IRB). Written informed consent was obtained from all participants recruited under active IRB protocols IRB#99-099 and IRB#21-019 included under the clinical trial NCT00588315. All research was performed strictly in accordance with the Declaration of Helsinki and relevant guidelines and regulations. The main research objectives for this study were as follows: (i) investigate TiME phenotyping in skin cancer patients, (ii) investigate biology and immunological states underlying TiME phenotyping, (iii) correlate specific TiME features and phenotypes with response to a toll-like receptor agonist (TLRA) therapy. This observational cross-sectional study involved RCM imaging on human patients presenting with skin lesions referred for dermatology consultation in the Dermatology Service at Memorial Sloan Kettering Cancer Center (MSKCC), New York. Patients (aged 18 or older) with either a previously biopsied or clinically suspected skin cancer or drug rash amenable for imaging were prospectively enrolled between November 2018 and November 2021. On a subset of patients, patients undergoing topical TLRA treatment (5x/week for 6 weeks) as standard of care were imaged at baseline before treatment (T0), and end of treatment to confirm tumor clearance. Validation of phenotyping was investigated on tissues obtained as part of standard of care or research by histopathology, bulk RNA-sequencing, multiplexed IF and dual IHC to correlate with underlying immune and vascular features. Automated quantification of TiME features, and modeling of treatment response on TiME features and phenotypes was performed for near-future clinical applications in predicting treatment responses (Fig. 1).

In vivo imaging

In vivo RCM imaging was performed prospectively on 118 lesions using RCM (VivaScope 1500 or handheld VivaScope 3000, (Caliber Imaging and Diagnostics, Rochester, NY, USA) and/or an integrated handheld RCM-OCT prototype. VivaScan v10.0 (Caliber Imaging and Diagnostics, Rochester, NY, USA) was used to acquire images and images were interpreted in real-time at the bedside to select representative areas with tumor, immune cells and blood vessels across the lesion by 2 investigators (MC and AS) having more than 5 years of RCM reading experience. Surrounding normal skin was also imaged as control, but excluded from analysis since it lacked tumor and TiME features. Mosaics (large area sampling), stacks (depth sampling), scanning and single field-of-view (FOV) videos were acquired from multiple regions within the lesion and saved in an online database (Vivanet, Caliber ID, Rochester, NY) or on a local drive. Patients visiting Dermatology service were recruited in this study. Twenty-eight BCC lesions from 20 patients (9 females, 11 males; age range 43–88 years) were used for the BCC phenotyping analysis. Thirteen melanoma lesions from 12 patients (4 females, 8 males; age range 46–87 years) were used for melanoma phenotyping. Thirteen lesions on 8 patients (4 females, 4 males; age range 54–70 years) undergoing TLRA agonist (imiquimod) treatment were included in this study. Imaging was performed before treatment or standard of care biopsy for all patients. Imaging was performed once for a given lesion and not repeated due to clinical constraints. In the imiquimod dataset, patients were imaged after treatment to investigate tumor clearance.

Patient tissue

Biopsies (targeted or non-targeted) taken as standard of care or for research use were used for histopathological, immunohistochemical, RNA-sequencing and flow cytometry correlations. Formalin-fixed paraffin embedded (FFPE) specimens from 39 lesions (27- basal cell carcinoma, 11- melanoma, 1-lichen-planus like keratosis) were used for histopathological and immunohistochemical (IHC) correlations. RNA-extraction was performed on 25 FFPE specimens with adequate tissue. However, RNA-seq was performed only on 14 out of 25 specimens representing the two groups Inflam^HIGHVasc^HIGH and Inflam^LOWVasc^HIGH due to limited RNA quantity. Multiplexed IF was performed on 27 FFPE BCC specimens, 3 specimens were excluded from the analysis since the tissue quality was heavily deteriorated during multiplexed staining. Imaging-guided small (2–3 mm) targeted biopsies⁷¹ was performed on 5 lesions for frozen section histopathology and IHC (n = 2), and flow cytometry (n = 3). Although flow cytometry provides more quantitative estimates of cellular populations, acquiring adequate research specimens (≥ 3 mm) can potentially compromise patient care in a diagnostic setting and is not routinely practiced. Since we anticipated challenges in acquiring additional fresh tissue for flow cytometry beyond the first few samples, we concentrated our efforts on multiplexed IF and dual IHC on archived pathology specimens to validate the phenotypes.

Manual RCM and histopathology evaluation

RCM features were manually evaluated (Fig. S1c, Table 1) by either of the 4 readers with at least 5 years’ experience (AS) or >20 year experience in interpreting RCM images (MC, SG, CMAF). The major features evaluated on manual reading included number of vessels, dilated vessels, trafficking, intratumoral inflammation, peritumor inflammation and perivascular inflammation. These features were graded on a scale of 0–3 after exhaustive review of data from each patient. Data from 1 patient was excluded due to motion blur during image acquisition. For melanoma, specific features such as lichenoid inflammation, total lymphocyte-like cells were also evaluated along with spatial distribution of inflammation, immune cells and vasculature. For imiquimod response study, spatial distribution of vessels and type of three immune morphologies (dendritic cells, lymphocyte-like cells and macrophages), tumor-infiltrating lymphocyte-like cells, mucin and tumor regressing areas were also noted for more comprehensive assessment and correlation with treatment response. Same TiME features evaluated on RCM were graded on digitized histopathological slides by board-certified dermatopathologists (MG, MP). Manually evaluated features were used for statistical clustering for phenotyping and response prediction.

Statistical clustering for TiME phenotyping

Unsupervised statistical clustering on manually evaluated TiME features was performed to explore classification trends or phenotypes using HCPC and PCA in R using FactoMineR⁷², FactoExtra⁷³ and ggplot2⁷⁴ packages. HCPC was performed using the HCPC function while PCA was performed using the prcomp function in R. The dendrograms were plotted using the fviz_dend functions. For PCA, the percentage contribution scree plot, the variable contribution and the biplot were obtained using fviz_contrib, fviz_pca_var and the fviz_pca_biplot functions, respectively.

RNA extraction

FFPE sections from n = 25 specimens were deparaffinized using the mineral oil method. Briefly, 800 µL mineral oil was mixed with the sections and the sample was incubated at 65 °C for 10 min. Phases were separated by centrifugation in 360 µL Buffer PKD and Proteinase K was added for digestion. After a three-step incubation (65 °C for 45’, 80 °C for 15’, 65 °C for 30’) and additional centrifugation, the aqueous phase containing RNA was removed and DNase treated. The RNA was then extracted using the RNeasy FFPE Kit (QIAGEN catalog # 73504) on the QIAcube Connect (Qiagen) according to the manufacturer’s protocol with 285 µL input. Samples were eluted in 13 µL RNase-free water.

Transcriptome sequencing

After RiboGreen quantification and quality control by Agilent BioAnalyzer, 356–500 ng of total RNA with DV200% varying from 88-93 underwent ribosomal depletion and library preparation using the TruSeq Stranded Total RNA LT Kit (Illumina catalog # RS-122-1202) according to instructions provided by the manufacturer with 8 cycles of PCR. Samples were barcoded and run on a HiSeq 4000 in a PE100 run, using the HiSeq 3000/4000 SBS Kit (Illumina). On average, 78 million paired reads were generated per sample and 20% of the data mapped to mRNA. While RNA extraction was performed on 25 FFPE specimens, due to limited RNA quantity, RNA-seq could be performed only on 14 specimens representing the two phenotypes Inflam^HIGHVasc^HIGH and Inflam^LOWVasc^HIGH.

Differential analysis of gene expression

Differentially expressed transcripts between RCM Inflam^HIGH and Inflam^LOW groups were determined using pairwise comparisons in edgeR^75,76,77 performing exact test (FDR < 0.05) on TMM-normalized read counts with filtering to remove lowly expressed transcripts (using filterByExpr argument in edgeR). R version 3.6.3 was used in analyses.

To determine sets of genes associated with RCM phenotypes, Co-Expression Modules identification Tool (CEMiTool)²⁸ was implemented in R to identify and analyze gene co-expression modules. Default parameters were used after assessing normal distribution of TMM normalized and log₂ transformed transcripts across samples (correlation method = Pearson, R² > 0.8, filtering pvalue = 0.1). Using CEMiTool²⁸, resulting gene modules were assessed for enrichment in RCM phenotype classes and inspected for expression patterns of individual genes within modules. Gene modules with significant enrichment in RCM phenotype were annotated for biological relevance using R interface with Enrichr^78,79,80 database (https://cran.r-project.org/web/packages/enrichR/vignettes/enrichR.html) for following curated gene sets: Gene Ontology (GO), pathway enrichment in curated MsigDB Hallmark gene set^81,82, cell/tissue-specific gene expression profiles using GTEx Tissue and Human Gene Atlas⁸³. To discover module enrichment for disease processes, enrichment in GTEx Aging and OMIM repositories were performed as well. Gene-gene interactions to discover module gene connections with functional gene regulatory networks defined for T-lymphocytes, macrophage, skin, and blood were generated using gene pairs defined from TissueNexus²⁹ (https://www.diseaselinks.com/TissueNexus/index.php). Interaction data was used in CEMiTool²⁸ to identify overlap with module genes, discover network hub genes, and visualize network interactions. Network hub genes were used as input in TissueNexus database, selecting for appropriate tissue/cell type, to query top-connected neighbors followed by GO enrichment analysis using GOTermFinder. Resulting top 5 enriched GO terms (adj pvalue< 0.05) were reported for M2 and M5. Eigen gene values summarizing expression of all genes within modules M2 and M5 were correlated (method = Spearman) with TiME phenotypes to interpret disease relevance and visualized using a correlation matrix along with scatterplot reporting significant R-value using Hmisc⁸⁴ package in R (https://cran.r-project.org/web/packages/Hmisc/Hmisc.pdf). To inspect major sources of variation in gene expression contributing to RCM phenotypes, principal component analysis (PCA) was performed using prcomp function and confidence ellipses in PCA were generated using factoextra⁷³ package in R (https://rpkgs.datanovia.com/factoextra/index.html).

CIBERSORTx analysis

CIBERSORTx was used for the immune cell analysis to delineate immune subsets using 547 genes for 22 immune cell types⁸⁵. Transcript per million values were used as input. CIBERSORTx chooses the record with the highest mean expression across the mixtures during analysis. The gene expression file with 14 cases was uploaded to CIBERSORTx as a mixture file, and CIBERSORTx was run with the following options: relative and absolute modes together, LM22 signature gene file, 100 permutations, and quantile normalization disabled. Sample distance matrix resulting from immune cell distribution and k-means clustering were used to interpret CIBERSORTx output.

Immunohistochemistry

IHC for CD1a, CD68, CD3 and CD20 was performed on Bond Rx system (Leica Biosystems, US). The protocol for the Bond Rx platform included heat retrieval followed by primary antibody incubation (Santa Cruz Biotech, US). Polymer detection was through DAB Kit (Leica Biosystems, catalog #DS9800). For the dual CD3/CD20 IHC sequential staining, we performed heat retrieval, incubation with primary antibodies (anti-CD3, anti-CD20) followed by polymer detection kits (Leica Biosystems, US, catalog #DS9800, DS9390). The list of antibodies and their dilutions have been reported in Supplementary Methods as Table S1. The IHC slides were digitized on a slide scanner (Aperio Imagescope, Leica Biosystems, US).

Multiplexed immunofluorescence and analysis

The Opal ‘7-color manual IHC kit’ (Akoya Biosciences, Marlborough, MA) was used to conduct IF staining on the autostainer LabSat Research autostainer (Lunaphore Technologies, Tolochenaz, Switzerland). The stains were visualized using a Vectra Polaris Automated Quantitative Pathology Imaging System (Akoya Biosciences, Marlborough, MA). The quantitative multiplexed IF results were derived using the image analysis platform HALO version 2.3 (Indica Labs, Albuquerque, NM, USA). Depending on the size of the tissue, 1–12 (mean = 5.25) representative intra-tumoral and peritumoral region-of-interests (ROIs) were chosen, focusing on areas of immune infiltrates. Cell counts were summed up by compartment for every patient and values were normalized by DAPI + cells. Twenty-four specimens were available for evaluation (tissue quality was heavily deteriorated precluding analysis in 3 samples). Nuclear segmentation parameters were set using DAPI as the reference channel, then an analysis mask was overlayed on the image allowing the software to locate and segment each cell. Using a combination of the analysis mask and the scanned slide image, minimum intensity thresholds are set for each channel. Once the thresholds are set the analysis outputs a percentage of positive cells based on the number of cells positive for a specific phenotype and the number of DAPI + cells. All antibodies were diluted in Antibody diluent/block (ARD1001EA, Akoya Biosciences, Marlborough, MA, US). The list of antibodies and their dilution have been included in Supplementary File as Table S1.

IHC evaluation quantification

Each immune marker was quantified using Positive Pixel counting algorithm (Aperio, Leica Biosystems, IL, US)⁸⁶. Thresholding was performed on brown, pink, and total brown and pink areas, and total area was determined by hematoxylin-stained area. Integrated positive pixel area was used to compute the relative proportion of cells. Parameters for threshold, hue, and saturation were kept constant across all patient specimens.

Immunophenotyping

Freshly excised 3 mm punch biopsies from 3 BCC lesions were collected in DMEM media. Tissue were transported to cell culture lab on ice and stored in DMEM media for 24–48 h at 4 °C. Cell suspensions were generated according to the following protocol⁸⁷. Cells were processed for surface labeling with anti-CD3, anti-CD45, anti-CD4, and anti-CD8 antibodies. Live cells are distinguished from dead cells by using the fixable dye eFluor506 (eBioscience, Thermo Fisher Scientific, MA, USA). They were further permeabilized using a FOXP3 fixation and permeabilization kit (eBioscience, Thermo Fisher Scientific, MA, USA) and stained for Ki-67, FOXP3, and granzymeB. Data were acquired using the Aurora Five Laser flow cytometer (Cytek Biosciences, CA, US). Data were analyzed with FlowJo software version 10.5.3. (Tree Star Inc. OR, USA)⁸⁸.

RCM feature quantification

RCM data

Individual images (0.75 × 0.75 mm) from stacks and single FOV frames from videos were used for automated quantification of immune cells, and vasculature, respectively. Machine learning-based immune cell quantification was explored on 1026 frames from 93 lesions (skin cancers and rashes) in 74 patients (34-females, 40-males; age range 30–88 years). Each case contributed 5-27 independent images. For vascular feature quantification, 438 single FOV videos (39, 813 frames). Each lesion contributed 1-31 videos.

Machine learning for immune cells

A pixel-wise segmentation model was trained for 4 important morphological patterns (dendritic cells, macrophages, leukocytic round-ellipsoid cells and miscellaneous immune cells). We binned them into 2 classes as Class 1: Dendritic cells and macrophages, Class 2: round-ellipsoid leukocyte-like cells. We labelled a third class called background comprising of areas that did not contain any of the immune cell patterns. A total of 1026 RCM images from 92 lesions were labelled pixelwise for these 3 classes in a non-exhaustive manner, where we only labelled examples of these patterns (Fig. S6a). A total of 9% of the pixels were labelled (6% Class 1, 3% Class 2), 91% was used as Class 3. We trained a 3 class UNet⁸⁹segmentation model using the MONAI framework⁹⁰. We used 926 images for training and 100 independent images for testing the model. Based on our former studies^59,91, we downsampled the RCM images to 256 by 256 pixels (corresponding to 2 µm resolution) for the sake of computational efficiency. We used a learning rate of 5e-2, batch size of 64, and SGD optimizer with Nesterov momentum. We also used image augmentation such as random rotation, flipping, elastic-affine deformation, intensity scaling, to increase the training dataset size. The model is trained for 90 epoch using DICE loss. After 90 epochs we did not see any improvement in the loss. We found a Dice similarity coefficient of 0.72 for these 3 classes (Table 2).

Vascular features

For vascular feature quantification, we used 438 single FOV videos (39, 813 frames) from 48 cases. For all video frames, we used a two-step image stabilization procedure to remove the significant motion found in each movie segment. Firstly, a linear pre-alignment is performed to minimize large-scale motion in FIJI⁹² using the SIFT feature plugin Plugins->Registration-> Linear Stack Alignment with SIFT and default parameters. Stabilized images are then automatically cropped in MATLAB (mathworks.com) to remove black background and include only areas within the FOV during the entirety of the movie segment. The crop rectangle is computed automatically by iteratively removing the row or column of pixels which contains the most blank pixels in a temporal min image until all outer edge rows and columns that contain more than three-quarters blank pixels are removed. We then performed a second custom nonlinear stabilization in MATLAB to remove large-scale tissue deformations over time. Frame t + 1 first has its histogram equalized to match frame t and then is aligned to frame t using the imregdemon procedure with four pyramid levels and steeply decreasing iterations of alignment at successively finer scales (iterations, [100,50,10,1]). Frame t + 2 is then aligned with the transformed frame t + 1 and so on. Cropping of all regions not in view throughout the movie was performed again via the same procedure.

Blood vessel segmentation

We performed manual segmentation of blood vessels using an open-source segmentation platform called 3D Splicer (https://www.slicer.org/)⁹³ on 25 randomly selected videos. Two videos were discarded from analysis due to extreme Z-motion. We processed the remaining 23 videos to display only every 10th frame to mimic the automated segmentation approach; each frame in the resulting file was manually segmented for blood vessels. We exported the entire video segmentation as a Nifti (.nii) file format and imported into MATLAB as a 3D image array, where consecutive images in the array correspond to consecutive frames in the RCM video. Ensuring that the consecutive frames are registered, our assumption for detecting the vessels was that the areas of high variation between consecutive frames correspond to vessels. In order to suppress the variation due to speckle noise in the RCM images, we first applied a gaussian smoothing filter (sigma = 1px). Then we applied a finite impulse response high pass filter (F = [0.5,−1,0.5]) and smooth out the extracted pixel-wise variation in time using a 7-by-7 median filter. We then subtracted the mean variation of each frame to eliminate the slowly varying areas, and obtain a variation map for the whole video by accumulating the variation over the entire video sequence. We finally applied otsu thresholding on the final variation map to locate the areas with vessels. To smooth the border of the vessels and clean out the noise in the segmentation, we applied morphological closing operation on the binary segmentation map and clean segmented areas smaller than 0.1% and larger 10% of the entire frame. Finally, we calculated Dice similarity coefficients by comparing manual and automated vessel segmentation (Fig. S6b, Table 2).

Trafficking

Background subtraction

We estimated a background image for each frame as the median per pixel over a temporal window of 6 s centered on the current frame. Where movie temporal resolution differed, we adjusted the window in frames accordingly. This background estimate is subtracted out of the current frame, largely isolating moving cells on a dark background. We also tested alternative common methods for background subtraction, including sparse linear factorization methods as well as mean and min background estimates and dividing through by, rather than subtracting, background estimates, which desirably enhanced dim cells. None of these methods were found to provide satisfactory results on our data sets.

Tracking

We exported background subtracted images from MATLAB as 32-bit OME tiffs and imported into FIJI. Tracking is then performed in Trackmate⁹⁴ using DoG spot detection (subpixel = true; radius = 7.5pixels (7.5/1.33 = 5.63 µm); threshold = 1.6) and the LAP tracker with no splitting, merging or gap closing, and a max match distance of 20 pixels (20/1.33 = 15.03 µm). The tracklets found are then filtered in MATLAB to remove spurious tracklets corresponding to imperfectly removed background elements (this occurs particularly during changes in z during imaging) or tracks strung together from different fast moving circulating blood cells while preserving the desired target population. Features used to measure tracklet desirability are detailed below. We set the thresholds quantitatively and automatically to maximize correspondence between automated results and manual counts on an initial training set of 40 movies (approx. 10% of overall data). We investigated three different temporal windows ranging from 3-5 frames (0.6 s, 0.8 s and 1 s) for total quantification of rolling, crawling and adherent cells. We adopted constrained optimization within a restricted range, although we also investigated fully independent threshold optimization (Fig. S6c). We found moderate-high correlation (0.79–0.82) during first optimization ((Table 2) following which trafficking was quantified on remaining videos. Final validation using manual counts on a subset of videos (~2.5% of total data) by two readers, we found high inter-reader concordance and high correlation (0.74–0.0.89) for different temporal windows (Fig. S6d, e). We selected temporal window 3 (0.6 s) for subsequent analysis to ensure inclusion of especially faster trafficking processes (rolling cells) in shorter blood vessels. The correlation was worse for videos with remnant motion after two-step motion minimization strategy, suggesting need for minimizing axial and lateral motion during data acquisition, and use of more efficient motion removal algorithms in future. The existing code can be found on Github and zenodo⁹⁵.

The Tracklet Parameters used are as follows:

Displacement = [15.41,16.92,16.92] µm or [20.5,22.5,22.5] pixels
Consistency = [58,58,58] degrees
Quality = [1.6,1.65,1.75] arbitrary units
Length = [0.6 s, 0.8 s or 1 s]

where,

Displacement – total displacement between tracklet start and end point, in pixels (tracklets with lower displacement are discarded)

Motion Consistency – average angle between the motion vector of the track at successive timepoints in degrees (tracklets with higher angular difference are discarded)

Quality – average quality of detections making up the tracklet as measured by Trackmate (lower average quality tracklets are discarded)

Length – duration in s of tracklet, in all cases this was set to the thresholds used in manual counts (shorter tracklets are discarded).

Response to Immunotherapy analysis

We analyzed correlation of TiME features and phenotypes with response to topical TLRA imiquimod on 13 lesions on 8 patients (4 females, 4 males; age range 54–70 years). The patients were imaged at baseline (T0) and end of treatment. We performed HCPC clustering in R to identify clustering patterns based on response. To assign phenotypes, we projected these patients on the original BCC PCA model. Towards developing quantitative models for response prediction, we first performed linear regression modeling to quantitatively identify the predictor variables for response to imiquimod, and compared against the known “standard” which is tumor-infiltrating lymphocyte-like cells and intratumoral inflammation. In order to measure the predictive power of each feature, we trained predictive models in a leave-one-out cross-validation fashion and measured the model performance by inferring on the left-out test sample (out-of-bag estimates). This procedure was followed in an iterative manner, where we selected a single feature that gives the highest performance and added a new feature that provided the highest performance in each iteration. Model performance was measured by calculating specificity (the higher, the better) on the out-of-bag estimates and Akaike Information Criterion (the lower, the better) value of the model. In this way, the features were prioritized according to their predictive power. The best performance among the AIC prioritizing models was 85% sensitivity, 66% specificity and AIC = −15.06 with 8 variables while the best performance among the specificity prioritizing model was 71% sensitivity, 83% specificity and AIC = −15.06 with 13 variables. Moreover, we also examined the linear separability of (i) individual features by looking at the histogram of feature values for each sample, and (ii) each pairwise feature combination by examining kernel density estimation plots.

Statistical analysis

Data was collected using prospective patient imaging, no statistical method was used to predetermine sample size. Data from 1 patient was excluded due to motion blur during image acquisition during TiME phenotyping and all subsequent analyses. Being an observational, non-intervention study, the samples were not randomized. Further, the investigators were not blinded to allocation during experiments and outcome assessment. To elucidate agreements between two readers’ manual evaluations for RCM features, Cohen’s kappa coefficients were computed. The agreement regarding the extent of each feature presence between RCM and histology was quantified using linearly weighted Gwet’s AC1. Unsupervised statistical clustering for phenotyping was performed using hierarchical clustering on principal components (HCPC) and principal component analysis (PCA) using PCs with largest eigenvalues that explained at least 95% of total variance. Mean, median and confidence intervals were computed for the multiplexed IHC/IF data analyzed and significance was estimated using either two-tailed unpaired Mann–Whitney test or Kruskal-Wallis test corrected for multiple group comparisons by Dunn’s method. Differentially expressed transcripts between RCM Inflam^HIGH and Inflam^LOW groups were determined using pairwise comparisons in edgeR performing quasi-likelihood F-test (FDR < 0.05) on TMM-normalized read counts with filtering to remove lowly expressed transcripts (using filterByExpr argument in edgeR). Co-Expression Modules identification Tool (CEMiTool) was used to identify and analyze gene co-expression modules using default parameters after assessing normal distribution of TMM normalized and log₂ transformed transcripts across samples (correlation method = Pearson, R² > 0.8, filtering pvalue = 0.1). Two-tailed Mann–Whitney tests were used to evaluate p-value for CIBERSORTx immune cell proportions, and prevalence of TiME features across responders and non-responders. Spearman correlation was used to estimate correlation between quantified RCM features and immune-related, trafficking-related and vascular-related genes. Linear regression modeling and leave-one-out estimates were used for developing prediction models of treatment response. Model performance was evaluated by specificity and Akaike information criterion.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Data availability

The RNA-expression datasets generated during this study have been made publicly available on Gene Expression Omnibus (GEO ID: GSE181037). Publicly available datasets used in this study include MsigDB [https://www.gsea-msigdb.org/gsea/msigdb/], gene ontology [http://geneontology.org/], GTex [https://gtexportal.org/home/] and OMIM repository [https://www.ncbi.nlm.nih.gov/omim]. De-identified raw images will be freely available upon request. Requests will be considered for 10 years after the publication of this article. As per MSKCC guidelines, a data sharing agreement and ethical permissions will have to be necessarily set up with requesting colleagues and their institutions. Source data are provided with this paper.

Code availability

Custom image and data analysis scripts for FIJI and MATLAB developed for quantification of imaging data are available on GitHub [https://github.com/mskccmccf/TiME-analysis] or Zenodo [https://doi.org/10.5281/zenodo.6712717] and the RNA-seq analysis pipeline is available on GitHub [https://github.com/mskccmccf/TiME-analysis/tree/main/RNA_Seq_Analysis]. Other publicly available R packages used for the gene expression and statistical analysis have been cited throughout the manuscript.

References

Topalian, S. L., Drake, C. G. & Pardoll, D. M. Immune checkpoint blockade: a common denominator approach to cancer therapy. Cancer Cell. 27, 450–461 (2015).
Article CAS PubMed PubMed Central Google Scholar
Wolchok, J. D. et al. Overall survival with combined nivolumab and ipilimumab in advanced melanoma. N. Engl. J. Med. 377, 1345–1356 (2017).
Article CAS PubMed PubMed Central Google Scholar
Ribas, A. & Wolchok, J. D. Cancer immunotherapy using checkpoint blockade. Science 359, 1350–1355 (2018).
Article ADS CAS PubMed PubMed Central Google Scholar
Darvin, P., Toor, S. M., Nair, V. S. & Elkord, E. Immune checkpoint inhibitors: recent progress and potential biomarkers. Exp. Mol. Med. 50, 1–11 (2018).
Article PubMed CAS Google Scholar
Friedman, C. F. & Postow, M. A. Emerging tissue and blood-based biomarkers that may predict response to immune checkpoint inhibition. Curr. Oncol. Rep. 18, 21 (2016).
Article PubMed PubMed Central CAS Google Scholar
Sharma, P., Hu-Lieskovan, S., Wargo, J. A. & Ribas, A. Primary, Adaptive, and Acquired Resistance to Cancer Immunotherapy. Cell 168, 707–723 (2017).
Article CAS PubMed PubMed Central Google Scholar
Galon, J. & Bruni, D. Approaches to treat immune hot, altered and cold tumours with combination immunotherapies. Nat. Rev. Drug Discov. 18, 197–218 (2019).
Article CAS PubMed Google Scholar
Filipovic, A., Miller, G. & Bolen, J. Progress Toward Identifying Exact Proxies for Predicting Response to Immunotherapies. Front. Cell Dev. Biol. 8, 155 (2020).
Article PubMed PubMed Central Google Scholar
Fridman, W. H., Zitvogel, L., Sautès–Fridman, C. & Kroemer, G. The immune contexture in cancer prognosis and treatment. Nat. Rev. Clin. Oncol. 14, 717 (2017).
Article CAS PubMed Google Scholar
Yang, Y., Li, C., Liu, T., Dai, X. & Bazhin, A. V. Myeloid-derived suppressor cells in tumors: from mechanisms to antigen specificity and microenvironmental regulation. Front. Immunol. 11, 1371 (2020).
Article CAS PubMed PubMed Central Google Scholar
Chen, D. S. & Mellman, I. Elements of cancer immunity and the cancer–immune set point. Nature 541, 321–330 (2017).
Article ADS CAS PubMed Google Scholar
De Guillebon, E. et al. Beyond the concept of cold and hot tumors for the development of novel predictive biomarkers and the rational design of immunotherapy combination. Int. J. Cancer. 147, 1509–1518 (2020).
Article PubMed CAS Google Scholar
Paijens, S. T., Vledder, A., de Bruyn, M. & Nijman, H. W. Tumor-infiltrating lymphocytes in the immunotherapy era. Cell. Mol. Immunol. 18, 842–859 (2021).
Tang, T. et al. Advantages of targeting the tumor immune microenvironment over blocking immune checkpoint in cancer immunotherapy. Signal Transduct. Target. Ther. 6, 1–13 (2021).
CAS Google Scholar
Schaaf, M. B., Garg, A. D. & Agostinis, P. Defining the role of the tumor vasculature in antitumor immunity and immunotherapy. Cell Death Dis. 9, 115 (2018).
Article PubMed PubMed Central CAS Google Scholar
Hendry, S. A. et al. The role of the tumor vasculature in the host immune response: implications for therapeutic strategies targeting the tumor microenvironment. Front. Immunol. 7, 621 (2016).
Article PubMed PubMed Central CAS Google Scholar
Klein, D. The tumor vascular endothelium as decision maker in cancer therapy. Front. Oncol. 8, 367 (2018).
Article ADS PubMed PubMed Central Google Scholar
Motz, G. T. & Coukos, G. The parallel lives of angiogenesis and immunosuppression: cancer and other tales. Nat. Rev. Immunol. 11, 702–711 (2011).
Article CAS PubMed Google Scholar
Huinen, Z.R., Huijbers, E.J.M., van Beijnum, J.R., Nowak-Sliwinska, P. & Griffioen, A.W. Anti-angiogenic agents — overcoming tumour endothelial cell anergy and improving immunotherapy outcomes. Nat. Rev. Clin. Oncol. 18, 527–540 (2021).
Fisher, D. T. et al. Intraoperative intravital microscopy permits the study of human tumour vessels. Nat. Commun. 7, 1–9 (2016).
Article Google Scholar
Rajadhyaksha, M., Grossman, M., Esterowitz, D., Webb, R. H. & Anderson, R. R. In vivo confocal scanning laser microscopy of human skin: melanin provides strong contrast. J. Invest. Dermatol. 104, 946–952 (1995).
Article CAS PubMed Google Scholar
Rajadhyaksha, M., Anderson, R. R. & Webb, R. H. Video-rate confocal scanning laser microscope for imaging human tissues in vivo. Appl. Opt. 38, 2105–2115 (1999).
Article ADS CAS PubMed Google Scholar
Rajadhyaksha, M., Marghoob, A., Rossi, A., Halpern, A. C. & Nehal, K. S. Reflectance confocal microscopy of skin in vivo: From bench to bedside. Lasers Surg. Med. 49, 7–19 (2017).
Article PubMed Google Scholar
Dinnes, J. et al. Cochrane Skin Cancer Diagnostic Test Accuracy Group. Reflectance confocal microscopy for diagnosing keratinocyte skin cancers in adults. Cochrane Database Syst. Rev. 12, CD013191 (2018).
González, S., Sackstein, R., Anderson, R. R. & Rajadhyaksha, M. Real-time evidence of in vivo leukocyte trafficking in human skin by reflectance confocal microscopy. J. Investigative Dermatol. 117, 384 (2001).
Article Google Scholar
Ahlgrimm-Siess, V. et al. The vasculature of nonmelanocytic skin tumors in reflectance confocal microscopy: vascular features of basal cell carcinoma. Arch. Dermatol. 146, 353 (2010).
PubMed Google Scholar
Saknite, I. et al. Individual cell motion in healthy human skin microvasculature by reflectance confocal video microscopy. Microcirculation 27, e12621 (2020).
Article CAS PubMed PubMed Central Google Scholar
Russo, P. S. et al. CEMiTool: a Bioconductor package for performing comprehensive modular co-expression analyses. BMC Bioinforma. 19, 1–13 (2018).
Article CAS Google Scholar
Lin, C.-X., Li, H.-D., Deng, C., Guan, Y. & Wang, J. TissueNexus: a database of human tissue functional gene networks built with a large compendium of curated RNA-seq data. Nucl. Acids Res. 50, D710–D718 (2022).
Article CAS PubMed Google Scholar
Steen, C.B., Liu, C.L., Alizadeh, A.A. & Newman, A.M. Profiling cell type abundance and expression in bulk tissues with CIBERSORTx. in Stem Cell Transcriptional Networks 135–157 (Springer, 2020).
Cabrita, R. et al. Tertiary lymphoid structures improve immunotherapy and survival in melanoma. Nature 577, 561–565 (2020).
Article ADS CAS PubMed Google Scholar
Sautès-Fridman, C., Petitprez, F., Calderaro, J. & Fridman, W. H. Tertiary lymphoid structures in the era of cancer immunotherapy. Nat. Rev. Cancer 19, 307–325 (2019).
Article PubMed CAS Google Scholar
Dice, L. R. Measures of the Amount of Ecologic Association Between Species. Ecology 26, 297–302 (1945).
Article Google Scholar
Paulson, K. G., Lahman, M. C., Chapuis, A. G. & Brownell, I. Immunotherapy for skin cancer. Int. Immunol. 31, 465–475 (2019).
Article CAS PubMed PubMed Central Google Scholar
Argenziano, G. et al. Vascular structures in skin tumors: a dermoscopy study. Arch. Dermatol. 140, 1485–1489 (2004).
Article PubMed Google Scholar
Hall, E. T., Fernandez-Lopez, E., Silk, A. W., Dummer, R. & Bhatia, S. Immunologic characteristics of nonmelanoma skin cancers: implications for immunotherapy. Am. Soc. Clin. Oncol. Educ. Book. 40, 398–407 (2020).
Article Google Scholar
Huang, Y. et al. Improving immune–vascular crosstalk for cancer immunotherapy. Nat. Rev. Immunol. 18, 195–203 (2018).
Article CAS PubMed PubMed Central Google Scholar
Spranger, S., Bao, R. & Gajewski, T. F. Melanoma-intrinsic β-catenin signalling prevents anti-tumour immunity. Nature 523, 231–235 (2015).
Article ADS CAS PubMed Google Scholar
Yang, L., Li, A., Lei, Q. & Zhang, Y. Tumor-intrinsic signaling pathways: key roles in the regulation of the immunosuppressive tumor microenvironment. J. Hematol. Oncol. 12, 1–14 (2019).
Article Google Scholar
Samstein, R. M. et al. Tumor mutational load predicts survival after immunotherapy across multiple cancer types. Nat. Genet. 51, 202–206 (2019).
Article CAS PubMed PubMed Central Google Scholar
Luke, J. J., Bao, R., Sweis, R. F., Spranger, S. & Gajewski, T. F. WNT/β-catenin pathway activation correlates with immune exclusion across human cancers. Clin. Cancer Res. 25, 3074–3083 (2019).
Article CAS PubMed PubMed Central Google Scholar
Grasso, C. S. et al. Genetic mechanisms of immune evasion in colorectal cancer. Cancer Discov. 8, 730–749 (2018).
Article CAS PubMed PubMed Central Google Scholar
Kis-Toth, K. et al. Voltage-gated sodium channel Nav1. 7 maintains the membrane potential and regulates the activation and chemokine-induced migration of a monocyte-derived dendritic cell subset. J. Immunol. 187, 1273–1280 (2011).
Article CAS PubMed Google Scholar
Sasai, M. et al. Uncovering a novel role of PLCβ4 in selectively mediating TCR signaling in CD8+ but not CD4+ T cells. J. Exp. Med. 218, e20201763 (2021).
Article CAS PubMed PubMed Central Google Scholar
Jiang, H. et al. Phospholipase C β4 is involved in modulating the visual response in mice. Proc. Natl Acad. Sci. 93, 14598–14601 (1996).
Article ADS CAS PubMed PubMed Central Google Scholar
Imai, T. et al. Epigenetic inactivation of RASSF2 in oral squamous cell carcinoma. Cancer Sci. 99, 958–966 (2008).
Article CAS PubMed Google Scholar
Bath-Hextall, F. et al. Surgical excision versus imiquimod 5% cream for nodular and superficial basal-cell carcinoma (SINS): a multicentre, non-inferiority, randomised controlled trial. lancet Oncol. 15, 96–105 (2014).
Article CAS PubMed Google Scholar
Papanikolaou, M. & Lawrence, C. Long‐term outcomes of imiquimod‐treated lentigo maligna. Clin. Exp. Dermatol. 44, 631–636 (2019).
Article CAS PubMed Google Scholar
Urech, M. et al. Dermoscopic ulceration is a predictor of basal cell carcinoma response to imiquimod: a retrospective study. Acta. Derm. Venereol. 97, 117–119 (2017).
Halse, H. et al. A distinct pretreatment immune gene signature in lentigo maligna is associated with imiquimod response. J. Invest. Dermatol. 140, 869–877.e816 (2020).
Article CAS PubMed Google Scholar
Gautschi, M. et al. Prognostic markers in lentigo maligna patients treated with imiquimod cream: a long-term follow-up study. J. Am. Acad. Dermatol. 74, 81–87.e81 (2016).
Article CAS PubMed Google Scholar
Lucena, S. R. et al. Combined treatments with photodynamic therapy for non-melanoma skin cancer. Int. J. Mol. Sci. 16, 25912–25933 (2015).
Article CAS PubMed PubMed Central Google Scholar
Ren, L. et al. Hypoxia-induced CCL28 promotes recruitment of regulatory T cells and tumor growth in liver cancer. Oncotarget 7, 75763 (2016).
Article PubMed PubMed Central Google Scholar
Cheng, D. T. et al. Memorial Sloan Kettering-Integrated Mutation Profiling of Actionable Cancer Targets (MSK-IMPACT): a hybridization capture-based next-generation sequencing clinical assay for solid tumor molecular oncology. J. Mol. Diagnostics. 17, 251–264 (2015).
Article CAS Google Scholar
Farahani, K., Tata, D. & Nordstrom, R. J. QIN benchmarks for clinical translation of quantitative imaging tools. Tomography 5, 1–6 (2019).
Article PubMed PubMed Central Google Scholar
Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44–56 (2019).
Article CAS PubMed Google Scholar
Vestweber, D. How leukocytes cross the vascular endothelium. Nat. Rev. Immunol. 15, 692–704 (2015).
Article CAS PubMed Google Scholar
Bozkurt, A. et al. Skin strata delineation in reflectance confocal microscopy images using recurrent convolutional networks with attention. Sci. Rep. 11, 1–11 (2021).
Article CAS Google Scholar
Kose, K. et al. Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via a multiscale encoder-decoder network (MED-Net). Med. Image Anal. 67, 101841 (2021).
Article PubMed Google Scholar
Sahu, A. et al. Evaluation of a Combined Reflectance Confocal Microscopy–Optical Coherence Tomography Device for Detection and Depth Assessment of Basal Cell Carcinoma. JAMA Dermatol. 154, 1175–1183 (2018).
Article PubMed PubMed Central Google Scholar
Schuh, S. et al. Imaging blood vessel morphology in skin: dynamic optical coherence tomography as a novel potential diagnostic tool in dermatology. Dermatol. Ther. 7, 187–202 (2017).
Article Google Scholar
Vakoc, B. J. et al. Three-dimensional microscopy of the tumor microenvironment in vivo using optical frequency domain imaging. Nat. Med. 15, 1219–1223 (2009).
Article CAS PubMed PubMed Central Google Scholar
Lin, R. et al. Longitudinal label-free optical-resolution photoacoustic microscopy of tumor angiogenesis in vivo. Quant. Imaging Med. Surg. 5, 23 (2015).
PubMed PubMed Central Google Scholar
Brown, E. B. et al. In vivo measurement of gene expression, angiogenesis and physiological function in tumors using multiphoton laser scanning microscopy. Nat. Med. 7, 864–868 (2001).
Article CAS PubMed Google Scholar
Wu, P.-C., Hsieh, T.-Y., Tsai, Z.-U. & Liu, T.-M. In vivo Quantification of the Structural Changes of Collagens in a Melanoma Microenvironment with Second and Third Harmonic Generation Microscopy. Sci. Rep. 5, 8879 (2015).
Article CAS PubMed PubMed Central Google Scholar
Brown, E., Brunker, J. & Bohndiek, S. E. Photoacoustic imaging as a tool to probe the tumour microenvironment. Dis. Model. Mech. 12, dmm039636 (2019).
Walsh, A. J. et al. Classification of T-cell activation via autofluorescence lifetime imaging. Nat. Biomed. Eng. 5, 77–88 (2021).
Navarrete-Dechent, C. et al. Optical imaging guided-‘precision’biopsy of skin tumors: a novel approach for targeted sampling and histopathologic correlation. Arch. Dermatol. Res. 313, 517–529 (2021).
Mantri, M. et al. Spatiotemporal single-cell RNA sequencing of developing chicken hearts identifies interplay between cellular differentiation and morphogenesis. Nat. Commun. 12, 1–13 (2021).
Article ADS CAS Google Scholar
Hunter, M.V., Moncada, R., Weiss, J.M., Yanai, I. & White, R.M. Spatially resolved transcriptomics reveals the architecture of the tumor/microenvironment interface. Nat. Commun. 12, 6278 (2021).
Sahu, A. et al. In vivo optical imaging-guided targeted sampling for precise diagnosis and molecular pathology. Sci. Rep. 11, 1–9 (2021).
Article ADS CAS Google Scholar
Lê, S., Josse, J. & Husson, F. FactoMineR: an R package for multivariate analysis. J. Stat. Softw. 25, 1–18 (2008).
Article Google Scholar
Kassambara, A. & Mundt, F. Factoextra: extract and visualize the results of multivariate data analyses. R. Package Version. 1, 337–354 (2017).
Google Scholar
Wickham, H., Chang, W. & Wickham, M. H. Package ‘ggplot2’. Creat. Elegant Data visualizations Using Gramm. Graph. Version 2, 1–189 (2016).
MATH Google Scholar
Chen, Y., Lun, A.T. & Smyth, G.K. From reads to genes to pathways: differential expression analysis of RNA-Seq experiments using Rsubread and the edgeR quasi-likelihood pipeline. F1000Research. 5,1438(2016).
McCarthy, D. J., Chen, Y. & Smyth, G. K. Differential expression analysis of multifactor RNA-Seq experiments with respect to biological variation. Nucl. Acids Res. 40, 4288–4297 (2012).
Article CAS PubMed PubMed Central Google Scholar
Robinson, M. D., McCarthy, D. J. & Smyth, G. K. edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics 26, 139–140 (2010).
Article CAS PubMed Google Scholar
Xie, Z. et al. Gene set knowledge discovery with enrichr. Curr. Protoc. 1, e90 (2021).
Article CAS PubMed PubMed Central Google Scholar
Kuleshov, M. V. et al. Enrichr: a comprehensive gene set enrichment analysis web server 2016 update. Nucl. Acids Res. 44, W90–W97 (2016).
Article CAS PubMed PubMed Central Google Scholar
Chen, E. Y. et al. Enrichr: interactive and collaborative HTML5 gene list enrichment analysis tool. BMC Bioinforma. 14, 1–14 (2013).
Article Google Scholar
Liberzon, A. et al. The Molecular Signatures Database (MSigDB) hallmark gene set collection. Cell Syst. 1, 417–425 (2015).
Article CAS PubMed PubMed Central Google Scholar
Subramanian, A. et al. Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proc. Natl Acad. Sci. 102, 15545–15550 (2005).
Article ADS CAS PubMed PubMed Central Google Scholar
Consortium, G. The GTEx Consortium atlas of genetic regulatory effects across human tissues. Science 369, 1318–1330 (2020).
Article CAS Google Scholar
Harell Jr, F. Hmisc: Harrell Miscellaneous. R package version 4.6–0. (2021).
Newman, A. M. et al. Robust enumeration of cell subsets from tissue expression profiles. Nat. Methods 12, 453–457 (2015).
Article CAS PubMed PubMed Central Google Scholar
Olson, A.H. Image analysis using the Aperio ScanScope. Technical manual. Aperio Technologies Inc. 1–13 (2006).
Zamarin, D. et al. Localized oncolytic virotherapy overcomes systemic tumor resistance to immune checkpoint blockade immunotherapy. Sci. Transl. Med. 6, 226ra232–226ra232 (2014).
Article CAS Google Scholar
Dai, P. et al. Intratumoral delivery of inactivated modified vaccinia virus Ankara (iMVA) induces systemic antitumor immunity via STING and Batf3-dependent dendritic cells. Sci. Immunol. 2, eaal1713 (2017).
Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. 234–241 (Springer International Publishing, Cham, 2015).
1.The MONAI Consortium. Project MONAI. https://doi.org/10.5281/zenodo.4323059 (2020).
D’Alonzo, M. et al. Semantic segmentation of reflectance confocal microscopy mosaics of pigmented lesions using weak labels. Sci. Rep. 11, 1–13 (2021).
Article CAS Google Scholar
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods. 9, 676–682 (2012).
Article CAS PubMed Google Scholar
Fedorov, A. et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging. 30, 1323–1341 (2012).
Article PubMed PubMed Central Google Scholar
Tinevez, J.-Y. et al. TrackMate: An open and extensible platform for single-particle tracking. Methods 115, 80–90 (2017).
Article CAS PubMed Google Scholar
Anthony Santella, K.K. mskccmccf/TiME-analysis for vessels and immune cells. (Zenodo, 2022).

Download references

Acknowledgements

We would like to acknowledge Dr. Anjali Rajadhyaksha for assistance with experimental planning and scientific discussion. We would also like to acknowledge Dr. Sanjee Abeytunge, Dr. Veronica Rotemberg and Dr. Mario Lacouture for critical feedback. We also acknowledge MSKCC Cores: Integrated Genomics Operation Core, funded by the NCI Cancer Center Support Grant (CCSG, P30 CA08748), Cycle for Survival, and the Marie-Josée and Henry R. Kravis Center for Molecular Oncology, Molecular Cytology Core Facility, Pathology Core, Flow Cytometry Core. In addition, Ms. Cassidy Cobbs, Ms. Marina Asher, Mr. Afsar Barlas, Mr. Eric Chan, Mr. Stephen Wilson and Mr. Reza Afzalneia for assistance with experimental planning, data analysis, tissue processing and immunohistochemical staining. The authors would also like to acknowledge funding support from the following centers:

NIH/NCI Cancer Center Support Grant P30 CA008748 (MSKCC)

NIH/NIBIB R01EB028752 and NIH/NCI R01240771 (MR)

Melanoma Research Alliance (AS) (https://doi.org/10.48050/pc.gr.141697, https://doi.org/10.48050/pc.gr.89583)

The Chan-Zuckerberg Initiative (Anthony S)

Swiss National Science Foundation (LK)

Roux Institute and Harold Alfond Foundation (CB)

Ludwig Institute for Cancer Research, Sephora, Hazen Polsky Foundation, the Parker Institute for Cancer Immunotherapy and Swim Across America (Merghoub/Wolchok Lab).

Author information

Authors and Affiliations

Memorial Sloan Kettering Cancer Center, New York, NY, USA
Aditi Sahu, Kivanc Kose, Teguru Tembo, Anthony Santella, Anabel Alfonso, Madison Li, Miguel Cordova, Nicholas Kurtansky, Shen Yin, Paras Mehta, Cristian Navarrete-Dechent, Gary Peterson, Kimeil King, Stephen Dusza, Ning Yang, Shuaitong Liu, William Phillips, Anthony Rossi, Allan Halpern, Liang Deng, Melissa Pulitzer, Ashfaq Marghoob, Chih-Shan Jason Chen, Taha Merghoub & Milind Rajadhyaksha
Parker Institute for Cancer Immunotherapy, Ludwig Collaborative and Swim Across America Laboratory, Human Oncology and Pathogenesis Program, Memorial Sloan Kettering Cancer Center, New York, NY, USA
Lukas Kraehenbuehl, Aliya Holland & Taha Merghoub
Roux Institute, Northeastern University, Portland, ME, USA
Candice Byers
Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA
Candice Byers
SUNY Downstate Health Sciences University, Brooklyn, NY, USA
Teguru Tembo & Melissa Gill
Department of Clinical Pathology and Cancer Diagnostics, Karolinska University Hospital Solna, Stockholm, Sweden
Melissa Gill
Faculty of Medicine and Health Sciences, University of Alcala, Madrid, Spain
Melissa Gill & Salvador Gonzalez
Caliber Imaging and Diagnostics, Rochester, NY, USA
Christi Fox
Icahn School of Medicine at Mount Sinai, New York, NY, USA
Piyush Kumar
ORIC Pharmaceuticals, San Francisco, USA
Amber Weiching Wang
Tata Memorial Hospital, Mumbai, India
Pratik Chandrani
Pontificia Universidad Católica de Chile, Santiago, Chile
Cristian Navarrete-Dechent
Sydney Melanoma Diagnostic Center, Sydney, NSW, Australia
Pascale Guitera
Melanoma Institute Australia, Wollstonecraft, NSW, Australia
Pascale Guitera
Weill Cornell Medicine, New York, NY, USA
Liang Deng & Taha Merghoub

Authors

Aditi Sahu
View author publications
You can also search for this author in PubMed Google Scholar
Kivanc Kose
View author publications
You can also search for this author in PubMed Google Scholar
Lukas Kraehenbuehl
View author publications
You can also search for this author in PubMed Google Scholar
Candice Byers
View author publications
You can also search for this author in PubMed Google Scholar
Aliya Holland
View author publications
You can also search for this author in PubMed Google Scholar
Teguru Tembo
View author publications
You can also search for this author in PubMed Google Scholar
Anthony Santella
View author publications
You can also search for this author in PubMed Google Scholar
Anabel Alfonso
View author publications
You can also search for this author in PubMed Google Scholar
Madison Li
View author publications
You can also search for this author in PubMed Google Scholar
Miguel Cordova
View author publications
You can also search for this author in PubMed Google Scholar
Melissa Gill
View author publications
You can also search for this author in PubMed Google Scholar
Christi Fox
View author publications
You can also search for this author in PubMed Google Scholar
Salvador Gonzalez
View author publications
You can also search for this author in PubMed Google Scholar
Piyush Kumar
View author publications
You can also search for this author in PubMed Google Scholar
Amber Weiching Wang
View author publications
You can also search for this author in PubMed Google Scholar
Nicholas Kurtansky
View author publications
You can also search for this author in PubMed Google Scholar
Pratik Chandrani
View author publications
You can also search for this author in PubMed Google Scholar
Shen Yin
View author publications
You can also search for this author in PubMed Google Scholar
Paras Mehta
View author publications
You can also search for this author in PubMed Google Scholar
Cristian Navarrete-Dechent
View author publications
You can also search for this author in PubMed Google Scholar
Gary Peterson
View author publications
You can also search for this author in PubMed Google Scholar
Kimeil King
View author publications
You can also search for this author in PubMed Google Scholar
Stephen Dusza
View author publications
You can also search for this author in PubMed Google Scholar
Ning Yang
View author publications
You can also search for this author in PubMed Google Scholar
Shuaitong Liu
View author publications
You can also search for this author in PubMed Google Scholar
William Phillips
View author publications
You can also search for this author in PubMed Google Scholar
Pascale Guitera
View author publications
You can also search for this author in PubMed Google Scholar
Anthony Rossi
View author publications
You can also search for this author in PubMed Google Scholar
Allan Halpern
View author publications
You can also search for this author in PubMed Google Scholar
Liang Deng
View author publications
You can also search for this author in PubMed Google Scholar
Melissa Pulitzer
View author publications
You can also search for this author in PubMed Google Scholar
Ashfaq Marghoob
View author publications
You can also search for this author in PubMed Google Scholar
Chih-Shan Jason Chen
View author publications
You can also search for this author in PubMed Google Scholar
Taha Merghoub
View author publications
You can also search for this author in PubMed Google Scholar
Milind Rajadhyaksha
View author publications
You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: A.S., K.K., M.C., M.G., C.A.F., S.G., A.R., A.H., L.D., M.P., A.M., C.J.C., T.M., M.R. Methodology development: A.S., K.K., L.K., A.H., C.B., T.T., A.S., P.C., S.Y., P.M., M.C., C.N.D., S.D., C.J.C. Imaging and histopathology: A.S., M.C., C.N.D., P.K., M.G., C.A.F., S.G., G.P., K.K., P.G., M.P., W.P. Molecular analyses: A.S., L.K., C.B., A.H., T.T., N.Y., S.L., W.P. Data analysis and visualization: A.S., K.K., L.K., C.B., A.H., T.T., A.S., A.A., M.L., K.K., P.K., A.W.W., G.P., P.C., S.Y., P.M., N.K., S.D., N.Y., S.L. Funding acquisition: L.D., T.M., M.R. Supervision: M.C., M.G., C.A.F., S.G., A.H., L.D., P.G., A.R., A.M., M.P., C.J.C., T.M., M.R. Writing: A.S., K.K., L.K., C.B., A.H., A.S., A.A., M.L., A.W.W., N.K., S.D., A.M., C.J.C., M.R.

Corresponding authors

Correspondence to Aditi Sahu or Milind Rajadhyaksha.

Ethics declarations

Competing interests

M.G. is a consulting investigator for DBV technologies; research consultant: Dermatology Service, MSKCC. Christi Alessi-Fox: employee of and owns equity in Caliber I.D., manufacturer of the VivaScope RCM. Dr. Rossi: Mavig (travel accommodation), Merz, DynaMed, Canfield Scientific, Evolus, Biofrontera, QuantiaMD, Lam Therapeutics, Cutera (consultant); Allergan (advisory board). A.H. : consultant to Canfield Scientific and an advisory board member of Scibase. L.D. : cofounder and holds equity in IMVAQ Therapeutics, patents on applications related to work on oncolytic viral therapy (US 20220056475 A1: recombinant poxviruses for cancer immunotherapy; US 20180236062 A1: use of inactivated nonreplicating modified vaccinia virus ankara (mva) as monoimmunotherapy or in combination with immume checkpoint blocking agents for solid tumors). A. M.: honorarium for dermoscopy lectures (3GEN), royalties for books/book chapters, dermoscopy equipment for testing, payment for organizing and lecturing (American Dermoscopy Meeting). C-S.J.C. : research funding from Apollo Medical Optics, Inc. Milind Rajadhyaksha: was employee of and owns equity in Caliber I.D. VivaScope is the commercial version of a laboratory prototype he developed at Massachusetts General Hospital, Harvard Medical School. T.M. has acted as a consultant for Immunogenesis, Immunos Therapeutics, Daiichi Sankyo, Leap therapeutics and Pfizer, has received research support from Adaptive Biotechnologies, Aprea, Bristol Myers Squibb, Infinity Pharmaceuticals, Kyn Therapeutics, Leap Therapeutics, Peregrine Pharmaceuticals and Surface Oncology, is a cofounder of and holds an equity in IMVAQ Therapeutics and is listed as a co-inventor on patents relating to the use of oncolytic viral therapy, alphavirus-based vaccines, antibodies targeting CD40, GITR, OX40, PD-1 and CTLA-4 and neo-antigen modelling (US 20220056475 A1: recombinant poxviruses for cancer immunotherapy; US 20210179714 A1: Inhibition of CTLA-4 and/or PD-1 For Regulation of T Cells; US 20200232040 A1: neoantigens and uses thereof for treating cancer; US 20200113984 A1: Alphavirus Replicon Particles Expressing TRP2; US 20180236062 A1: use of inactivated nonreplicating modified vaccinia virus ankara (mva) as monoimmunotherapy or in combination with immume checkpoint blocking agents for solid tumors). The remaining authors declare no competing interests.

Peer review

Peer review information

Nature Communications thanks Agata Stanek, Zlatko Trajanoski and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Reporting Summary

Description of Additional Supplementary Files

Supplementary Movie 1

Supplementary Movie 2

Supplementary Movie 3

Peer Review File

Source data

Source Data

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Cite this article

Sahu, A., Kose, K., Kraehenbuehl, L. et al. In vivo tumor immune microenvironment phenotypes correlate with inflammation and vasculature to predict immunotherapy response. Nat Commun 13, 5312 (2022). https://doi.org/10.1038/s41467-022-32738-7

Download citation

Received: 09 July 2021
Accepted: 12 August 2022
Published: 09 September 2022
DOI: https://doi.org/10.1038/s41467-022-32738-7

This article is cited by

The macrophage-associated prognostic gene ANXA5 promotes immunotherapy resistance in gastric cancer through angiogenesis
- Zhijun Hong
- Peizhen Wen
- Huiqin Zhuo
BMC Cancer (2024)
Miniaturized microscope for non-invasive imaging of leukocyte-endothelial interaction in human microcirculation
- Arutyun Bagramyan
- Charles P. Lin
Scientific Reports (2023)

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Subjects

Abstract

Similar content being viewed by others

Introduction

Results

Immune and vascular features on RCM correlate with histopathology and demonstrate unique TiME phenotypes

Molecular landscape associates with InflamHIGH and VascHIGH signatures

InflamHIGH phenotypes correlate with abundant CD8+ and CD8+ PD1+ cells

Lymphocytes show strong association while tertiary lymphoid structures (TLS) show minimal association with TiME phenotypes

Melanoma TiME phenotypes correlate with histopathology and T-cell infiltrates

Automated quantification of TiME features is feasible

Treatment response to imiquimod can be predicted by combined immune and vascular features and TiME phenotyping

Discussion

Methods

Study design

In vivo imaging

Patient tissue

Manual RCM and histopathology evaluation

Statistical clustering for TiME phenotyping

RNA extraction

Transcriptome sequencing

Differential analysis of gene expression

CIBERSORTx analysis

Immunohistochemistry

Multiplexed immunofluorescence and analysis

IHC evaluation quantification

Immunophenotyping

RCM feature quantification

RCM data

Machine learning for immune cells

Vascular features

Blood vessel segmentation

Trafficking

Background subtraction

Tracking

Response to Immunotherapy analysis

Statistical analysis

Reporting summary

Data availability

Code availability

References

Acknowledgements

Author information

Authors and Affiliations

Contributions

Corresponding authors

Ethics declarations

Competing interests

Peer review

Peer review information

Additional information

Supplementary information

Source data

Rights and permissions

About this article

Cite this article

Share this article

This article is cited by

Comments

Search

Quick links

Molecular landscape associates with Inflam^HIGH and Vasc^HIGH signatures

Inflam^HIGH phenotypes correlate with abundant CD8⁺ and CD8⁺ PD1⁺ cells