The immune cell component of the tumor microenvironment is an important modulator of tumor progression. In patients with breast cancer, tumor-infiltrating lymphocytes (TILs) and tertiary lymphoid structures (TLS) represent core aspects of anti-tumor immunity, both increasingly recognized for clinical relevance. In this study, we evaluated immune-related histology features using whole slide H&E images of the TCGA-BRCA dataset (n=1035) and analyzed these distinct features relative to gene expression, PAM50 subtypes, and patient survival. H&E images were evaluated for TILs, plasma cells (PCs), high-endothelial venule associated lymphoid aggregates (HALA) and mature TLS. For HALA and TLS, location relative to tumor (non-tumor, peritumor, and intratumor) was determined. HER2-enriched (HER2E) and basal-like breast tumors exhibited the highest mean TILs and presence of PCs. HALA were present in 35.1% of cases and TLS in 6.5% of cases, also predominantly in HER2E and basal-like tumors. We derived gene expression signatures for 10 histologically defined immune features and tested their clinical significance using transcriptomic and survival data from the SCAN-B cohort. Signatures related to TILs, PCs, HALA/TLS, TLS, and specifically intratumor HALA and TLS were associated with better survival in HER2E and basal-like tumors. Peritumor HALA/TLS and non-tumor signatures were non-significant or associated with worse outcomes. Further, we compared the immune microenvironment of high-TIL (TILs >
10%) tumors from TCGA-BRCA by PAM50 subtype through supervised analyses of 200+ immune gene expression signatures, and unique immune features were identified for each subtype. In high-TIL luminal tumors, enriched immune signatures had little relation to prognosis. High-TIL HER2E and basal-like tumors had distinct immune signatures linked to improved survival, related to B and T cells, respectively. Overall, PAM50 subtypes of breast cancer exhibit distinct immune microenvironments, both histologically and molecularly. These differences in immune properties should be considered when developing precise treatment strategies to achieve optimal therapeutic efficacy for patients.