To address the Urban Heat Island (UHI) effect-a significant urban climate challenge-detailed urban microclimate modeling is essential. Such modeling typically requires data on urban surface properties and morphologies from street canyons and buildings. Most urban surveying efforts have focused on morphological attributes such as sky view factor, vegetation or building surface ratio, while the mass-collection of facade materials has been hindered by the complexity of the segmentation task and the need for large and diverse labeled datasets. Recognizing the importance of mapping facade materials for urban thermal comfort, envelope heat emissions, and building energy studies, we employ computer vision-based state-of-the-art zero-shot learning paradigms for high-fidelity facade material extraction. Our approach circumvents the traditional need for extensive labeled training data, allowing for adaptation to a variety of urban contexts and material types. Tested in Dubai, Amsterdam, and Boston (three architecturally diverse cities), our algorithm successfully detects the predominant facade material in 68% of cases and identifies the top three present material classes in 85% of cases. Additionally, we show how material coverage identification is crucial for assessing outdoor thermal comfort, as evident in shifts in annual cold and heat stress hours across the climates of the three cities in a sample urban canyon.