Unmanned Aerial Vehicles (UAVs) integrated with lightweight visual cameras hold significant promise in renewable energy asset inspection and monitoring. This study presents an AI-assisted soiling detection methodology for inspecting solar photovoltaic (PV) panels, using UAV-captured RGB images. The proposed scheme introduces an autonomous end-to-end soiling detection model for common types of soiling in solar panel installations, including bird droppings and dust. Detecting soiling, particularly bird droppings, is critical due to their pronounced negative impact on power generation, primarily through hotspot formation and their resistance to natural cleaning processes such as rain. A dataset containing aerial RGB images of PV panels with dust and bird droppings is collected as a prerequisite. This study addresses the unique challenges posed by the small size and indistinct features of bird droppings in aerial imagery in contrast to relatively large-sized dust regions. To overcome these challenges, we developed a custom model, named SDS-YOLO (Soiling Detection System YOLO), which features a Convolutional Block Attention Module (CBAM) and two dedicated detection heads optimized for dust and bird droppings. The SDS-YOLO model significantly improves detection accuracy for bird droppings while maintaining robust performance for the dust class, compared with YOLOv5, YOLOv8, and YOLOv11. With the integration of CBAM, we achieved a substantial 40.2% increase in mean Average Precision (mAP50) and a 26.6% improvement in F1 score for bird droppings. Dust detection metrics also benefited from this attention-based refinement. These results underscore the CBAM's role in improving feature extraction and reducing false positives, particularly for challenging soiling types. Additionally, the SDS-YOLO parameter count is reduced by 24%, thus enhancing its suitability for edge computing applications.