This paper proposed an improved YOLOv5s-based method to address the challenging detection of cracks in retaining walls due to their irregular development and small size. This approach was developed by applying the YOLOv5s framework and incorporating several enhancements. Specifically, the BotNet module was introduced into the Backbone to enhance the extraction of long-term dependence and global features. The GhostNetV2 module was also utilized to reduce the network complexity, making it suitable for edge-based detection. In the Neck, the deployment of the ODConv module improved the extraction of multi-scale features. Another incorporated tool was the rotated bounding boxes for accurately localizing irregularly oriented cracks. The experimental results showed that the improved algorithm reduced the number of YOLOv5s parameters by 12% and increased the average precision by 1.5% compared to YOLOv5s using only the rotating prediction frames. In summary, the proposed algorithm outperformed the traditional YOLOv5s in better crack localization, quantity calculation, and a smaller parameter volume, presenting a suitable tool for retaining wall crack detection.