Psidium guajava L. is an important tropical and subtropical fruit. Due to its geographical location and suitable climate, Taiwan produces Psidium guajava L. all year round. Quality standardization is therefore a crucial issue. The primary objective was to detect appearance defects on harvested fruits. We divided the defects into thirteen classes, including damage from pests, diseases, and humans. We obtained 189 Psidium guajava L. fruits from different farms and collected 1701 images as samples. The YOLO v4 pretrained network architecture achieved excellent performance in defect detection, including a false positive rate of 6.62%, a false negative rate of 5.03%, and accuracy of 88.15%. Moreover, in the detection of Colletotrichum gloeosporoides, Pestalotiopsis psidii, and Phyllosticta psidiicola, the false positive and false negative detection rates were less than 9%. The applicability of the model in real-time harvesting and grading operations was demonstrated by a minimum detectable defect size of 13 × 14 pixels and computation speed of 12 FPS demonstrated.