Chili pepper (Capsicum annuum L.), one of the most economically important vegetable crops globally, faces significant economic risks from anthracnose, leading to yield losses of 10% as well as decreasing marketability. Early and accurate detection is essential for mitigating these effects. Recent advancements in deep learning, particularly in image recognition, offer promising solutions for plant disease detection. This study applies deep learning models-MobileNet, ResNet50v2, and Xception-using transfer learning to diagnose anthracnose in chili peppers. A key challenge is the need for large, labeled datasets, which are costly to obtain. The study aims to identify the minimum dataset size required for accurate and efficient disease diagnosis using limited data. Performance metrics, including precision, recall, F1-score, and accuracy, were evaluated across different dataset sizes (500, 1,000, 2,000, 3,000, and 4,000 samples). Results indicated that model performance improves with larger datasets, with ResNet50v2 and Xception requiring more data to achieve optimal accuracy, while MobileNet showed strong generalization even with smaller datasets. These findings underscore the effectiveness of transfer learning-based models in plant disease detection, offering practical guidelines for balancing data availability and model performance in agricultural applications. Source code available at https://github.com/smart-able/Anthracnose.git.