INTRODUCTION: Potatoes are a vital global product, and prompt identification of foliar diseases is imperative for sustaining healthy yields. Computer vision is essential in precision agriculture, facilitating automated disease diagnosis and decision-making through real-time data. Inconsistent data in uncontrolled contexts undermines classic image classification techniques, hindering precise illness detection. METHODS: We present a novel model that integrates EfficientNet-LITE for enhanced feature extraction with KE-SVM Optimization for effective classification. KE-SVM Optimization cross-references misclassified instances with correct classifications across kernels, iteratively refining the confusion matrix to improve accuracy across all classes. EfficientNet-LITE improves the model's emphasis on pertinent features through Channel Attention (CA) and 1-D Local Binary Pattern (LBP), while preserving computational economy with a reduced model size of 12.46 MB, fewer parameters at 3.11M, and a diminished FLOP count of 359.69 MFLOPs. RESULTS: Before optimization, the SVM classifier attained an accuracy of 79.38% on uncontrolled data and 99.07% on laboratory-controlled data. Following the implementation of KE-SVM Optimization, accuracy increased to 87.82% for uncontrolled data and 99.54% for laboratory-controlled data. DISCUSSION: The model's efficiency and improved accuracy render it especially appropriate for settings with constrained computational resources, such as mobile or edge devices, offering substantial practical advantages for precision agriculture.