The removal of non-tobacco related materials (NTRMs) is crucial for improving tobacco product quality and consumer safety. Traditional NTRM detection methods are labor-intensive and inefficient. This study proposes a novel approach for real-time NTRM detection using hyperspectral imaging (HSI) and an enhanced YOLOv8 model, named Dual-branch-YOLO-Tobacco (DBY-Tobacco). We created a dataset of 1,000 images containing 4,203 NTRMs by using a hyperspectral camera, SpectraEye (SEL-24), with a spectral range of 400-900 nm. To improve processing efficiency of HSIs data, three characteristic wavelengths (580nm, 680nm, and 850nm) were extracted by analyzing the weighted coefficients of the principal components. Then the pseudo color image fusion and decorrelation contrast stretch methods were applied for image enhancement. The DBY-Tobacco model features a dual-branch backbone network and a BiFPN-Efficient-Lighting-Feature-Pyramid-Network (BELFPN) module for effective feature fusion. Experimental results demonstrate that the DBY-Tobacco model achieves high performance metrics, including an F1 score of 89.7%, mAP@50 of 92.8%, mAP@50-95 of 73.7%, and a processing speed of 151 FPS, making it suitable for real-time applications in dynamic production environments. The study highlights the potential of combining HSI with advanced deep learning techniques for improving tobacco product quality and safety. Future work will focus on addressing limitations such as stripe noise in HSI and expanding the detection to other types of NTRMs. The dataset and code are available at: https://github.com/Ikaros-sc/DBY-Tobacco.