Electroencephalogram (EEG) signals enhance human-machine interaction but pose challenges in speech recognition due to noise and complexity. This study proposes an Efficient Deep Learning Approach (EDLA) integrating the Gannet Optimization Algorithm (GOA) and Elman Recurrent Neural Network (ERNN) for speaker identification. EEG data is preprocessed using a Savitzky-Golay filter, and key features are selected via recursive feature elimination. Evaluated on the Kara One dataset, EDLA achieves 95.2% accuracy, outperforming baseline methods. This framework advances EEG based speech recognition aiding brain-computer interfaces and assistive technologies for individuals with speech disorders.