Enhancer-promoter interactions (EPIs) are crucial in gene transcription regulation and cell differentiation. Traditional biological experiments are costly and time-consuming, motivating the development of computational prediction methods. However, existing EPI prediction methods inadequately capture the intricate direct interactions between enhancer and promoter sequences, which limits their prediction performance to some extent. In this work, we propose an innovative attention-based approach RAEPI, which uses convolutional neural networks to extract initial features of enhancers and promoters, combined with a specially designed Restricted Attention mechanism with Query-Key-Value constrained to simulate the interactions between them for further feature extraction. To improve cross-cell line prediction, we employ a transfer learning strategy for pre-training. Furthermore, we extracted sequence motifs to evaluate the RAEPI's effectiveness from a visualization perspective. Experimental results show that RAEPI achieves competitive prediction performance to existing methods on the benchmark dataset.