Power transformers play a critical role in electric power systems by facilitating connections between subsystems with different voltage levels. Ensuring their reliable operation and protection against faults is essential for maintaining network stability. Percentage differential (PD) protection, commonly used in transformer protection relays, faces limitations such as dealing with current transformer (CT) saturation during external faults and difficulty in detecting certain fault types. This paper proposes an enhanced transformer protection scheme by integrating the alpha plane (AP) concept, traditionally used in transmission line protection, to improve transformer differential protection. This entails redefining AP restraint and operating zones based on transformer current behavior under various operating and fault conditions, in addition to introducing a novel algorithm utilizing a new mathematical approach to compute the alpha-plane complex ratio. The proposed methodology along with the PD approach are evaluated through MATLAB/SIMULINK on an autotransformer within a simulated real power system, supplemented by experimental evaluation. The results demonstrate the proposed scheme offers higher sensitivity to high impedance ground faults, maintains stability during border range of CT saturation under external faults, and delivers faster tripping times compared to the traditional PD method. This approach provides a robust and reliable solution to overcome the inherent challenges of conventional transformer protection techniques.