This paper presents a dual-observers-based nonsingular fast terminal sliding mode control scheme for quadrotor unmanned aerial vehicles (QUAVs) with unknown disturbances and time-varying delays. Firstly, to facilitate the controller design, the QUAVs model is decoupled into two subsystems: position subsystem and attitude subsystem. Secondly, for the position subsystem, a sliding mode controller is presented to control the position of the QUAVs. For the attitude subsystem, by introducing an exponential term, a nonsingular fast terminal sliding mode controller is obtained to ensure the fast convergence of the attitude angles. Moreover, based on the exponential term, the singularity problem of the conventional terminal sliding mode is solved. Thirdly, the disturbance and time-varying delay observers are presented by considering the time-varying delayed signals and unknown disturbances. Finally, the effectiveness and feasibility of the proposed control scheme are demonstrated by some computer simulations.