The imaging quality of single-pixel spectral imaging (SSI) methods is poor at a low sampling ratio (SR). To tackle this problem, a new Fourier single-pixel spectral imaging (FSSI) technique is proposed. Firstly, we introduce the low-rank tensor nuclear norm (TNN) to characterize the correlation between spectral images. Compared with the conventional method, TNN reconstructs image details better but brings image artifacts simultaneously. Therefore, local low-rank TNN (LTNN) constraint is proposed to ameliorate global ones and to reduce the distortion caused by TNN and low SR. Secondly, to make full use of the spectral information, the proposed constraint is used as the coarse prior, and the deep tensor prior (DTP) is introduced as the fine one to construct the joint priors. Different from the single prior, the joint method can make the two priors benefit and improve each other and further enhance the imaging quality. Finally, an efficient and high-quality SSI technique is achieved by deducing the closed-form solution algorithm. Experimental results show that our method significantly improves the quality of FSSI as much as 7-10 dB when compared to 3DTV at the SR of 5%.