Self-interference digital holography extends the application of digital holography to non-coherent imaging fields such as fluorescence and scattered light, providing a new solution, to the best of our knowledge, for wide field 3D imaging of low coherence or partially coherent signals. However, cross talk information has always been an important factor limiting the resolution of this imaging method. The suppression of cross talk information is a complex nonlinear problem, and deep learning can easily obtain its corresponding nonlinear model through data-driven methods. However, in real experiments, it is difficult to obtain such paired datasets to complete training. Here, we propose an unsupervised cross talk suppression method based on a cycle-consistent generative adversarial network (CycleGAN) for self-interference digital holography. Through the introduction of a saliency constraint, the unsupervised model, named crosstalk suppressing with unsupervised neural network (CS-UNN), can learn the mapping between two image domains without requiring paired training data while avoiding distortions of the image content. Experimental analysis has shown that this method can suppress cross talk information in reconstructed images without the need for training strategies on a large number of paired datasets, providing an effective solution for the application of the self-interference digital holography technology.