This article proposes a multi-scale deep learning network to classify different underwater acoustic signal sources. The proposed network is cleverly designed with multiple branches, creating a multi-scale block which allows learning various spatial features of Constant-Q Transform spectrograms. The network is trained and tested on the ShipsEAR dataset, which is augmented by the overlap segmentation technique to ensure a balance in ship label classes’ data. The experiment results show that our network achieves an average classification accuracy of up to 99.93% and an execution speed of 2.2 ms when configured with two multi-scale blocks and 32 filter channels. In comparison, our network remarkably outperforms other existing networks in terms of accuracy and execution time.