Cardiovascular disease (CVD) is caused by the abnormal functioning of the heart which results in a high mortality rate across the globe. The accurate and early prediction of various CVDs from the electrocardiogram (ECG) is vital for the prevention of deaths caused by CVD. Artificial intelligence (AI) is used to categorize and accurately predict various CVDs. Among different AI-based techniques, deep learning (DL)--based approaches are more effective in classifying various CVDs because they extract characteristics directly from the huge amounts of data needed to train the DL network. This paper proposes and compares the performance of a one-dimensional (1D), two-dimensional (2D) convolutional neural network (CNN), and a multi-branch convolutional neural network (MB-CNN) to classify various CVDs, namely, dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy (HCM), myocardial infarction (MI) and coronary artery disease (CAD) from spectrograms of one-dimensional (1D) ECG records. The 1D ECG records are classified using a 1D CNN is proposed which achieves a maximum performance of 97.60 %. To boost performance, the 1D ECG recordings are converted into 2D-ECG spectrograms via the continuous wavelet transform (CWT) and classified based on the proposed 2D-CNN with a maximum accuracy of 98.46 %. To further improve the classification performance, the obtained 2D- ECG spectrograms are classified using the proposed MB-CNN containing multiple branches which can capture various degrees of abstraction leading to a precise classification. The proposed approach using the MB-CNN model obtains an average test accuracy of 99.34 % for the classifications of five types of CVDs and 99.22 % for the classification of 5 classes of ECGs in the MIT-BIH database.