Skin cancer is recognized as one of the most harmful cancers worldwide. Early detection of this cancer is an effective measure for treating the disease efficiently. Traditional skin cancer detection methods face scalability challenges and overfitting issues. To address these complexities, this study proposes a random cat swarm optimization (CSO)with an ensemble convolutional neural network (RCS-ECNN) method to categorize the different stages of skin cancer. In this study, two deep learning classifiers, deep neural network (DNN) and Keras DNN (KDNN), are utilized to identify the stages of skin cancer. In this method, an effective preprocessing phase is presented to simplify the classification process. The optimal features are selected using the feature extraction phase. Then, the GrabCut algorithm is employed to carry out the segmentation process. Also, the CSO is employed to enhance the effectiveness of the method. The HAM10000 and ISIC datasets are utilized to evaluate the RCS-ECNN method. The RCS-ECNN method achieved an accuracy of 99.56%, a recall of 99.66%, a specificity value of 99.254%, a precision value of 99.18%, and an F1-score value of 98.545%, respectively. The experimental results demonstrated that the RCS-ECNN method outperforms the existing techniques.