Accurately extracting lesions from medical images is a fundamental but challenging problem in medical image analysis. In recent years, methods based on convolutional neural networks and Transformer have achieved great success in the medical image segmentation field. Combining the powerful perception of local information by CNNs and the efficient capture of global context by Transformer is crucial for medical image segmentation. However, the unique characteristics of many lesion tissues often lead to poor performance and most previous models failed to fully extract effective local and global features. Therefore, based on an encoder-decoder architecture, we propose a novel alternate encoder dual decoder CNN-Transformer network, AD2Former, with two attractive designs: 1) We propose alternating learning encoder can achieve real-time interaction between local and global information, allowing both to mutually guide learning. 2) We propose dual decoder architecture. The unique way of dual-branch independent decoding and fusion. To efficiently fuse different feature information from two sub-decoders during decoding, we introduce a channel attention module to reduce redundant feature information. Driven by these two designs, AD2Former demonstrates strong capture ability for target regions and fuzzy boundaries. Experiments on multi-organ segmentation and skin lesion segmentation datasets also demonstrate the effectiveness and superiority of AD2Former.