Optical coherence tomography (OCT) is promising to become an essential imaging tool for non-invasive oral mucosal tissue assessment, but it faces challenges like speckle noise and motion artifacts. In addition, it is difficult to distinguish different layers of oral mucosal tissues from gray level OCT images due to the similarity of optical properties between different layers. We introduce the Efficient Segmentation-Denoising Model (ESDM), a multi-task deep learning framework designed to enhance OCT imaging by reducing scan time from ∼8s to ∼2s and improving oral epithelium layer segmentation. ESDM integrates the local feature extraction capabilities of the convolution layer and the long-term information processing advantages of the transformer, achieving better denoising and segmentation performance compared to existing models. Our evaluation shows that ESDM outperforms state-of-the-art models with a PSNR of 26.272, SSIM of 0.737, mDice of 0.972, and mIoU of 0.948. Ablation studies confirm the effectiveness of our design, such as the feature fusion methods, which enhance performance with minimal model complexity increase. ESDM also presents high accuracy in quantifying oral epithelium thickness, achieving mean absolute errors as low as 5 µm compared to manual measurements. This research shows that ESDM can notably improve OCT imaging and reduce the cost of accurate oral epithermal segmentation, improving diagnostic capabilities in clinical settings.