Routine pathology assessment for the tumor grading is currently performed under the microscope by experienced pathologists which might be prone to interpersonal variability and requiring years of experience. Over the past decade, with the help of whole-slide scanning technology, it is now possible to generate whole-slide images. Indeed, this provides an opportunity to extract vision-based information latent in these images and automate and assist pathologists in their daily workflow. In this process, key machine learning algorithms have been developed enabling an automatic segmentation of pathology slides. Here, in this study, we present a novel dataset for Colorectal Cancer Tumor Grade Segmentation, which contains a total of 103 whole-slide images. The ground-truth annotations for these images were obtained from two independent pathologists. The annotations include pixelwise segmentation masks for "Grade-1", "Grade-2", "Grade-3" tumor classes, and "Normal-mucosa" for the normal class. To establish baseline results for this dataset, we trained and evaluated prominent convolutional neural network and transformer models. Our results show that SwinT, a transformer-based model, achieves 63 % mean-dice score, outperforming other transformer-based models and all CNN based models, aligning with the recent success of transformer-based models in the field of computer vision. Most importantly, our new dataset addresses the absence of publicly available datasets for tumor segmentation. Taken together, the findings from our study indicate that integrating various deep neural network structures is promising at facilitating a more unbiased and consistent tumor grading of colorectal cancer using a novel dataset which is publicly available to all researchers.