Monitoring the remaining food in patients' trays is a routine activity in healthcare facilities as it provides valuable insights into the patients' dietary intake. However, estimating food leftovers through visual observation is time-consuming and biased. To tackle this issue, we have devised an efficient deep learning-based approach that promises to revolutionize how we estimate food leftovers. Our first step was creating the LeFoodSet dataset, a pioneering large-scale open dataset explicitly designed for estimating food leftovers. This dataset is unique in its ability to estimate leftover rates and types of food. To the best of our knowledge, this is the first comprehensive dataset for this type of analysis. The dataset comprises 524 image pairs representing 34 Indonesian food categories, each with images captured before and after consumption. Our prediction models employed a combined visual feature extraction and late fusion approach utilizing soft parameter sharing. Here, we used multi-task (MT) models that simultaneously predict leftovers and food types in training. In the experiments, we tested the single task (ST) model, the ST Model with Ground Truth (ST-GT), the MT model, and the MT model with Inter-task Connection (MT-IC). Our AI-based models, particularly the MT and MT-IC models, have shown promising results, outperforming human observation in predicting leftover food. These findings show the best with the ResNet101 model, where the Mean Average Error (MAE) of leftover task and food classification accuracy task is 0.0801 and 90.44% in the MT Model and 0.0817 and 92.56% in the MT-IC Model, respectively. It is proved that the proposed solution has a bright future for AI-based approaches in medical and nursing applications.