Accurate tracking of live cells using video microscopy recordings remains a challenging task for popular state-of-the-art image processing-based object tracking methods. In recent years, many applications have attempted to integrate deep-learning frameworks for this task, but most still heavily rely on consecutive frame-based tracking or other premises that hinder generalized learning. To address this issue, we aimed to develop a novel deep-learning-based tracking method that assumes cells can be tracked by their spatio-temporal neighborhood, without a restriction to consecutive frames. The proposed method has the additional benefit that the motion patterns of the cells can be learned by the predictor without any prior assumptions, and it has the potential to handle a large number of video frames with heavy artifacts. The efficacy of the proposed method is demonstrated through biologically motivated validation strategies and compared against multiple state-of-the-art cell tracking methods on budding yeast recordings and simulated samples.