OBJECTIVE: Tracking tumors with multi-leaf collimators and X-ray imaging can be a cost-effective motion management method to reduce internal target volume margins for lung cancer patients, sparing normal tissues while ensuring target coverage. To realize that, accurate tumor localization on X-ray images is essential. We aimed to develop a systematic method for automatically generating tumor segmentation ground truth (GT) on CBCT projections and use it to help refine and validate our patient-specific AI-based tumor localization model. APPROACH: To obtain the tumor segmentation GT on CBCT projections, we propose a 4DCBCT-aided GT generation pipeline consisting of three steps: breathing phase extraction and 10-phase 4DCBCT reconstruction, manual segmentation on phase 50%, deformable contour propagation to other phases, and forward projection of the 3D segmentation to the CBCT projection of the corresponding phase. We then used the CBCT projections from one fraction in the angular range of [-10°, 10°] and [80°, 100°] to refine a Retina U-Net baseline model, which was pretrained on 1140231 digitally reconstructed radiographs generated from a public lung dataset for automatic tumor delineation on projections, and used later-fraction CBCT projections in the same angular range for testing. Six LMU University Hospital patient CBCT projection sets were reserved for validation and 11 for testing. Tracking accuracy was evaluated as the center-of-mass (COM) error and the Dice similarity coefficient (DSC) between the predicted and ground-truth segmentations. MAIN RESULTS: Over the 11 testing patients, each with around 40 CBCT projections tested, the patient-refined models had a mean COM error of 2.3±0.9mm / 4.2±1.7mm and a mean DSC of 0.83±0.06 / 0.72±0.13 for angles within [-10°, 10°] / [80°, 100°]. The mean inference time was 68 ms/frame. The patient-specific training segmentation loss was found to be correlated to the segmentation performance at [-10°, 10°]. SIGNIFICANCE: Our proposed approach allows patient-specific real-time markerless lung tumor tracking, which could be validated thanks to the novel 4DCBCT-aided GT generation approach.