Off-axis digital holographic microscopy is a high-throughput, label-free imaging technology that provides three-dimensional, high-resolution information about samples, which is particularly useful in large-scale cellular imaging. However, the hologram reconstruction process poses a significant bottleneck for timely data analysis. To address this challenge, we propose a novel reconstruction approach that integrates deep learning with the physical principles of off-axis holography. We initialized part of the network weights based on the physical principle and then fine-tuned them via supersized learning. Our off-axis hologram network (OAH-Net) retrieves phase and amplitude images with errors that fall within the measurement error range attributable to hardware, and its reconstruction speed significantly surpasses the microscope's acquisition rate. Crucially, OAH-Net, trained and validated on diluted whole blood samples, demonstrates remarkable external generalization capabilities on unseen samples with distinct patterns. Additionally, it can be seamlessly integrated with other models for downstream tasks, enabling end-to-end real-time hologram analysis. This capability further expands off-axis holography's applications in both biological and medical studies.