A technique based on infrared digital holography with a U-shaped residual SE network driven by simulated hologram data is proposed for real-time and high-fidelity monitoring of crystal growth. This method achieves high-quality reconstruction with accurate amplitude and phase information, enhanced noise suppression, and resolution, offering a novel, to the best of our knowledge, approach for non-destructive crystal growth observation.