Detecting electromagnetic radiation scattered from a tip-sample junction has enabled overcoming the diffraction limit and started the flourishing field of polariton nanoimaging. However, most techniques only resolve amplitude and relative phase of the scattered radiation. Here, we utilize field-resolved detection of ultrashort scattered pulses to map the dynamics of surface polaritons in both space and time. Plasmon polaritons in graphene serve as an ideal model system for the study, demonstrating how propagating modes can be visualized and modeled in the time domain by a straightforward mathematical equation and normalization method. This novel approach enables a direct assessment of the polaritons' group and phase velocities, as well as the damping. Additionally, it is particularly powerful in combination with a pump-probe scheme to trace subcycle changes in the polariton propagation upon photoexcitation. Our method readily applies to other quantum materials, providing a versatile tool to study ultrafast nonequilibrium spatiotemporal dynamics of polaritons.