A comprehensive insight into ultrafast dynamics of photo-switchable materials is desired for efficient control of material properties through light excitation. Here, we study a polycrystalline spin crossover thin film as a prototypical example and reveal the sequential photo-switching dynamics, from local molecular rearrangement to global lattice deformation. On the earliest femtosecond timescale, the local molecular structural rearrangement occurs within a constant unit-cell volume through a two-step process, involving initial Fe-ligand bond elongation followed by ligand rotation. The highly-oriented structure of the nanocrystalline films and the experimental geometry enables resolving the full anisotropic lattice structural dynamics in and out of the sample plane separately. While both molecular switching and lattice heating influence lattice volume, they exert varying degrees of impact at disparate time scales following photoexcitation. This study highlights the opportunities provided by Mega-electron-volt electron and X-ray free electron laser to advance the understanding of ultrafast dynamics of photo-switchable materials.