In order to utilize a large amount of waste scallop shells in road engineering and develop environmentally friendly and high-performance asphalt, the impact of scallop shell powder (SSP) on the high temperature performance of base asphalt and styrene-butadiene-styrene (SBS)-modified asphalt was evaluated in this study. The effects of SSP on the conventional properties of base asphalt and SBS-modified asphalt were investigated according to penetration, softening point, ductility, and rotational viscosity at 135 °C, and recommended dosing amounts were given, respectively. The high-temperature rheological properties of SSP-modified asphalt were evaluated according to temperature sweep and multiple stress creep and recovery (MSCR) tests. The modification mechanism of SSP-modified asphalt was analyzed by scanning electron microscopy (SEM) and Fourier transform infrared spectroscopy (FTIR). The results indicated that an appropriate amount of SSP can effectively improve the consistency and shear deformation resistance of base asphalt and SBS-modified asphalt but can also affect the ductility and fatigue cracking resistance. The recommended weight amounts of SSP in base asphalt and SBS-modified asphalt were 9% and 12%, respectively. Moreover, SSP can improve the thermal stability and resistance to permanent deformation of asphalt, but excessive SSP may weaken the improvement effect of the high-temperature rheological properties of asphalt. The SSP is evenly distributed and tightly combined with asphalt at an appropriate amount, and the modification process of asphalt mainly involves physical changes.