Agricultural non-point source pollution (ANPSP) impacts water quality in the core water source area (CWSA) of the Middle Route of the South North Water Diversion Project. Effective governance is required to manage water pollution in this area. An inventory analysis method was used in the CSWA to estimate the ANPSP loads between 2006 and 2020. The Spatial Durbin Model was applied to identify the factors that influenced ANPSP. From 2006 to 2020, pollutant loads in the CWSA significantly decreased. The total nitrogen (TN) and total phosphorus (TP) loads in the ANPSP are decreasing. The TN and TP pollution intensity initially increased and then decreased, with the main sources shifting from livestock breeding to fertilizers. There was a significant spatial correlation of ANPSP in the CWSA, and agricultural economic growth and agglomeration exacerbated the ANPSP loads. There were regional differences in the impacts of urbanization, agricultural labor transfer, mechanization, and agricultural production efficiency on ANPSP. The adjustment of planting structure reduced ANPSP loads and produced a positive spillover effect. Finally, policy recommendations were made to support the protection of local water quality and enhance the efficiency of subsequent pollution control projects.