Due to the complexity and variability of application scenarios and the increasing demands for assembly, single-agent algorithms often face challenges in convergence and exhibit poor performance in robotic arm assembly processes. To address these issues, this paper proposes a method that employs a multi-agent reinforcement learning algorithm for the shaft-hole assembly of robotic arms, with a specific focus on square shaft-hole assemblies. First, we analyze the stages of hole-seeking, alignment, and insertion in the shaft-hole assembly process, based on a comprehensive study of the interactions between shafts and holes. Next, a reward function is designed by integrating the decoupled multi-agent deterministic deep deterministic policy gradient (DMDDPG) algorithm. Finally, a simulation environment is created in Gazebo, using circular and square shaft-holes as experimental subjects to model the robotic arm's shaft-hole assembly. The simulation results indicate that the proposed algorithm, which models the first three joints and the last three joints of the robotic arm as multi-agents, demonstrates not only enhanced adaptability but also faster and more stable convergence.