Trust in agents within Virtual Reality is becoming increasingly important, as they provide advice and influence people's decision-making. However, previous studies show that encountering speech recognition errors can reduce users' trust in agents. Such errors lead users to ignore the agent's advice and make suboptimal decisions. While agents can offer an apology to repair trust, its effectiveness is often limited because it fails to fully repair the original level of trust. Therefore, we examined the use of social touch, a social interaction involving physical interaction between users and the virtual agent, to enhance the effect of an apology on trust repair and to increase the acceptability of its advice. In a controlled experiment (N=24), participants experienced a robotic arm touching the back of their hands while interacting with the agent before decision-making. The results showed that social touch did not repair participants' trust in agents. However, participants were more likely to accept the agent's advice when they experienced touch with physical feedback, regardless of the level of trust in the agent. We discuss the role of presenting physical haptic feedback and its influence on human-agent interactions in VR.