Colloidal gels, ubiquitous in industrial applications, can undergo reversible solid-to-liquid transitions. Recent work demonstrates that adding surface roughness to primary particles enhances the toughness and influences the self-healing properties of colloidal gels. In the present work, we first use colloidal probe atomic force microscopy (CP-AFM) to assess the quantitative changes in adhesive and frictional forces between thermoresponsive particles as a function of their roughness. The presence of static friction, generated by interparticle adhesion results in noncentral forces, leading to network structures that are more readily constrained in their nodes. Systems with higher friction exhibited increased sedimentation stability, a decrease in percolation threshold and a more abrupt elastic to plastic transition, but an enhanced capacity in storing elastic energy until fluidification. Additional experiments with geometrically smooth but "chemically rough" (patchy) particles further emphasized the importance of static interparticle friction in the macroscopic yielding and recovery behavior of colloidal gels.