Scanning magnetometry with nitrogen-vacancy (NV) centers in diamond has led to significant advances in the sensitive imaging of magnetic systems. The spatial resolution of the technique, however, remains limited to tens to hundreds of nanometers, even for probes where NV centers are engineered within 10 nm from the tip apex. Here, we present a correlated investigation of the crucial parameters that determine the spatial resolution: the mechanical and magnetic stand-off distances, as well as the subsurface NV center depth in diamond. We study their contributions using mechanical approach curves, photoluminescence measurements, magnetometry scans, and nuclear magnetic resonance (NMR) spectroscopy of surface adsorbates. We first show that the stand-off distance is mainly limited by features on the surface of the diamond tip, hindering mechanical access. Next, we demonstrate that frequency-modulated (FM) atomic force microscopy feedback partially overcomes this issue, leading to closer and more consistent magnetic stand-off distances (26-87 nm) compared with the more common amplitude-modulated feedback (43-128 nm). FM operation thus permits improved magnetic imaging of sub-100-nm spin textures, shown for the spin cycloid in BiFeO