Neuroanatomical sex differences estimated in neuroimaging studies are confounded by total intracranial volume (TIV) as a major biological factor. Employing a matching approach widely used for causal modeling, we disentangled the effect of TIV from sex to study sex-differentiated brain aging trajectories, their relation to functional networks and cytoarchitectonic classes, brain allometry, and cognition. Using data from the UK Biobank, we created subsamples that removed, maintained, or exaggerated the TIV differences in the original sample. We compared regional and vertex-level sex estimates across subsamples. The overall sex-related differences diminished in head size-matched subsamples, suggesting that most of the observed variability results from TIV differences. Furthermore, bidirectional sex differences in brain neuroanatomy emerged that were previously masked by the effect of TIV. Allometry remained fairly consistent across lifespan and was not sex-differentiated. Finally, the matching process changed the direction of the estimated sex differences in "verbal and numerical reasoning" and "working memory", suggesting that behavioral sex difference investigations can benefit from additional biological analysis to uncover the underlying factors contributing to cognition. Taken together, we provide new evidence disentangling sex differences from TIV as a relevant biological confound.