The Anxiety Scale for Autism-Adults (ASA-A) captures the autistic anxiety experience, but we do not know whether it is structurally equivalent for men and women, or non-autistic people. Measurement invariance analysis considered gender and diagnostic status (342 cis-gender autistic men (N = 105) and women (N = 237), 316 cis-gender non-autistic men (N = 104) and women (N = 212)). Strict invariance was achieved between autistic men and women and between non-autistic men and women, but the ASA-A structure is quantitatively different in autistic compared to non-autistic adults. Therefore, this tool cannot be used to directly compare anxiety between these groups. Autistic women scored significantly higher on the ASA-A than autistic men and Uncertainty was the highest scoring factor for autistic participants. Future research should include alternative gender identities.