Unsustainably high numbers of patients attending emergency departments (ED) is a serious issue worldwide, with consequences for the quality and timeliness of emergency care. Avoidable visits, i.e. unnecessary or that should be dealt with elsewhere, exacerbate this issue. Most studies focussed on avoidable attendances use clinical data collected by hospital staff, while this study relies on survey data collected from patients asked to recall their last ED attendance and reflect on its necessity. We apply a Recursive Bivariate Probit model to quantify the factors affecting patients' perception of an ED visit being avoidable (or not), unveiling how it relates to socio-demographic and contextual factors. We find that patients who do not trust their General Practitioner (GP) are less likely to think their ED visit was avoidable. The perception of whether an ED visit was avoidable is also associated with symptoms experienced, patients' ethnicity and waiting time for a GP appointment.