Autophagy is an evolutionarily conserved self-degradation process in eukaryotes. It not only plays a role in plant growth and development but also is involved in plant responses to biotic and abiotic stresses. Plants can initiate autophagy to degrade the surplus or damaged cytoplasmic materials and organelles, thus coping with abiotic and biotic stresses. The initiation of autophagy depends on autophagy-related genes (ATGs). The transcription factors can directly bind to the promoters of ATGs to activate autophagy and regulate their transcriptional levels and post-translational modifications. Furthermore, ATGs can directly or indirectly interact with plant hormones to regulate plant responses to stresses. When plants are exposed to salinity, drought, extreme temperatures, nutrient deficiencies, and pathogen stress, ATGs are significantly induced, which enhances the autophagy activity to facilitate the degradation of the denatured and misfolded proteins, thereby enhancing plant tolerance to adversity stresses. This article summarizes the discovery, structures, and classification of plant ATGs, reviews the research progress in the mechanisms of ATGs in plant responses to abiotic and biotic stresses, and prospects the future research directions. This review is expected to provide the genetic resources and a theoretical foundation for the genetic improvement of crops in responses to stress tolerance.