The quantitative trait loci (QTL)-by-environment (Q × E) interaction effect is hard to detect because there are no effective ways to control the genomic background. In this study, we propose a linear mixed model that simultaneously analyzes data from multiple environments to detect Q × E interactions. This model incorporates two different kinship matrices derived from the genome-wide markers to control both main and interaction polygenic background effects. Simulation studies demonstrate that our approach is more powerful than the meta-analysis and inclusive composite interval mapping methods. We further analyze four agronomic traits of rice across four environments. A main effect QTL is identified for 1000-grain weight (KGW), while no QTL are found for tiller number. Additionally, a large QTL with a significant Q × E interaction is detected on chromosome 7 affecting grain number, yield, and KGW. This region harbors two important genes, PROG1 and Ghd7. Furthermore, we apply our mixed model to analyze lodging in barley across six environments. The six regions exhibiting Q × E interaction effects identified by our approach overlap with the SNPs previously identified using EM and MCMC-based Bayesian methods, further validating the robustness of our approach. Both simulation studies and empirical data analyses show that our method outperformed all other methods compared.