Grandparents were anticipated to participated in grand-rearing. The COVID-19 pandemic had detached grandparents from rearing grandchildren. The research questions of this study were as follows: How does the change in family relations impact the well-being (SWB) of grandparents and parents? We examined how family structure influenced subjective SWB before and after COVID-19. We focused on the effects of children, grandchildren, and their gender on grandparents and parents. We found that compared with the happiness level before COVID-19, (1) granddaughters increased their grandmothers SWB after COVID-19, (2) both daughters and sons reduced their fathers SWB after COVID-19, whereas neither daughters nor sons changed their mothers SWB, and (3) the negative effect of sons reduced substantially if their fathers had younger brothers. Learning from interactions with younger brothers in childhood, fathers could avoid the deterioration of relationships with their sons, even when unexpected events possibly changed the lifestyle of the family and their relationship.