The prior research about the relationship between living arrangement and elderly people’s mental health is usually based on cross-sectional data and is usually devoted to evaluate the average treatment effect of “empty nest” on a range of health outcomes. Employing the two-wave panel data of China Health and Retirement Longitudinal Study (CHARLS) in 2011 and 2013 this study further investigates the heterogeneity of this effect and gets two findings. Firstly, living arrangement does have a significant effect on elderly people’s mental health but this effect depends both on the gender of the co-residing child and on the distance between parents and the nearest non-co-residing child. In comparison living with daughter is most beneficial to parent’s mental health while living far away from all children is most detrimental; and the other two kinds of living arrangement, namely living with son and living close to children, are situated between the two. Secondly, the same living arrangement may have different effect on different people. The positive effect of living with children or living close to children is most significant when parent is disable or has no spouse. Considering the proportion of elderly people who have imperious demand on children’s support but live far away from all children is very low in our sample we argue that the psychological problem resulted from the change of family structure is minor and only confined in a small fraction of elderly people.