Toxicity remains one of the major challenges that prevent Pb-based halide perovskites from widespread utilization. Ideally, non-toxic alternatives can be identified while still maintaining the superior power conversion efficiency of the Pb-based perovskite solar cells. Using the currently most promising candidate, the Sn-based halide perovskites, as an example, we show that a trade-off exists between toxicity and efficiency in the Sn- versus Pb-based halide perovskites. Indeed, the dominant nonradiative recombination center in the Sn-based halide perovskites differs from the one in its Pb-based counterparts, resulting in the nonradiative capture coefficient in CsSnI3 being an order of magnitude higher than that in CsPbI3. We attribute this difference to the band alignment. Our results indicate that development of halide perovskites beyond the Pb and Sn bases is essential for efficient yet environmentally friendly perovskite solar cells.