Earth's energy budget is sensitive to the spatial distribution of sea surface temperature and sea ice concentration (SIC) change, but the global radiative effect of changes in SIC spatial distribution has not been quantified. We show that SIC-induced radiation anomalies at the top of the atmosphere are sensitive to the location of SIC reduction in each season, which qualitatively explains how and why the effect of sea ice loss on Earth's energy budget is determined by its spatial pattern. Idealized experiments indicate that SIC-induced surface warming is greater in the Arctic regions, resulting in a more negative Planck feedback. Global low-level cloud cover responses to Arctic and Antarctic SIC reduction are also distinct, leading to more negative SIC-cloud feedback in Arctic regions. SIC-induced albedo feedback is sensitive to latitude due to inhomogeneous solar radiation at the surface. As a result, the simulated radiative effect of SIC anomalies during 1980-2019 is dominated by variations in the spatial pattern of SIC.