Causally connecting neural activity patterns to behavioral decisions is essential to understand the neural code but requires direct perturbation of the pattern of interest with high specificity. We combined two-photon imaging and cellular-resolution holographic optogenetic photostimulation to causally test how neural activity in the mouse visual cortex is read out to detect visual stimuli. Contrary to expectations, targeted activation of visually sensitive neural ensembles did not preferentially modify behavior compared with targeting randomly selected ensembles. Instead, an activated ensemble's effect on local network activity was the main predictor of its impact on perception. This suggests that downstream regions summate visual cortex activity without preferentially weighting more informative neurons, a notion confirmed by analyzing the impact of photostimulation on decoding models of neural activity. This work challenges conventional notions for how sensory representations mediate perception and demonstrates that perturbing activity is essential to determine which features of neural activity drive behavior.