Miniaturized pixel sizes in near-eye digital displays lead to pixel emission patterns with large divergence angles, necessitating efficient beam collimation solutions to improve the light coupling efficiency. Traditional beam collimation optics, such as lenses and cavities, are wavelength-sensitive and cannot simultaneously collimate red (R), green (G), and blue (B) light. In this work, we employed inverse design optimization and finite-difference time-domain (FDTD) simulation techniques to design a collimator comprised of nano-sized photonic structures. To alleviate the challenges of the spatial incoherence nature of micro-LED emission light, we developed a strategy called dual-task optimization. Specifically, the method models light collimation as a dual task of color routing. By optimizing a color router, which routes incident light within a small angular range to different locations based on its spectrum, we simultaneously obtained a beam collimator, which can restrict the output of the light emitted from the routing destination with a small divergence angle. We further evaluated the collimation performance for spatially incoherent RGB micro-LED light in an FDTD using a multiple-dipole simulation method, and the simulation results demonstrate that our designed collimator can increase the light coupling efficiency from approximately 30% to 60% within a divergence angle of ±20° for all R/G/B light under the spatially incoherent emission.