This paper studies a geometric attitude tracking control problem of quadrotor unmanned aerial vehicles (UAVs) with adaptive extended state observers (AESOs). Through coordinate transformation, the error dynamic of AESO can be transformed into a canonical form which is easier to analyze by linear time-varying theory. In the presence of unknown disturbances and system uncertainties, the estimated error and stability of AESO can be analyzed. Simultaneously, a geometric tracking controller is developed on SO(3) with AESOs which guarantees that the error exponentially converges to a bounded set related to the estimated error. Some examples and a flight test are given to verify the availability of the control scheme.