Accurate and timely segmentation of COVID-19 infection regions is critical for effective diagnosis and treatment. While convolutional neural networks (CNNs) exhibit strong performance in medical image segmentation, they face challenges in handling complex lesion morphologies with irregular boundaries. Transformer-based approaches, though demonstrating superior capability in capturing global context, suffer from high computational costs and suboptimal multi-scale feature integration. To address these limitations, we proposed Hierarchical Agent Transformer Network (HATNet), a hierarchical encoder-bridge-decoder architecture that optimally balances segmentation accuracy with computational efficiency. The encoder employs novel agent Transformer blocks specifically designed to capture subtle features of small COVID-19 lesions through agent tokens with linear computational complexity. A diversity restoration module (DRM) is innovatively embedded within each agent Transformer block to counteract feature degradation. The hierarchical structure simultaneously extracts high-resolution shallow features and low-resolution fine features, ensuring comprehensive feature representation. The bridge stage incorporates an improved pyramid pooling module (IPPM) that establishes hierarchical global priors, significantly improving contextual understanding for the decoder. The decoder integrates a full-scale bidirectional feature pyramid network (FsBiFPN) with a dedicated border-refinement module (BRM), collectively enhancing edge precision. The HATNet were evaluated on the COVID-19-CT-Seg and CC-CCII datasets. Experimental results yielded Dice scores of 84.14% and 81.22% respectively, demonstrating superior segmentation performance compared to state-of-the-art models. Furthermore, it achieved notable advantages in model parameters and computational complexity, highlighting its clinical deployment potential.