The Internet of Drones (IoD) is an emerging industry that offers convenient services for humans due to the high mobility and flexibility of drones. The IoD substantially enhances human life by enabling diverse drone applications across various domains. However, a malicious adversary can attempt security attacks because communication within an IoD environment is conducted through public channels and because drones are vulnerable to physical attacks. In 2023, Sharma et al. proposed a physical unclonable function (PUF)-based authentication and key agreement (AKA) scheme for the IoD. Regrettably, we discover that their scheme cannot prevent impersonation, stolen verifier, and ephemeral secret leakage (ESL) attacks. Moreover, Sharma et al.'s scheme cannot preserve user untraceability and anonymity. In this paper, we propose a secure and lightweight AKA scheme which addresses the shortcomings of Sharma et al.'s scheme. The proposed scheme has resistance against diverse security attacks, including physical capture attacks on drones, by leveraging a PUF. Furthermore, we utilize lightweight operations such as hash function and XOR operation to accommodate the computational constraints of drones. The security of the proposed scheme is rigorously verified, utilizing "Burrows-Abadi-Needham (BAN) logic", "Real-or-Random (ROR) model", "Automated Validation of Internet Security Protocols and Application (AVISPA)", and informal analysis. Additionally, we compare the security properties, computational cost, communication cost, and energy consumption of the proposed scheme with other related works to evaluate performance. As a result, we determine that our scheme is efficient and well suited for the IoD.