Calibrating agent-based models (ABMs) in economics and finance typically involves a derivative-free search in a very large parameter space. In this work, we benchmark a number of search methods in the calibration of a well-known macroeconomic ABM on real data, and further assess the performance of "mixed strategies" made by combining different methods. We find that methods based on random-forest surrogates are particularly efficient, and that combining search methods generally increases performance since the biases of any single method are mitigated. Moving from these observations, we propose a reinforcement learning (RL) scheme to automatically select and combine search methods on-the-fly during a calibration run. The RL agent keeps exploiting a specific method only as long as this keeps performing well, but explores new strategies when the specific method reaches a performance plateau. The resulting RL search scheme outperforms any other method or method combination tested, and does not rely on any prior information or trial and error procedure.Comment: 9 pages and 5 figures, presented at the AAAI bridge program 'AI for Financial Institutions' (https://aaai23.bankit.art/), at the ICLR bridge program 'AI4ABM' (https://ai4abm.org/workshop_iclr2023/) and at ICAIF '23 (https://ai-finance.org/). Proceedings of the Fourth ACM International Conference on AI in Finance, (ICAIF 23), Association for Computing Machinery, New York, NY, USA