Image steganalysis, detecting hidden data in digital images, is essential for enhancing digital security. Traditional steganalysis methods typically rely on large, pre-labeled image datasets, which are difficult and costly to compile. To address this, this paper introduces an innovative approach that combines active learning and off-policy Deep Reinforcement Learning (DRL) to improve image steganalysis with minimal labeled data. Active learning allows the model to intelligently choose which unlabeled images should be annotated, thus reducing the amount of labeled data needed for effective training. Traditional active learning strategies often use static selection methods that restrict flexibility and do not adjust well to dynamic environments. To overcome this, our method incorporates off-policy DRL for strategic data selection. The off-policy in DRL can increase sample efficiency and significantly enhance learning outcomes. We also use the Differential Evolution (DE) algorithm to fine-tune the hyperparameters of the model, reducing its sensitivity to different settings and ensuring more stable results. Our testing on the extensive BossBase 1.01 and BOWS-2 datasets demonstrates the robust ability of the approach to distinguish between unaltered and steganographic images, achieving an average F-measure of 93.152% on BossBase 1.01 and 91.834% on the BOWS-2 dataset. In summary, this research enhances digital security by employing advanced image steganalysis to detect hidden data, significantly improving detection accuracy with minimal labeled data.