Many artificial intelligence (AI) algorithms struggle to adapt effectively in dynamic real-world scenarios, such as complex classification tasks and object relationship extraction, due to their predictable but non-adaptive behavior. This paper introduces HybridBranchNetV2, an optimized hybrid architecture designed to address these challenges. The key novelty of our approach lies in the integration of reinforcement learning for adaptive feature extraction and the use of graph-based techniques to analyze object relationships in complex environments. By dynamically adjusting feature extraction based on feedback from the environment, the model improves adaptability, while graph-based methods allow for a more comprehensive analysis of object relationships. Our extensive evaluations demonstrate that HybridBranchNetV2 achieves average 91.75% accuracy over four different challenging datasets. In particular, a 14% improvement obtained on the Visual Genome dataset and ImageNet 1K compared to the original HybridBranchNet model. Additional testing on CIFAR, Flowers, and ImageNet datasets revealed improvements of 6%, 1%, and 6%, respectively. These advancements not only enhance classification accuracy but also ensure efficient computation, making HybridBranchNetV2 suitable for real-time applications with minimal risk of overfitting. The proposed framework demonstrates significant improvements in adaptability, performance, and computational efficiency, addressing critical limitations in current AI models.