This article addresses challenges of autonomous decisions making influenced by discrete system states, underlying continuous dynamics, and evolving operational environments. A comprehensive framework is proposed, encompassing new modeling, problem formulation, control design, and stability analysis. The framework integrates continuous system dynamics, used for low-level control, with discrete Markov decision processes (MDP) for high-level decision making. To capture the interactions between these domains, the decision-making system is modeled as a hybrid system consisting of a controlled MDP and autonomous (uncontrolled) continuous dynamics, collectively referred to as the hybrid Markov decision process (HMDP). The design focuses on ensuring safety and optimality by accounting for both discrete and continuous state variables across different levels. With the help of the model predictive control (MPC) concept, a decision-making scheme is developed for the hybrid model, with guarantees for recursive feasibility and stability. The proposed framework is applied to the autonomous lane changing system for intelligent vehicles, and simulation shows its capability to handle diverse behaviors in dynamic and complex environments.