The rut depth created by off-road vehicles affects vehicle performance and soil compaction, and its accurate prediction is essential to improve vehicle performance and reduce soil compaction. Due to the complex and nonlinear interactions between variables and rut depth, the error in estimating rut depth with conventional methods is significant. Therefore, the present study aims to predict the rut depth created by off-road vehicles in soil using the Categorical Boosting (CatBoost) machine learning algorithm and combining it with three optimization methods, the Gray Wolf Optimization (GWO) algorithm, Particle Swarm Optimization (PSO), and the Secretary Bird Optimization Algorithm (SBOA). The experimental data included 270 samples with vertical load variables (2, 3, and 4 kN), movement speed (1, 2, and 3 km/h), two traction devices (pneumatic tire and tracked wheel), and the number of passes (15 levels), which were collected under indoor conditions using a soil bin equipped with a single-wheel tester. The model hyperparameters were adjusted using the GWO and SBOA algorithms to increase the prediction accuracy and reduce the model error. The results showed that the SBOA-CatBoost hybrid model, with a Root Mean Square Error of 0.35 mm and a coefficient of determination of 0.97707, performed better than the other models. Furthermore, the SBOA-CatBoost hybrid model outperformed the other models with a Mean Absolute Percentage Error of 1.2%.