Accurately assessing mental states-such as mental workload and fatigue- is crucial for ensuring the reliability and effectiveness of brain-computer interface (BCI)-based applications. Relying on signals from a limited brain region with low spatial resolution may fail to capture the full scope of relevant information. To address this, high-density diffuse optical tomography (HD-DOT), an emerging form of functional near-infrared spectroscopy (fNIRS) was employed in this study, which provides higher spatial resolution for hemodynamic measurements and enables the reconstruction of 3D brain images. An experiment protocol was designed to investigate both mental workload and fatigue, two critical components of cognitive state that often fluctuate concurrently in real-world scenarios. Machine learning methods were applied for subject-specific classification, achieving 95.14% mean accuracy for fatigue/non-fatigue and 97.93% for four n-back tasks using Random Forest, outperforming Support Vector Machines. These results highlight the transformative potential of HD-DOT in advancing multifaceted cognitive state assessment, paving the way for more precise, adaptable, and powerful BCI applications.