Computational psychiatry aims to quantify individual patients' psychiatric pathology by measuring behavior during psychophysical tasks and characterizing the neurocomputational parameters underlying specific decision-making systems. While this approach has great potential for informing us about specific computational processes associated with psychopathology, the fundamental psychometric properties of computational assessments remain understudied. Optimizing these psychometric properties, including test-retest reliability, is essential for clinical utility. To address this gap, we assessed the test-retest reliability of manifest behavior and computational model parameters of a probabilistic reward and reversal learning task, two-armed Bandit, using intraclass correlations (ICCs) in 179 adults, including those with various psychosis-spectrum disorders and undiagnosed controls. We studied two computational models from recent literature: regression modeling of choice strategies and a hidden Markov model. The test-retest reliability for both manifest behavior (0.24 ≤ ICCs ≤ 0.54) and computational parameters (0.30 ≤ ICCs ≤ 0.61) ranged from poor to moderate, which was not explained by practice effects. Computational parameters did not outperform manifest behavior parameters. The reliability of computational parameters was generally-though not significantly-higher in healthy adults, which may potentially reflect the internal heterogeneity of categorical psychiatric diagnoses. Computational modeling holds promise, but tasks and analyses must be optimized for greater reliability before proceeding into clinical use. (PsycInfo Database Record (c) 2025 APA, all rights reserved).