We provide a finite sample inference method for the structural parameters of a semiparametric binary response model under a conditional median restriction originally studied by Manski (1975, 1985). Our inference method is valid for any sample size and irrespective of whether the structural parameters are point identified or partially identified, for example due to the lack of a continuously distributed covariate with large support. Our inference approach exploits distributional properties of observable outcomes conditional on the observed sequence of exogenous variables. Moment inequalities conditional on this size n sequence of exogenous covariates are constructed, and the test statistic is a monotone function of violations of sample moment inequalities. The critical value used for inference is provided by the appropriate quantile of a known function of n independent Rademacher random variables. We investigate power properties of the underlying test and provide simulation studies to support the theoretical findings.