Rapid antibiotic susceptibility tests (ASTs) are an increasingly important part of clinical care as antimicrobial resistance (AMR) becomes more common in bacterial infections. Here, we use the spatial distribution of fluorescently labelled ribosomes to detect intracellular changes associated with antibiotic susceptibility in E. coli cells using a convolutional neural network (CNN). By using ribosome-targeting probes, one fluorescence image provides data for cell segmentation and susceptibility phenotyping. Using 60,382 cells from an antibiotic-susceptible laboratory strain of E. coli, we showed that antibiotics with different mechanisms of action result in distinct ribosome phenotypes, which can be identified by a CNN with high accuracy (99%, 98%, 95%, and 99% for ciprofloxacin, gentamicin, chloramphenicol, and carbenicillin). With 6 E. coli strains isolated from bloodstream infections, we used 34,205 images of ribosome phenotypes to train a CNN that could classify susceptible cells with 91% accuracy and resistant cells with 99% accuracy. Such accuracies correspond to the ability to differentiate susceptible and resistant samples with 99% confidence with just 2 cells, meaning that this method could eliminate lengthy culturing steps and could determine susceptibility with 30 min of antibiotic treatment. The ribosome phenotype method should also be able to identify phenotypes in other strains and species.