UNLABELLED: This study introduces a new framework for the artificial intelligence-based characterization of Gram-stained whole-slide images (WSIs). As a test for the diagnosis of bloodstream infections, Gram stains provide critical early data to inform patient treatment in conjunction with data from rapid molecular tests. In this work, we developed a novel transformer-based model for Gram-stained WSI classification, which is more scalable to large data sets than previous convolutional neural network-based methods as it does not require patch-level manual annotations. We also introduce a large Gram stain data set from Dartmouth-Hitchcock Medical Center (Lebanon, New Hampshire, USA) to evaluate our model, exploring the classification of five major categories of Gram-stained WSIs: gram-positive cocci in clusters, gram-positive cocci in pairs/chains, gram-positive rods, gram-negative rods, and slides with no bacteria. Our model achieves a classification accuracy of 0.858 (95% CI: 0.805, 0.905) and an area under the receiver operating characteristic curve (AUC) of 0.952 (95% CI: 0.922, 0.976) using fivefold nested cross-validation on our 475-slide data set, demonstrating the potential of large-scale transformer models for Gram stain classification. Results were measured against the final clinical laboratory Gram stain report after growth of organism in culture. We further demonstrate the generalizability of our trained model by applying it without additional fine-tuning on a second 27-slide external data set from Stanford Health (Palo Alto, California, USA) where it achieves a binary classification accuracy of 0.926 (95% CI: 0.885, 0.960) and an AUC of 0.8651 (95% CI: 0.6337, 0.9917) while distinguishing gram-positive from gram-negative bacteria. IMPORTANCE: This study introduces a scalable transformer-based deep learning model for automating Gram-stained whole-slide image classification. It surpasses previous methods by eliminating the need for manual annotations and demonstrates high accuracy and generalizability across multiple data sets, enhancing the speed and reliability of Gram stain analysis.