BACKGROUND: People with schizophrenia often present with cognitive impairments that may hinder their ability to learn about their condition. Education platforms powered by Large Language Models (LLMs) have the potential to improve accessibility of mental health information. However, the black-box nature of LLMs raises ethical and safety concerns regarding the controllability over chatbots. In particular, prompt-engineered chatbots may drift from their intended role as the conversation progresses and become more prone to hallucinations. OBJECTIVE: To develop and evaluate a Critical Analysis Filter (CAF) system that ensures that an LLM-powered prompt-engineered chatbot reliably complies with predefined its instructions and scope while delivering validated mental health information. METHODS: For a proof-of-concept, we prompt-engineered an educational schizophrenia chatbot powered by GPT-4 that can dynamically access information from a schizophrenia manual written for people with schizophrenia and caregivers. In the CAF, a team of prompt-engineered LLM agents are used to critically analyze and refine the chatbot's responses and deliver real-time feedback to the chatbot. To assess the ability of the CAF to re-establish the chatbot's adherence to its instructions, we generate three conversations (by conversing with the chatbot with the CAF disabled) wherein the chatbot starts to drift from its instructions towards various unintended roles. We use these checkpoint conversations to initialize automated conversations between the chatbot and adversarial chatbots designed to entice it towards unintended roles. Conversations were repeatedly sampled with the CAF enabled and disabled respectively. Three human raters independently rated each chatbot response according to criteria developed to measure the chatbot's integrity
specifically, its transparency (such as admitting when a statement lacks explicit support from its scripted sources) and its tendency to faithfully convey the scripted information in the schizophrenia manual. RESULTS: In total, 36 responses (3 different checkpoint conversations, 3 conversations per checkpoint, 4 adversarial queries per conversation) were rated for compliance with the CAF enabled and disabled respectively. Activating the CAF resulted in a compliance score that was considered acceptable (≥2) in 67.0% of responses, compared to only 8.7% when the CAF was deactivated. CONCLUSIONS: Although more rigorous testing in realistic scenarios is needed, our results suggest self-reflection mechanisms could enable LLMs to be used effectively and safely in educational mental health platforms. This approach harnesses the flexibility of LLMs while reliably constraining their scope to appropriate and accurate interactions.