Generative artificial intelligence (AI), with its increasing ubiquity and power, will likely transform forensic psychiatry, sparking both advances and new challenges for the field. A possible consequence of the technology is that it will be used to assist malingerers in learning about and feigning psychiatric symptoms. In this study, the AI chatbot ChatGPT was asked to provide information about the insanity defense and psychosis and to use this information to assist the user in simulating a psychotic illness to avoid legal consequences. We found that ChatGPT 3.5 demonstrated a relatively nuanced understanding of typical symptoms of psychosis and that it could translate that knowledge into practical guidance on how to exploit the mental health system for secondary gain. Our findings suggest that, although significant limitations exist with the technology in its current form, forensic psychiatrists should be prepared for its increasing sophistication and the potential consequences in malingering assessments.