We piloted using Large Language Models (LLMs) for organizing AMIA 2024 Informatics Summit. LLMs were prompt engineered to develop algorithms for reviewer assignments, group presentations into sessions, suggest session titles, and provide one-sentence summaries for presentations. These tools substantially reduced planning time while enhancing the coherence and efficiency of conference organization. Our experience shows the potential of generative AI and LLMs to complement human expertise in academic conference planning.