Consistency and Predictability of Short Stories Generated by LLMs (2024)
Undergraduate: Nicholas Sanaie
Faculty Advisor: Snigdha Chaturvedi
Department: Computer Science
Our research investigates the consistency and predictability of short story generation by Large Language Models (LLMs) by focusing on several narrative elements including character relationships, story emotion arcs and word choices. Our study uses the ROCStories and WritingPrompts datasets alongside short stories generated by GPT-3.5 to explore similarities and predictable patterns in machine generated stories in comparison to human created stories. Analysis of 500 stories from each dataset reveals intriguing patterns, such as less emotional arc variation in machine generated stories and predictable character sentiment evolution throughout the story. There are also slight differences between machine generated datasets. Our research also discusses the challenges of annotating emotional arcs and character sentiments through highlighting the discrepancies between human annotations and those generated by LLMs. One such finding is the preference in machine-generated annotations to confused “Man in a Hole” and “Rags to Riches” emotional arcs. This study contributes to the understanding of the capabilities and limitations of LLMs in narrative generation and also underscores the potential need for new approaches to enhance narrative diversity and creativity. Our research is still ongoing and we are currently collecting and verifying our results so far.