Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT) [64], which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose Torrance Test of Creative Writing (TTCW) to evaluate creativity as product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.

The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions has led to increased interest in their use across various support tools. We investigate the effectiveness of contemporary LLMs in assisting professional writers via an empirical user study (n=30). The design of our collaborative writing interface is grounded in the cognitive process model of writing [17]. This allows writers to obtain model help in each of the three non-linear cognitive activities in the writing process: planning, translating and reviewing. Participants write short fiction/non-fiction with model help and are subsequently asked to submit a post-completion survey to provide qualitative feedback on the potential and pitfalls of LLMs as writing collaborators. Upon analyzing the writer-LLM interactions, we find that while seeking help across all three types of cognitive activities, writers find LLMs more helpful in translation and reviewing. Our findings from analyzing both the interactions and the survey responses highlight future research directions in creative writing assistance using LLMs.

Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, Smaranda Muresan. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Contribution: Demonstrates how to use experiential learning (EL) to improve argumentative writing. Presents the design and development of a natural language processing (NLP) application for aiding instructors in providing feedback on student essays. Discusses how EL combined with automated support provides an analytical approach to improving written-communication skills. Background: High-quality, timely, feedback is an effective way to improve students’ writing. However, large class sizes and limited instructor backgrounds often make formative feedback impossible. Recent trends, including lowering entry requirements, have added to these challenges. Assistive technologies for implementing inclusive education provide viable solutions. Research Questions: 1) How and why can EL be used to develop argumentative writing skills in university STEM students? 2) How can technologies be developed to support using EL in teaching writing? and 3) How might the holistic impact of using such analytic techniques be evaluated? Methodology: Participants in an EL project were assigned two essays in sequence. They were given instructions on making good arguments and shown how to use an analytic rubric to maximize their scores. The essays were hand scored by tutors who provided scores for each dimension of the rubric. Subsequently, the content and argumentation of the essays were analyzed using NLP techniques to obtain independent scores. Qualitative data were also collected. Findings: The project produced transformative writing experiences for the participants. It showed how analytical techniques help improve writing skills and how relevant automated instructor assistance can be developed using NLP technologies.

Ramy Eskander, Cass Lowry, Sujay Khandagale, Judith Klavans, Maria Polinsky, Smaranda Muresan. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Large language models (LLMs) constitute a paradigm shift in Natural Language Processing and its applications across all domains. To move towards human-centric NLP designed for social good and responsible computing, I argue we need knowledge-aware NLP systems and human-AI collaboration frameworks. NLP systems that interact with humans need to be knowledge aware (e.g., linguistic, commonsense, sociocultural norms) and context aware (e.g., social, perceptual) so that they communicate better and in a safer and more responsible fashion with humans. Moreover, NLP systems should be able to collaborate with humans to create high-quality datasets for training and/or evaluating NLP models, to help humans solve tasks, and ultimately to align better with human values. In this talk, I will give a brief overview of my research agenda in the context of NLP for social good and responsible computing (e.g., misinformation detection, NLP for education and public health, building NLP technologies with language and culture diversity in mind). I will highlight key innovations on theory-guided and knowledge-aware models that allow us to address two important challenges: lack of training data, and the need to model commonsense knowledge. I will also present some of our recent work on human-AI collaboration frameworks for building high-quality datasets for various tasks such as generating visual metaphors or modeling cross-cultural norms similarities and differences.