Quantitative Evaluation of AI Writing Tools: Insights from Likert Scale Responses

Written by teleplay | Published 2024/05/23
Tech Story Tags: ai-generated-scripts | hierarchical-language-models | human-machine-co-creativity | dramatron | ai-writing-tools | language-model-evaluation | future-of-ai | likert-scale-evaluation

TLDRSupplementary figures showcase participant responses in the AI writing evaluation, categorized by experience with AI tools and expertise in different domains like improvisation, scripted theatre, and film/TV, offering insights through Likert scale analysis.via the TL;DR App

Authors:

(1) PIOTR MIROWSKI and KORY W. MATHEWSON, DeepMind, United Kingdom and Both authors contributed equally to this research;

(2) JAYLEN PITTMAN, Stanford University, USA and Work done while at DeepMind;

(3) RICHARD EVANS, DeepMind, United Kingdom.

Table of Links

Abstract and Intro

Storytelling, The Shape of Stories, and Log Lines

The Use of Large Language Models for Creative Text Generation

Evaluating Text Generated by Large Language Models

Participant Interviews

Participant Surveys

Discussion and Future Work

Conclusions, Acknowledgements, and References

A. RELATED WORK ON AUTOMATED STORY GENERATION AND CONTROLLABLE STORY GENERATION

B. ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM

C. DETAILS OF QUANTITATIVE OBSERVATIONS

D. SUPPLEMENTARY FIGURES

E. FULL PROMPT PREFIXES FOR DRAMATRON

F. RAW OUTPUT GENERATED BY DRAMATRON

G. CO-WRITTEN SCRIPTS

D SUPPLEMENTARY FIGURES

Figure 7 shows the participants’ responses to the quantitative evaluation, on a Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree), and broken down by groups of participants. For the first group, we defined a binary indicator variable (Has experience of AI writing tools). For the second group, we defined a three-class category for their primary domain of expertise (Improvisation, Scripted Theatre and Film or TV).

This paper is available on arxiv under CC 4.0 license.


Written by teleplay | From teleplay to technology, we weave a narrative tapestry that dances between writing, CGI, and "action!"
Published by HackerNoon on 2024/05/23