Human vs AI prompts: Exploring differences in grammatical choices of writer's intent
Presentation Type
Abstract
Faculty Advisor
Libby Barak
Access Type
Event
Start Date
25-4-2025 10:30 AM
End Date
25-4-2025 11:29 AM
Description
Previous research points to key differences between human and ChatGPT-generated prompts such as rhetorical and lexical choices (Reinhart et al., 2024). This study aims to analyze grammatical differences using Part-of-Speech (POS) as a metric of writer’s intent variation (Biber, 1988). We explored a dataset of human-generated questions from Quora and their ChatGPT-paraphrased prompts (Vorobev, 2023). We tokenized the data and assigned a POS tag for each token while preserving sentence boundaries to keep grammatical structure intact. Then, we extracted POS unigrams, bigrams, and trigrams (e.g. NN, JJ NN, and PRP VBD TO for one, two, and three POS tags respectively) to analyze differences in verb-structure aspects in relation to writer’s intent. Results suggested that humans write in a much more involved style, according to Biber’s Multidimensional Analysis, due to a larger number of present participles and infinitives compared to AI prompts. Human prompts are also more narrative than AI Prompts, as indicated by the commonality of the bigram PRP VBD (Personal-pronoun followed by past-tense participle verb). Meanwhile, AI prompts rely less on context to generate responses, but instead elaborate much more than human writers, e.g. bigrams such as NN NNS concretely explain nouns by pairing them with other nouns. Such bigrams also exemplify the prevalent abstract writing style of AI prompts, while human writing contains more adverbs and modalities which can be attributed to an overt-persuasive style. In current work-in-progress, we pursue a more fine-grained approach to POS distribution to further analyze its role in AI prompt engineering.
Human vs AI prompts: Exploring differences in grammatical choices of writer's intent
Previous research points to key differences between human and ChatGPT-generated prompts such as rhetorical and lexical choices (Reinhart et al., 2024). This study aims to analyze grammatical differences using Part-of-Speech (POS) as a metric of writer’s intent variation (Biber, 1988). We explored a dataset of human-generated questions from Quora and their ChatGPT-paraphrased prompts (Vorobev, 2023). We tokenized the data and assigned a POS tag for each token while preserving sentence boundaries to keep grammatical structure intact. Then, we extracted POS unigrams, bigrams, and trigrams (e.g. NN, JJ NN, and PRP VBD TO for one, two, and three POS tags respectively) to analyze differences in verb-structure aspects in relation to writer’s intent. Results suggested that humans write in a much more involved style, according to Biber’s Multidimensional Analysis, due to a larger number of present participles and infinitives compared to AI prompts. Human prompts are also more narrative than AI Prompts, as indicated by the commonality of the bigram PRP VBD (Personal-pronoun followed by past-tense participle verb). Meanwhile, AI prompts rely less on context to generate responses, but instead elaborate much more than human writers, e.g. bigrams such as NN NNS concretely explain nouns by pairing them with other nouns. Such bigrams also exemplify the prevalent abstract writing style of AI prompts, while human writing contains more adverbs and modalities which can be attributed to an overt-persuasive style. In current work-in-progress, we pursue a more fine-grained approach to POS distribution to further analyze its role in AI prompt engineering.
Comments
Poster presentation at the 2025 Student Research Symposium.