Sponsor
This work was supported by National Institute on Deafness and Other Communication Disorders Grant R01DC015999 (Principal Investigators: Steven Bedrick and Gerasimos Fergadiotis).
Published In
Journal of Speech, Language, and Hearing Research
Document Type
Pre-Print
Publication Date
11-2023
Subjects
Aphasia -- research, Aphasia -- Case studies, Coherence (Linguistics)
Abstract
Purpose:
To date, there are no automated tools for the identification and fine-grained classification of paraphasias within discourse, the production of which is the hallmark characteristic of most people with aphasia (PWA). In this work, we fine-tune a large language model (LLM) to automatically predict paraphasia targets in Cinderella story retellings.
Method:
Data consisted of 332 Cinderella story retellings containing 2,489 paraphasias from PWA, for which research assistants identified their intended targets. We supplemented these training data with 256 sessions from control participants, to which we added 2,415 synthetic paraphasias. We conducted four experiments using different training data configurations to fine-tune the LLM to automatically “fill in the blank” of the paraphasia with a predicted target, given the context of the rest of the story retelling. We tested the experiments' predictions against our human-identified targets and stratified our results by ambiguity of the targets and clinical factors.
Results:
The model trained on controls and PWA achieved 50.7% accuracy at exactly matching the human-identified target. Fine-tuning on PWA data, with or without controls, led to comparable performance. The model performed better on targets with less human ambiguity and on paraphasias from participants with fluent or less severe aphasia.
Conclusions:
We were able to automatically identify the intended target of paraphasias in discourse using just the surrounding language about half of the time. These findings take us a step closer to automatic aphasic discourse analysis. In future work, we will incorporate phonological information from the paraphasia to further improve predictive utility.
Rights
Copyright the author(s) 2023
Locate the Document
DOI
10.1044/2023_JSLHR-23-00121
Persistent Identifier
https://archives.pdx.edu/ds/psu/40962
Citation Details
Published as: Salem, A. C., Gale, R. C., Fleegle, M., Fergadiotis, G., & Bedrick, S. (2023). Automating intended target identification for paraphasias in discourse using a large language model. Journal of Speech, Language, and Hearing Research, 1-18.
Description
This is the author’s version of a work that was accepted for publication in Journal of Speech, Language, and Hearing Research. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Journal of Speech, Language, and Hearing Research.