Joint Quantitative Brownbag

Speaker

Dr. Kylie L. Anglin

Dr. Kylie L. Anglin
Department of Educational Psychology
University of Connecticut

Given psychology is largely the study of internal processes—such as thoughts, feelings, and mental states– many psychological constructs are latent and cannot be observed directly. However, such constructs can often be inferred from client writing or speech. Thus, in studying psychological constructs, researchers often require high-quality, labeled, text data. This talk demonstrates a rigorous process for the use of pre-trained large language models (LLMs) to obtain such data, with zero-shot classification, few-shot classification, and fine-tuning. The approach is demonstrated through the application of LLMs to the identification of negative core beliefs and meaning making in expressive writing samples. Validity considerations will be discussed throughout the talk and Python code will be provided.

Title

Large Language Models and the Measurement of Psychological Constructs: Zero-Shot, Few-Shot, and Fine-Tuning Approaches

Abstract

Kylie Anglin is an Assistant Professor in Research Methods, Measurement, and Evaluation at the University of Connecticut. Her research aims to increase the quality and depth of evaluations of interventions and policies, most often through the rigorous use of artificial intelligence (AI). Her research provides tools for leveraging insights from data that are either difficult to locate or difficult to analyze using traditional means. In doing so, she addresses critical resource constraints in education research while simultaneously addressing risks in the use and interpretation of AI output. Her research has been funded by the Institute of Education Sciences (IES) and has been published in journals just as the Journal of Educational and Behavioral Statistics, Educational Evaluation and Policy Analysis, AERA Open, the Journal of Research on Educational Effectiveness, and Prevention Science.