Overview
This assignment will help you design an experiment where AI functions as a research participant or is compared to human behavior using existing research methods. You will integrate theory, identify research methods used in studies of human behavior, and apply them to experiments involving AI or large language models (LLMs). This assignment builds on your work in Applied Assignments 1 and 2 by focusing on experimental research design.
Instructions:
Step 1: Theoretical Foundation
- Choose a Theory: Start by selecting a theory from consumer behavior, psychology, or economics that you think could also influence AI behavior. You can use the course materials or search through academic journals on Google Scholar, following the methods outlined in Assignment 2.
- Find a Related Study: Identify a published research paper that uses this theory to explain human behavior. Focus on studies that use experiments with survey instruments to test behaviors such as trust, decision-making, or reciprocity.
Step 2: Survey Instrument Identification
- Examine the Study’s Methods: Look closely at the survey methods, sample questions, and experimental design used in your selected study. These will form the basis for your AI experiment.
- Survey Adaptation: Think about how these questions might need to be adapted for use with an AI language model. For example, if the study used questions that measured human trust in other humans, how would you measure trust in AI interactions?
Step 3: Designing the Experiment
- Formulate Research Questions and Hypotheses: Translate your theoretical framework into research questions and hypotheses that test AI behavior. Examples could include testing whether AI responds to certain types of prompts with more accurate or cooperative responses.
- Select AI Participants: Decide whether you want to use one or two AI models (e.g., GPT-4 vs. GPT-3.5 or another LLM) and explain the rationale for your choice. How do you expect the models to differ in behavior?
- Experimental Conditions: Define the conditions under which you will test the AI. For example, will you prompt one AI with questions related to trust, while asking a different set of questions to the other model? Will you vary the prompts or response conditions?
Step 4: Sampling Method
- Select a Sampling Method: While your "participants" may be AIs, consider whether you want to use specific instances of AI (e.g., different prompts or temperatures) and how many trials you will run. Discuss the reasoning behind your sample size and method of generating responses.
Deliverables:
- Report Length: 2-4 pages (double-spaced, 12pt font, 1-inch margins) outlining the following:
- Theory and Related Study: Explanation of the theory and the study it is based on, including the survey instrument details.
- Experimental Design: Your research questions, hypotheses, and experimental conditions.
- Sampling Method: Explanation of how you will sample AI responses, including rationale for sample size and method.
- Reflection on Adaptation: Discuss any challenges you faced adapting a human behavior study to an AI context.