Week 13 Experimentation, Surveys, and Questionnaires

This week, our focus turns to experimentation, surveys, and questionnaires. In today’s discussion, we’ll delve into various research designs, starting with an examination of experimental research. Following that, we’ll explore surveys and questionnaires before summarizing the key points of today’s session.

Before delving into our main topics, let’s review some pertinent terms relevant to experimentation and surveys. Firstly, let’s consider variables. Variables can be categorized as independent variables, which serve as predictors, and outcome variables, which are the factors under investigation or prediction. These variables further classify into nominal, categorical, ordinal, and numeric variables.

Nominal variables are categorical with only two categories, such as “on” or “off” or “yes” or “no.” Categorical variables encompass more than two categories, like nationalities, and are unordered. When dealing with an ordered categorical variable, it’s known as an ordinal variable. Numeric variables, on the other hand, are measured in numerical values, such as reaction times or age. Likert scale items are variables measured on an ordinal scale, usually ranging from one extreme to another, with various options in between.

Moving on, we’ll touch upon the concepts of population and sample. A sample is a subset drawn from a population, representing elements or individuals that reflect the larger population. Inferential statistics allows us to make educated guesses about population parameters based on sample data.

Lastly, let’s discuss control, test, or experimental groups. In experimental settings, a control group receives traditional treatment or instruction, while a test or experimental group is subjected to a new method or intervention. By comparing the outcomes between these groups, researchers can evaluate the effectiveness of the new approach. Understanding these foundational concepts is vital for designing and interpreting research studies in the fields of experimentation and survey methodology.

We previously discussed stimuli and response in our conversation. Stimuli encompass anything presented to a subject or participant in an experiment, such as sentences heard, pictures viewed, or objects manipulated. Response refers to the participant’s reaction or behavior, such as button presses, verbal responses, or eye movements, elicited by the stimuli.

Now, let’s transition to research designs. The table presented illustrates various research designs commonly employed in the language sciences, categorized into experimental and observational research. Experimental research uniquely allows for establishing cause and effect relationships, whereas observational research only identifies correlations. Notably, experimental designs offer direct access to phenomena and enable causal inferences, unlike observational approaches.

Among the listed research designs, experimental research often involves Reaction Time measurements, where participants respond to stimuli by pushing buttons to indicate true or false statements, allowing for reaction time measurement. Behavioral studies, like corpus analyses, involve observing and transcribing natural behavior. Archival or review research, akin to literature reviews, involves systematically reviewing existing literature on a topic. Introspection involves self-assessment based on grammatical knowledge, while questionnaires solicit specific information or opinions from participants, such as language use surveys.

Each research design carries distinct advantages and disadvantages. For instance, experimental research offers high directness but is labor-intensive and may lack external validity due to controlled settings. Conversely, observational approaches capture natural behavior but may lack control and experimental manipulation. Experimentation aims to elicit specific participant performance in a controlled setting to understand behavior, with the goal of applying findings to the natural world.

Setting: Now, there are different experimentation types, right? So you have Behavioral Studies which look at conscious performance, and that would be, for example, making judgments about the stimulus and indicating a response by pressing a button. So there, basically, you really ask the participants to press a button or to give their responses about something. And so, responding to a stimulus with a certain option like a greeting response or selecting names for objects or descriptions of pictures would be, you know, that type of behavior that you want to look at. So those would be Behavioral Studies because you’re interested in the behavior of people, and it’s conscious behavior mostly.

Then you have neuroscientific studies which look at unconscious performance. So the participant is presented with a stimulus, but their underlying neural mechanisms are recorded and studied. So it’s not their behavior; it’s really what their brain does and what is really out of the conscious control more or less of the participant. And there, you have neuroimaging techniques like fMRI where you analyze the blood flow in certain regions of your brain, and that tells you where the brain is active because you know when it’s active, it needs blood because the cells require oxygen. And then you have electrophysiological techniques like EEG where you measure the electric activity of brain cells.

Now, because we’re in the language sciences, we also need to talk about experimentation in speech production specifically. Speech production plays, of course, a crucial role in communication, but it also serves as a tool for self-identification. And when we speak, it’s actually very cognitively demanding, so when we’re speaking, we’re actually not good at doing a lot of other things. So that’s why when you’re driving a car, sometimes it’s not optimal to talk to other people about complex issues because that will take away your ability to concentrate on your surroundings and on driving.

Now, when you design a production task where you want to elicit behavior or speech from participants, you need to take into account the interactivity. For example, you have to think about if your subjects are only there to produce or if they are there to interact. That is something that you really have to think about. Are you interested only in the production itself, or are you interested, for example, in how people behave or produce utterances in a conversation? So is it interactive or not? Is it conversational or is it individual performance?

Then, also, you have to take into consideration the spontaneity. Whether you want natural speech or elicited speech, like, do you want, for example, your subjects to just come up with what they want to talk about, or do you want them to say certain words or repeat certain words or sentences? And also, of course, you have to think about the presentation of stimuli, like what type of stimuli do you want to present to the subjects? Should it be picture prompts or text audio stimuli? So, for example, where they listen to something and then they, for example, have to repeat it or respond to Something. Um, here’s an example of a reading aloud study, and there you can, for example, present participants typically with individual words that they are supposed to read aloud or minimal pairs or sentences or paragraphs and larger texts. So when you have a reading aloud study, then you have typically these types of stimuli, either individual words or minimal pair sentences or paragraphs and larger texts.

Now, the advantages and disadvantages of these reading aloud studies is that they generate uniform data sets by controlling for phonological, for example segmental context or prosodic properties, and nonphonological like lexical choice, syntactic complexity variables. So basically, by giving people things like individual words or minimal pairs or sentences that they are supposed to repeat, then basically all the subjects will provide the same data, right? So you can basically control for these other features like phonological and non-phonological factors. So that’s a very efficient way to acquire the performance that you’re interested in, right? So it’s efficient in terms of the amount of required time to elicit all required performance because basically, you can design the stimuli in a certain way so that you really get what you’re interested in. And when you just ask people to speak, they might not actually produce what you’re interested in. So when you have people read aloud, then basically you can perfectly control what they will say.

Now, the problem is that, of course, it elicits typically high-performance and natural speech because when people read, it’s not really natural. They don’t read the way that they typically read at home or that they speak. And the problem is that most of the time when we read, we don’t read aloud, right? Unless we read for someone like for your kids, for example, right?

Another option to elicit speech production are map tasks. And there, for example, you have people who are typically separated by a screen, right? And then the one person describes how the other person needs to get to a certain target, and sometimes the maps are actually not identical, so they don’t have the same things on the map, right? And so that’s really interesting because then the participants need to basically negotiate what’s on the maps that they have in front of them. So that’s actually quite ideal for discourse analysis because it requires a lot of linguistic cooperation, right? So there’s a lot of back and forth and turn-taking in those map task experiments. So when you’re interested in turn-taking and people producing turns and communicating with each other, then map tasks are actually a very good way to analyze.

Speech. Now, there’s another type of experimentation which is not speech production but speech perception, right? And here, a couple of things to consider. In speech perception, we translate sounds into meaning, and in real time, that means that speech is continuous, of course, right? So when you listen to me, I produce different sounds all the time, right? So my speech is essentially continuous. I’m sorry for that. So the listener then decodes each sound while new sounds are being produced. So, that’s actually quite demanding, and whenever we encounter a new speaker or when there’s a new term, all that begins again. So you have this continuous stream of speech, and then you have to decode the meaning.

Also, when we listen to someone, we form assumptions not only about them but we also monitor the conversation for appropriate responses or, for example, we also monitor the conversation for conversational opportunities. So, for example, when is it my turn? When can I say something? And then we prepare our responses. So actually, when you look at what happens in a conversation, it’s really quite interesting because there’s so much going on. It’s like you decode the speech stream of these continuous sounds that are produced, you derive meaning from them, you analyze the speaker, you analyze the intentions, you analyze when there’s a pause in what they’re saying or when they are going to allow you to say something, right? So when you can take over the turn for Speech Perception. Now, there’s another type of experimentation which focuses on speech perception, distinct from speech production. In speech perception, it’s crucial to determine the specific aspect of perception you want to investigate. For instance, are you interested in the perception of sounds, also known as phones, or in the perception of social meaning as perceived by the listener? Here, we have an example of someone saying a word like “ship”, and then the listener may need to decipher if it means “ship” or “sheep”, distinguishing between a short and long ‘e’ sound. This illustrates the difference between the phonological level of analysis, concerned with sound, and the sociolinguistic level, concerned with the social meaning conveyed by the speaker’s characteristics.

Another approach in speech perception experiments is matched guise tasks, commonly used in social evaluation studies. In these tasks, a listener hears recordings of the same speaker using different speech styles. For instance, one guise might feature standard speech inflection, while another uses non-standard pronunciation. The listener then evaluates attributes of the speaker based on these variations. This methodology reveals insights into how different speech styles influence perceived attributes such as intelligence or education level.

Moreover, experiments in speech perception can involve word learning, assessing whether learners have acquired correct grammar or an appropriate vocabulary for their age. These studies are valuable in developmental linguistics and speech disorder research, shedding light on the underlying issues contributing to impaired language abilities.

An intriguing example is the word test, where participants view a picture of a “work” and are asked to indicate the plural form. This task assesses understanding of pluralization rules and can be used to evaluate language proficiency in various populations, such as children or second language learners.

Transitioning to surveys and questionnaires, these research tools offer a convenient means of gathering data, yet they present challenges. While they provide ample data quickly, they lack direct access to the phenomenon under study, potentially leading to inaccuracies or biased responses.

When crafting questionnaires or surveys, several considerations are paramount. First, obtaining ethics approval is essential to ensure participant welfare and data integrity. Additionally, clearly informing participants about the study purpose, duration, and contact information for inquiries is crucial to obtaining informed consent.

Furthermore, questionnaires and surveys should be concise to maintain participant engagement and minimize fatigue. Randomizing question order and checking for response patterns can mitigate response biases. Pilot testing is also indispensable for refining questionnaire clarity and effectiveness before wider distribution.

In summary, experiments offer unparalleled insights into causal relationships but demand meticulous planning and resources. On the other hand, surveys and questionnaires provide efficient data collection but require careful design and validation to ensure accuracy and reliability. Understanding the strengths and limitations of each approach is essential for conducting rigorous research in language sciences.

I hope you found this lecture informative. Thank you for your attention and please feel free to explore the next section.