Internal Validity Tutorial


myAU\|	Login

i. Experimenter Bias

Expectations of an outcome by persons running an experiment may significantly influence that outcome. As with instrumentation, the reliability of the instrument used to gauge the dependent variable or deliver the independent variable is suspect, but here the reason for that unreliability is the impartiality of persons in direct contact with the subjects or the data. Suppose that subjects in two comparison groups differ with respect to the independent variable. Suppose also that the experimenter is responsible for administering the appropriate condition to each group and measuring the dependent variable. If scores on the dependent measure differ between the two groups, the discrepancy may be due to the independent variable or to differential treatment of the two groups by the experimenter who is under the influence of his or her hypothesis.

Background Information

Example

The name of each child in the classes was written on a separate slip of paper. All the slips were put in a bowl and mixed up thoroughly. Students were assigned to the Experimental Group and to the Control Group alternately as their names were pulled out of the bowl one at a time. One day at school, children in the Control Group were told to go to one room and children in the Experimental Group to another room, where they were exposed to their respective conditions. Two days later, the Generalization Probe was conducted. For ease of record keeping, all the Control Group children were tested first, then all the Experimental Group children. Both the student teacher, who recorded how the children responded to the confederate's lures, and the confederate who presented the lures, were heavily involved in the production of the interactive video, and both of them strongly believed in its efficacy. The mean score for children in the Control Group was 1.2 and the mean score for children in the Experimental Group was 3.4. We can conclude that the 20-minute interactive video was effective in changing what the children did in a potential abduction situation.

Nonexample

The name of each child in the classes was written on a separate slip of paper. All the slips were put in a bowl and mixed up thoroughly. Students were assigned to the Experimental Group and to the Control Group alternately as their names were pulled out of the bowl one at a time. One day at school, children in the Control Group were told to go to one room and children in the Experimental Group to another room, where they were exposed to their respective conditions. Two days later, the Generalization Probe was conducted, in which children were selected from class to be tested in random order. Both the student teacher, who recorded how the children responded to the confederate's lures, and the confederate who presented the lures, were heavily involved in the production of the interactive video, and both of them strongly believed in its efficacy. Only the experimenter knew which child was exposed to which condition. The mean score for children in the Control Group was 1.2 and the mean score for children in the Experimental Group was 3.4. We can conclude that the 20-minute interactive video was effective in changing what the children did in a potential abduction situation.

Analysis

The first item is an example in which experimenter bias is a threat to internal validity. We assume that both the student teacher and the confederate knew the experimental status of each child, given that they tested the Control Group first. Because both the student teacher and the confederate had a stake in the outcome, they may have inadvertently (or not) treated the two comparison groups differently which, in turn, would affect the results. For example, given a tough choice of deciding between a score of two or three for any particular child, the student teacher may be more likely to assign a score of two to a Control Group child and a score of three to an Experimental Group child. The confederate may be more persistent when attempting to lure Control Group children than when luring Experimental Group children. The higher Generalization Probe score by the Experimental Group may be due to exposure to the interactive video or to differential treatment of children in the two groups by the graduate students because of their biases.

In the second item, given that neither the student teacher nor the confederate was aware of whether a child was in the Control Group or the Experimental Group, they could not treat the children in the two groups differently, despite their expectations. We can be more confident that the higher Generalization Probe score by the Experimental Group was not the result of the biases of the persons running the experiment. (Of course it is possible that each child may have given subtle clues as to his or her experimental status. As a further safeguard against experimenter bias, the role of the student teacher and the confederate should be played by persons who have no knowledge of or stake in an expected outcome.)

You have reached the end of Part 1 of this tutorial. You may now proceed to Part 2, a 36-item exercise that will give you practice assessing the internal validity of hypothetical experiments. For those experiments that are not internally valid, you will be asked to go one step further and identify the source of the threat.

Proceed to the exercise...

i. Experimenter Bias

Next

Copyright © 1998 Athabasca University