Sociology 201A
Research Methods: Quantitative Approaches
Reading notes 3:  Research designs I

Reading guide:

Nachmias, chapter 5:  Research Designs: Experiments.

A research design is a logical model that guides data collection , analysis, and interpretation so as to draw a valid inference regarding the hypothesis.

The research design: An example.  Who, what, when, where, how?  Rosenthal's Pygmalion in the Classroom study.  20% of students chosen randomly, teachers told they were intellectually superior.  Dependent was change in IQ from pre-test to post-test.  Result showed significant mean differences between the groups with those who were expected to perform better, doing so.

The classic experimental design.  consists of two comparable groups, experimental and control, by random assignment.  With a pre-test and a post-test.
The Structure of the Classic Experimental Design.
Diagram with O for observations of Y, and X as intervention, R for random assignment.  Rosenthal experiment; electronic versus face-to-face groups experiment.
Why study experiments?  The classic experiment is a model to understand and compare against other designs.

Causal inferences.  The goal of an experiment is to infer a causal connection of change in X to change in Y.  Covariation, in itself, does not allow inference of causality.  Three conditions [in addition to a theoretical explanation] are needed to assure valid causal inference.
Covariation.  X and Y must covary in the direction and magnitude predicted.  "correlation" "association"
Nonspuriousness.    not an accidental connection; better: no omitted variable that is a cause of both X and Y.
Time Order. Change in X must precede consequent change in Y.

Components of a research design.  First three components are needed to establish causal (internally valid) inference.
Comparison.  Allows us to establish covariation.  Need for control group, or more generally, need for variation in X.
Manipulation.  Allows establishing time order.
Control: internal validity,  Allows establishing non-spuriousness.  Effort to rule out other causes by a) holding them constant or b) assigning them randomly to each level of X.  External threat:  It may not always be ethical or practical to randomly assign to groups -- hence possible "selection" effects.  Internal threats:   "History" change in Y occured as a result of something other than X.  "Maturation"  Change in Y is self-derived or trending, and would have occurred whether X changed or not.  "Experimental mortality"  differential loss makes X groups selected and non-comparable. "instrumentation"  change in measuring instruments generates change in result.  "testing"  measuring Y is a cause of change in Y.  "regression"  artifact -- extreme scores tend toward the average.  Interactions of the other factors with selection.

Procedures of control.  Matching may be used to try to make X groups equivalent -- either one-on-one case/control or group aggregate properties (by random selection or weighting) -- deals only with known factors; randomization deals with all unknown factors; use of a control group aids in solving internal validity problems -- history, maturation, instrumentation, testing can be controlled by using a control group.
Generalizability: external validity.  Main concerns are "representativeness" (sampling of cases) and "reactivity" of the experiment (lack of mundane realism).  random assignment is not the same as random sampling of subjects from some larger population.  Lack of mundane realism of the experiment may cause one to question generalizability. [n.b. the absence of multiple causes operating simultaneously also makes it impossible to generalize effects from an experiment to real-world conditions.  Experiments are good for determining that there is a causal effect, often useless for determining whether such an effect actually explains variation in real cases].

Design types.  Experimental designs allow comparison, manipulation, and control.  Quasi-experimental and cross sectional designs lack some of these elements (esp. manipulation and random assignment).  Pre-experimental designs also often lack comparison.
Controlled experimentation.  Classic design is an ideal type of a very internally strong design.
Solomon four-group design.  Adds two groups that are not pre-tested, so one can control testing effects.
Posttest only control group design. An alternative approach to controlling testing effects is to use two groups, but do no pre-testing.  
Experimental designs to study effects extended in time.  More groups can be added to be observed at subsequent time points if a trend study is needed. ("delayed effect" study).
Factorial Designs.  two study two or more manipulations, and often their interactions.  Factoral, simple crossed, design.  Use of more independent variables may improve generalizability.  They also allow the study of interaction.

Miller 2.4:  Elements of Research Design

Provides a nice checklist of issues to be taken into account in determining a choice of research design for a particular study:
how strong is the existing theory, how specific are it's requirements?
is the goal pre-experimental, experimental, or quasi-experimental
access to respondents
degree of control over the system being studied
type of data that are available
temporal dimension
sample or universe?
sample size
data source -- original, secondary, documentary?
method of data collection
number of independent variables
number of dependent variables
level of measurement of variables
existing scales and instruments?
distributional shape of dependent variable
duration of study
resources needed

Miller 2.5:  Choosing a Research design

What design will best enable one to observe the critical parameters of the theory (the causal paths)?  Start with strong design and strong control where possible.

There follows a table describing main characteristics and strengths of:  descriptive surveys and longitudinal surveys; sample surveys; field studies; case studies of persons; combined case and survey; prediction studies; controlled experiments (lab, natural, field).

Miller 2.13:  General considerations of research design.

Suchman:  "The Principles of Research Design.  A design is a "plan of study" there is no one best choice for all circumstances, and proof of hypotheses is never final.  All real research designs require compromise because of practical limitations.  Lastly, a research design is a plan -- it is not immutable, and must be changed as circumstances require.

Miller 2.14:  Factors Jeopardizing internal and external validity of research designs.  

Campbell and Stanley:  Internal validity is essential.  External validity is the less important.

Threats to internal validity (did, in fact, X have an effect on Y?)
1.  History
2.  Maturation
3.  Testing
4.  Instrument decay
5.  Statistical regression
6.  Bias of selection
7.  Survival bias
8.  Selection-maturation interaction, and other "blended threats"

Threats to External validity (ability to generalize results):
9.  Reactive or intervention effects
10.  Interaction of selection effects and treatment
11.  Reactive effects of experimental arrangements

[Note:  of course selection or sampling issues also dramatically affect external validity].