Course
materials for/by Peter L. Patrick. May
contain copyright material used for educational purposes. Please respect
copyright.
Notes for LG 554
Sociolinguistic Methods I
On reliability, validity, and representativeness
Reliability
Q:
Would this method work for someone else?
If methods are reliable, then – all other things being
equal – if anyone else used the method on the same group (or on a
representative sample of the same group), they would come up with the same
results; or if the same researcher used the method at another time (assuming no
change is in progress for the thing studied), or in a different representative
sample of the same group, she would come up with the same results.
This criterion is based on the experimental method in
science, and assumes that social behavior is governed by laws in a way similar
to the effect of physical forces and natural conditions (e.g. friction). That’s
not true, of course.
Some methods are seen as more reliable than others.
Anything that involves a single researcher in a situation that cannot be
repeated, or relies on fortuitous circumstances or remarkable personal
qualities of the researcher, is in danger of being thought unreliable – e.g.,
much of ethnographic participant-observation!
Validity
Q:
Do the data give a true picture of what I’m studying?
Evidence
is always partial, less than the whole. Does it distort the object of research,
or is it accurate? Note sense of “accurate” here – an arrow that lands close to
the bullseye – as opposed to “precise” – an arrow that lands close to the
previous shot (however far or near that may be from the bullseye!), a notion
that is akin to Reliability. Successive investigations of an object may agree,
i.e. be reliable/precise, yet all may be flawed in their view, i.e. not be
valid/accurate.
Self-report data are a good example.
How do we know that the answers people give to a survey Questionnaire about
some regular activity are in fact true? How many times a day do you look at
your watch? Your good-faith answer could, in fact, be far off the truth; you
may well not know the answer. Many or most surveys are best understood as
records of people’s attitudes towards behavior, or how they position
themselves regarding common norms - rather than as valid estimates of their
behavior. This is particularly important where language use is concerned,
because there are systematic aspects of language behavior which are not open to
introspection, and which run counter to most people’s beliefs about language
and about themselves.
Similarly,
with the experimental method we get a picture of how people behave in
experiments, but how do we know that this is how they behave in the real world?
Clearly there is a trade-off here: the more controlled
the environment, the more closely causes and effects can be related to each
other with confidence and things explained; but the more artificial, too, the
situation is. In real situations, independent variables cannot often be limited
or controlled, and we can never be sure that the cause-and-effect relationships
we’ve identified are the right ones.
Representativeness
Q: Is my
sample typical of the group I drew it from?
Representativeness assumes the non-uniqueness of
populations, the inter-changeability of their units for certain purposes. All
things being equal, the larger a population is, the better this assumption
holds up. Thus, sampling. If we’re not sure a sample
is typical, then we can’t generalize – we can’t be sure our study is relevant
to anyone else at all.
See further: Patrick
McNeill (1990) Research Methods, or any standard social science manual.
Last updated on 16 October
2007