Critical Thinking Across the Curriculum: Empirical Argument Analysis

Longview Community College, Lee's Summit, Missouri

Critical Thinking
Across the
Curriculum Project

Analyzing Statistical and Causal Arguments

Contributed by Michael Connelly, Longview Community College.

Evaluating Statistical Arguments:

Start with "Simple Statistical Statements"
These statements have the following features:

They refer to a group of items as the subject matter of the statement - in short, the POPULATION.
They say something about the members of the Population - they ascribe some PROPERTY to the members.
They claim that some PERCENTAGE of the Population has the Property referred to in the statement.

Thus, the standard form for Simple Statistical Statements is:

Percentage of the Population has Property .

Statistical Statements (Premises) are usually the conclusion of a Statistical or Survey argument. These sorts of arguments rely on a SAMPLE of a larger Population to draw their conclusions. The strongest sort of Statistical Argument is where the SAMPLE POPULATION (the population actually observed or surveyed) is identical to the TARGET POPULATION (the group you are drawing conclusions about). Since this is often impractical, the Sample is often smaller than the Population being generalized about.

We will also need to distinguish between the MEASURED PROPERTY and the TARGET PROPERTY. The Measured Property is the property of the Sample Population that was actually looked at. The Target Property is the property of the Target Population that the generalization represents.

For example, consider the following argument:

The administration wants to find out how much a particular computer lab is being used. They set up a tracking program which records when a computer is used in the lab, and how many are being used. They track the usage for two weeks and find that the data shows that a computer is on in the lab for 75 of the 100 hours a week it is open, and that 20 or more computers are on for 50 of those hours. They conclude that the lab is being used 75% of the time it is open, and that 50% of the time it is open it is being used heavily. (Given a capacity of 30 computers).

In this argument, the Measured property is the same as the Target property. Now consider this argument:

The administration wants to find out how much a particular computer lab is being used. They set up a table in the Campus Center and ask as many students as they can if they use the computer lab, and how often. Of the 200 students who responded, 150 said they used the lab at least once a week, and 50 said they did not use the lab at all. Using this data, They conclude that the lab is used by about 75% of the students at least once a week.

In this argument, the Measured Property is not the same as the Target Property. Saying you use the lab and actually doing so are two different things. Buried in this argument is the assumption that the students are accurately reporting their use of the lab.

With these distinctions, we can generate a preliminary version of a standard pattern for statistical arguments:

Description of sample (the particulars)
Results of sample
Conclusion about sample
Final conclusion (the Generalization)

In evaluating Statistical Arguments, we will focus on three areas:

The accuracy of survey - is the measured property a good indicator of the target property?
The representativeness of the sample - is the sample population truly representative of the target population?
The reporting of the results - are the results of the sample reported in a way which mis-represents the data collected? (what I call gee-whiz graphs)

Accuracy in the survey:

There are at least three ways the accuracy of the survey can be compromised:

Slanted questions
Dishonest answers
Inaccurate tests

Representativeness of the Sample:

There are many ways in which the Sample Population could not represent the Target Population:

Small samples - usually a small sample will not be representative of a larger population
Unrepresentative samples - the sample population is not demographically similar to the target population
random samples - phone surveys
stratified random samples - mail surveys, Nielsens
significant margin of error - if margin of error is +3%, and the difference between groups (candidates) is 4%, then the results are not statistically significant.

Reporting the results:

Many times the results of statistical analysis will be reported using graphs and charts. There are visual ways of presenting information about a sample population which will lead us to one conclusion, while a more accurate presentation of the information would not do so. For some examples of these sorts of graphs, check out the Web site of USA Today under "snapshots", or look at these prime examples of "Gee Whiz Graphs".

Evaluating Causal arguments:

In examining causal arguments, we first must note that all causal statements claim that some event A is responsible for bringing about some other event B. Thus we say that A caused B to occur, the implication being that had A not occurred, B would not have. How is the truth of such claims established? The most important thing to note about causal statements is that we never actually observe one event causing another event. What we do observe is that one event (B) often or always occurs shortly after another event (A) occurs. What we observe is a constant correlation between occurrences of A and B.

We note this not to make us skeptical about causal claims, but rather to point out that we can only verify them via observational evidence. In simple terms, we note that the events A and B are positively correlated. There are four possible ways to explain this correlation:

A causes B
B causes A (the causal influence is reversed)
A and B are both caused by some other event C (common causes)

The correlation between A and B is coincidental (no causal connection)

The analysis (and thus the veracity) of causal claims will focus around eliminating the explanations 2-4, leaving explanation 1 as the best alternative.
We will then wish to reconstruct the general form of a causal argument as follows:

P1: A is positively correlated with B
P2: If A is positively correlated with B, then either:

A causes B, or
the causal factors are reversed (B causes A), or
the correlation is the result of a common cause, or
the correlation is a coincidence.

P3: The causal factors are not reversed.
P4: The correlation is not the result of a common cause.
P5: The correlation is not a coincidence.
_______________________________
Q.E.D.- A causes B

The work in this analysis is proving the claims made by premises 3-5. Premise 3 can be established by observing the temporal sequence of the events. If A always happens before B in time, then it is a good bet that the causal relationship is not reversed. Premise 4 will require a bit more trouble - this is the work of science - eliminating the possibility of previous common causes. This is where we get experiments which attempt to control all of the possible external causes, etc. Premise 5 can be verified by observing the regularity of the correlation. Does B always occur after A does, or is it possible for A to occur without B occurring, or B without A? If so, then there is either another causal factor which is missing in some cases, or there are multiple causes for B. This makes the claim that A causes B a bit more difficult to establish. The most common mistake made in formulating a causal claim is that we move too quickly from the observation of a correlation to the conclusion that there is a casual relationship.

Copyright © 1996
Critical Thinking Across the Curriculum Project
Longview Community College , Lee's Summit, Missouri - U.S.A.
One of the Metropolitan Community Colleges
An Equal Opportunity/Affirmative Action Employer

Permission to reproduce these resource pages is granted for
non-profit educational use provided the above information
is retained on all copies.

Inquiries to: michael.connelly@mcckc.edu
Last modified: 03/02/04