Limitations of the Chi-Square Test

Although the chi-square test of statistical significance is very useful, it does have three important limitations, which you need to be aware of.

Limitations of the Chi-Square Test

The chi-square test does not give us much information about the strength of the relationship or its substantive significance in the population.
The chi-square test is sensitive to sample size. The size of the calculated chi-square is directly proportional to the size of the sample, independent of the strength of the relationship between the variables.
The chi-square test is also sensitive to small expected frequencies in one or more of the cells in the table.

Be sure that you understand these three limitations. It is very important to realize that statistical significance and substantive significance are different. The sensitivity of chi-square to sample size may make a weak relationship statistically significant if the sample is large enough. Therefore, we need to use tests of significance like chi-square together with measures of association like lambda, Cramer's V or gamma (which we will discuss later) to guide us in deciding whether a relationship is important and worth pursuing.

Because chi-square is sensitive to small expected frequencies in the cells, we have to be very cautious in interpreting chi-square if one or more cells are less than 5. As Frankfort-Nachmias and Leon-Guerrero note (2011:350), "most researchers limit the use of chi-square to tables that either (1) have no fe values below 5 in value or (2) have no more than 20 percent of the fe values below 5 in value." When you use SPSS to find chi-square it lets you know how many cells have an expected value less than 5.

"A Closer Look 11.1" on pages 370-371 in the textbook leads you through an example in which you end by failing to reject the null hypothesis. Please go through this example carefully to review and strengthen your understanding of chi-square.

The "Statistics in Practice" section on Education and Health Assessment on pages 352-353 in the textbook shows a chi-square example for a larger table. Also read through this example to see what conclusions are reached and why.

Finally, we have two more examples. The "Reading the Research Literature" section on pages 354-355 in the textbook refers to some fascinating research on the relationship between academic achievement and sibling cooperation among Vietnamese American high school students. Although the research is interesting, we use the section to see how chi-square is usually reported in the research literature or in reports.

Under Table 11.9 on page 355 in the textbook, you see the following:

Note: χ² = 29.33; p < .001

This tells us that for this table the obtained chi-square value is 29.33 and that it is statistically significant at less than the .001 level. We can reject the null hypothesis and be pretty confident that our decision is correct. Sometimes such notes reporting chi-square and its p value also include the degrees of freedom. What are the degrees of freedom for this table?

Degrees of freedom = (r -1)(c -1) = (3-1)(3 -1) = (2)(2) = 4

The authors discuss a final example of violent offense onset (dependent variable) by gender (independent variable).  Note that there is an error in Table 11.10.  The results shown for "Indigenous Status" are NOT statistically significant.