Overview
Even at the data collection stage, the social work researcher needs to know what type of data analysis will facilitate an answer to the research question. The researcher should understand the purpose of each method of analysis, the characteristics that must be present in the study for the design to be appropriate, and any weaknesses of the design that might limit the usefulness of the results. Only then can the researcher select the appropriate design.
Choosing the appropriate design enables the social work researcher to gather the most relevant information about the relationship being studied. Notice that it is not the statistical test itself that deems the research valid; rather, it is the research design. Social workers must be aware of and adjust any limitations of their chosen design that may impact the validity of the study.
In this Discussion, you examine a case study involving a quantitative design, determining whether the statistical information supports the program’s efficacy and whether there are limiting factors.
TO PREPARE
- Review the Learning Resources on analyzing evaluation data, threats to internal validity, and statistics.
- Access the Social Work Case Studies media and navigate to the Chi-Square case study.
- As you review the case, consider the confounding variables—that is, factors that might explain the difference between those in the program and those waiting to enter the program.
- Post a brief outline of the case study and consider the conclusion that “the vocational rehabilitation intervention program may be effective at promoting full-time employment.”
- What statistical information shows whether the program was effective (or not)?
- Review the factors that limit the internal validity of a study (history, maturation, testing, instrumentation, statistical regression, selection bias, and attrition).
- Select and explain which of these factors might limit the ability to draw conclusions regarding cause-and-effect relationships.
References:
- Dudley, J. R. (2020). Social work evaluation: Enhancing what we do (3rd ed.). Oxford University Press.
oChapter 10, “Analyzing Evaluation Data” (pp. 255–275)
Flannelly, K. J., Flannelly, L. T., & Jankowski, K. R. B. (2018). Threats to the internal validity of experimental and quasi-experimental research in healthcareLinks to an external site.. Journal of Health Care Chaplaincy, 24(3), 107–130. https://doi.org/10.1080/08854726.2017.1421019
Document: A Short Course in Statistics Download A Short Course in Statistics(PDF)
- Walden University, LLC. (2022). Social work case studiesLinks to an external site. [Interactive media]. https://waldenu.instructure.com
oNavigate to the Chi-Square case study.
Social Work Research: Chi Square for Counts in Categories
When is chi-square analysis used?
A chi-square test is used to discover if two categories of variables, such as gender and college major, ethnic group and home ownership, or preferred sport and political party are related.
Chi-square analysis involves counting cases in separate categories to answer research questions such as these:
(a) is there a significant relationship between gender and college major?
(b) how strong is the relationship between hair color and birth sign of the zodiac?
(c) are different types of eyebrows associated with narcissism and humility? (That was a real research study: Giacomin & Rule, 2018).
The chi-square is an appropriate statistical test when the variables (that is, categories or groups), offer mutually exclusive alternate choices, which means that a response cannot be in more than one category at once. These might include days of the week (it cannot be both Monday and Wednesday at the same time), choices of entrees a child selects at school (hot dogs, hamburgers, or pizza); treatment group vs. control group, or any other comparison between people or items where counts or amounts can be placed into separate, non-overlapping categories.
Question for the discussion this week: Is the employment program described in the case study better than having no program in helping parolees obtain employment after release from prison?
In this case study, the social workers evaluated the employment status of recent parolees who participated in a new type of employment program. They compared the employment status of the parolees who completed the program to those parolees who were placed on the waiting list to enter the program.
We have to make some important assumptions:
- We have to assume that program participants enrolled on a first-come-first-served basis; they were not specifically selected for the program by any sampling method, including random selection.
- We have to assume that each program participant or waiting list member was unemployed at the start of the program.
How do we analyze this case?
1. Determine the research design used
-
- Organization: single-case or multiple group design?
1a. Single case design
1b. Experimental or quasi-experimental designs: designs with multiple groups, where two or more groups are compared, including at least one control group
2. Discover how participants were assigned to groups:
Random assignment: participants are randomly assigned to different groups = true experiment.
Pre-selected groups: researchers use naturally formed groups (classrooms, neighborhoods, clients at different clinics) = quasi-experimental design.
Non-random allocation: for example, if the first 10 volunteers are assigned to a treatment group, but the second ten volunteers were placed in the control group. This non-random allocation may result in non-equivalent groups if there is a reason that the second ten delayed their participation
3. Determine the types of variables used: quantitative or categorical
3a. A quantitative variable is recorded as an amount (such as average scores on a test, number of alcoholic beverages consumed, and so on)
3b. A categorical variable describes a group where calculations are meaningless: Republicans, Democrats, and Independents; groups consisting of a treatment group and a medication group; choices of items (Ford, Toyota, and Cadillac) – it is not possible to have 3.5 Republicans, or half a Cadillac. Data are counted (14 Fords, 27 Toyotas, and 3 Cadillacs) or frequencies (three times/month, 4 hours apart, etc.)
4. Find the type of data analysis used
- Descriptions without analysis: common in survey data. Examples: 43.3% of people surveyed prefer Coca-Cola; 22.1% prefer Pepsi. The researcher reports what was found without manipulating any variables
- Chi-Square Analysis to determine associations (if any) between categories of variables. It does NOT prove (or disprove) cause-and effect relationships.
- p-values for a categorical analysis may suggest that a relationship is likely to exist, but the analysis cannot prove that the independent variable caused the effect in the dependent variable.
- A full experiment is required for establishing a cause and effect because categorical data analysis research does not control all the factors that could influence the result. Even if two variables appear to be linked, there may be a third variable that is causing them both to increase or decrease together.
- Cause-and-effect analyses (t-tests, ANOVAs, regression, multilevel, logistic regression, and many others ) ONLY true experiments can establish cause-and-effect relationships because they are designed to include three factors:
- they randomize the assignment of participants to groups
- they control the independent variables so that the only effect (if any) comes from the independent variable
- they attempt to control spurious or confounding variables that could have an outside effect. Look at the handout Threats to Internal Validity (attached) for examples.
How to analyze a chi-square value
The analysis of chi-square values can be tricky. It involves comparing the chi-square value that is obtained from a cross-tabulation table to a distribution that indicates what the chi-square value would be if there were NO differences among groups, which is the null hypothesis.
To make the chi-square easier to interpret, look for the p-value for the Pearson chi-square, found in the Asymp. Sig (2-sided) column in the Table 3 of the chi-square case study.
A p-value that is less than 0.05 is interpreted as being statistically significant. P-values that are smaller than 0.05 are desirable, and include values such as 0.025, .01, and .003.
A p-value that is smaller than 0.05 means that the probability that there are no differences between groups is smaller than 5 in 100. That’s not quite correct, but that is the commonly understood meaning. The actual interpretation is convoluted: the likelihood of error in rejecting the null hypothesis when it is actually true is less than 5 in 100.
Your task for this discussion:
Outline the study in this way:
- Define the two groups in the study (1-2 sentences)
- Describe the details of the intervention provided to the treatment group, including type of intervention, frequency, and number of participants (3-4 sentences)
- Provide the outcomes (the categories) of the treatment program that were tallied (1 sentence).
- State the organization of the research design (single-case or multiple groups) (brief phrase)
- Indicate whether randomization occurred: (yes or no – brief phrase)
- Identify the type of variables that were used (quantitative or categorical – brief phrase)
Explain the outcome
- Was the null hypothesis supported (no differences between group) or rejected? (1-2 sentences)
- What statistical value supports your decision? Where is it found? (1-2 sentences)
- Does the result prove that the treatment caused the outcome? -This is tricky. See the explanations above.
Offer alternative explanations for the outcome (there are always some!)
Consider common threats to validity (these are also considered confounding or spurious variables - a handout is attached). Identify two of these factors that might be influencing the outcome and explain your reasoning. Try to select factors that students posting previously did not select.
Chi Square case study
Chi-Square
Lesson 4 of 5
Molly, an administrator with a regional organization that advocates for alternatives to long-term prison sentences for nonviolent offenders, asked a team of researchers to conduct an outcome evaluation of a new vocational rehabilitation program for recently paroled prison inmates. The primary goal of the program is to promote full-time employment among its participants.
To evaluate the program, the evaluators decided to use a quasi-experimental research design. The program enrolled 30 individuals to participate in the new program. Additionally, there was a waiting list of 30 other participants who planned to enroll after the first group completed the program. After the first group of 30 participants completed the vocational program (the “intervention” group), the researchers compared those participants’ levels of employment with the 30 on the waiting list (the “comparison” group).
The research question for the study is: Is there a relationship between the independent variable, treatment, and the dependent variable, employment level? In other words, is there a difference in the number of participants who are not employed, employed part-time, and employed full-time in the program and the control group (i.e., waitlist group)?
Data Collection and Analysis
In order to collect data on employment levels, the probation officers for each of the 60 people in the sample (those in both the intervention and comparison groups) completed a short survey on the status of each client in the sample. The survey contained demographic questions that included an item that inquired about the employment level of the client. This was measured through variables identified as none, part-time, or full-time. A hard copy of the survey was mailed to each probation officer and a stamped, self-addressed envelope was provided for return of the survey to the researchers.
After the surveys were returned, the researchers entered the data into an SPSS program for statistical analysis. Because both the independent variable (participation in the vocational rehabilitation program) and dependent variable (employment outcome) used nominal/categorical measurement, the bivariate statistic selected to compare the outcome of the two groups was the Pearson chi-square.
The chi-square test for independence is used to determine whether there is a relationship between the two variables that are categorical in the level of measurement. In this case, the variables are: employment level and treatment condition. It tests whether there is a difference between groups.
The hypotheses are:
- bullet
H0 (The null hypothesis): There is no difference in the proportions of individuals in the three employment categories between the treatment group and the waitlist group. In other words, the frequency distribution for variable 2 (employment) has the same proportions for both categories of variable 1 (program participation).
Note: It is the null hypothesis that is actually tested by the statistic. A chi-square statistic that is found to be statistically significant (p < .05) indicates that we can reject the null hypothesis (understanding that there is less than a 5% chance that the relationship between the variables is due to chance).
- bullet
H1 (The alternative hypothesis): There is a difference in the proportions of individuals in the three employment categories between the treatment group and the waitlist group.
Note: The alternative hypothesis states that there is a difference. It would allow us to say that it appears that the treatment (vocational rehab program) is effective in increasing the employment status of participants.
Results
After all of the information was entered into the SPSS program, the following output charts were generated:
Table 1. Case Processing Summary
| Cases | ||||||
| Valid | Missing | Total | ||||
| N | Percent | N | Percent | N | Percent | |
| Program Participation *Employment | 59 | 98.3% | 1 | 1.7% | 60 | 100.0% |
The first table, titled "Case Processing Summary," provided the sample size (N = 59). Information for one of the 60 participants was not available, while the information was collected for all of the other 59 participants.
Table 2. Program Participation *Employment Cross Tabulation
| Employment |
| |||||
| None | Part-Time | Full-Time | ||||
| Program Participation | Intervention Group | Count % within Program Participation | 5 | 7 | 18 | 30 |
| Comparison Group | Count % within Program Participation | 16 | 7 | 6 | 29 | |
| Total | Count % within Program Participation | 21 | 14 | 24 | 59 |
The second table, "Program Participation Employment Cross Tabulation," provided the frequency table, which showed that among participants in the intervention group, 18 or 60% were found to be employed full time, while 7 or 23% were found to be employed part time, and 5 or 17% were unemployed. The corresponding numbers for the comparison group (parolees who had not yet enrolled in the program but were on the waiting list for admission) showed that only 6 or 21% were employed full-time, while 7 or 24% were employed part time, and 16 or 55% were unemployed.
Table 3. Chi-Square Tests
| Value | df | Asymp. Sig (2-sided) | |
| Pearson Chi-Square | 11.748a | 2 | .003 |
| Likelihood Ratio | 12.321 | 2 | .002 |
| Linear-by-Linear Association | 11.548 | 1 | .001 |
| N of Valid Cases | 59 |
a. 0 cells (.0%) have expected count less than 5. The minimum expected count is 6.88.
The third table, which provided the outcome of the Pearson chi-square test, found that the difference between the intervention and comparison groups were highly significant, with a p value of .003, which is significantly beyond the usual alpha-level of .05 that most researchers use to establish significance.
Discussion
These results indicate that the vocational rehabilitation intervention program may be effective at promoting full-time employment among recently paroled inmates. However, there are multiple limitations to this study, including that 1) no random assignment was used, and 2) it is possible that differences between the groups were due to preexisting differences among the participants (such as selection bias).
Potential future studies could include a matched comparison group or, if possible, a control group. In addition, future studies should assess not only whether or not a recently paroled individual obtains employment but also the degree to which he or she is able to maintain employment, earn a living wage, and satisfy other conditions of probation.