Instructions: The .ipynb file should be submitted as **searchable PDF **and not as a picture. Read the instructions carefully. Please download your assignment here. include my name majed to the csv files
- All answers along with their codes should be submitted as searchable PDF .
- Pictures or snapshots of your work will not be accepted.
- All generated csv files and .ipynb file must be submitted in a zip-folder as a secondary source.
- Ensure the zip-folder has four csv files (Your Name, college_data, Height, Sustainability).
- You may use Jupyter notebook or Colab as per your convenience.
Note: Reach out to your instructor for any question regarding csv files, codes or zip-folder.
Failure to comply with the instructions will result in 0 grade on the relevant portions of the assignment. Your instructor will grade your submission based on what you submitted. Failure to submit an assignment or submitting an assignment for another class will result in a 0 grade, without the opportunity to resubmit. Make sure that you submit your original work. Suspected plagiarism cases will be treated as possible academic misconduct and will be reported to the College Academic Integrity Committee for formal investigation. As part of this procedure, your instructor may require you to meet with them for an oral exam on the assignment.
Statistical Intuitions and Applications
Assignment #2
A - Statistical Intuition in Medical Science
Question 1:
The data is sourced from the 2020 annual CDC survey of 400k US adults regarding their health status. The dataset is a major part of the Behavioral Risk Factor Surveillance System (BRFSS), which conducts annual telephone surveys to comprehensively collect health-related information from residents across the United States.
The code given below select a random sample of 300 respondents from the original data set which you need to use to analyze the data for those 300 respondents.
Use the Heart Disease dataset to answer the following questions:
a. Is having heart disease independent of whether the respondent is diabetic? Include the two-way table and all your calculations in your answer.
b. Is there a relationship between having heart disease and the general health condition of the respondent? Include the two-way table, stacked bar graph, and all your calculations in your answer.
c. Is there a relationship between having heart disease and the gender of the respondent? Include the two-way table, stacked bar graph, and all your calculations in your answer.
d. Does doing physical activity lower the risk of having heart disease? If so, by how much?
B - Statistical Intuition in Data Science
Question 2:
a. Compare the probability distributions of any three classes of your choice and identify the key differences. Your answers should have proper justification.
b. Explain how will you determine which class (or classes) has the lowest number of students who fail.
c. In a class (or classes) that has the highest number of students fail, which grade is the highest?
d. Explain how will you calculate the number of students with B grade in Class_2 and Class_4.
e. After comparing students with C grade in Class_1 and Class_4, Amna concluded that Class_4 has 5% more students with C grade than in Class_1. Provide an evidence if she is right or wrong.
C - Statistical Intuition in Health Sciences
Question 3:
Dubai Health Authority (DHA) contacted you to analyze a particular situation in the two hospitals: Dubai Hospital and Rashid Hospital which is to assess their performance in terms of resources allocation (Staffing, medical equipment, space etc). DHA provided you the most updated information available to them which can be obtained through the code below. It gives you the probability distributions of number of patients in these hospitals who have been diagnosed with a certain medical condition along with expected values and standard deviations.
a. How do you interpret the differences in both the mean and standard deviation when assessing the patient load and healthcare demands at the two hospitals?
b. How does this difference in expected patient counts affect resource allocation, such as staffing and medical supplies, for the two hospitals?
D - Statistical Intuition in Social Innovation
Here we will consider a simple dataset from social sciences that records the height of individuals.
Question 4:
Consider a dataset from social sciences which is related to the heights of individuals in a population that often forms a normal distribution.
a. Explain how would you calculate the probability that a randomly selected individual has a height greater than 185 cm?
b. Explain how would you calculate the probability that a randomly selected individual has a height between 160 cm and 180 cm?
c. Calculate the 90th percentile of the height distribution. What does this value represent in the context of heights?
d. In a random sample of 49 individuals from this population, what is the probability that the sample mean height is within 1 cm of the population mean?
e. Suppose we want to find the height range that contains the middle 95% of the individuals in the population. Explain how do you obtain the lower and upper bounds of this range?
f. If you needed to cast a basketball team with the tallest players from this dataset, how many players would you need to ensure they win every game without jumping? Explain.
E - Statistical Intuition in Sustainability: COP-28
Imagine you have been invited to participate in the upcoming COP-28 which gives you an opportunity to contribute to the community using your statistical skills. You conducted a survey within your area and observed the recycling behavior in your community and recorded whether individuals in the community recycle ("1" for yes and "0" for no). You managed to get 100 records.
Question 5:
The answers to the following questions will provide you valuable insights that can be presented in COP-28.
a. What is the proportion of individuals in the community who recycle, based on the dataset?
b. Calculate a 95% confidence interval for the proportion of recyclers in the community. What does this interval tell us about the likely range of recycling behavior in the entire community?
c. If the 95% confidence interval for the proportion of recyclers is (0.55, 0.75), how would you interpret this interval? What level of confidence does it represent, and what can we conclude about the community's recycling behavior?
d. If a separate survey conducted earlier found that the recycling rate in the community was 70%, how does this rate compare to the point estimate obtained from the dataset, and what additional insights can you provide?
Learning Outcomes Added
- #CompProgramDesign
Generate working programs in a computer language that can solve computational problems; find and fix bugs that appear in them.Rubric
Rubric0Did not submit the assigned item, or submitted something that does not meet the minimum standard.
1
Does not generate working programs in a computer language that can solve computational problems or find and fix bugs that appear in them when prompted, or do so mostly or entirely inaccurately.
2
Generates working programs in a computer language that can solve computational problems only somewhat accurately, or finds and fixes bugs that appear in them in a way that fails to address the relevant problems or goals.
3
Accurately or effectively generates working programs in a computer language that can solve computational problems or finds and fixes bugs that appear in them in a way that addresses the relevant problems or goals.
4
Accurately or effectively generates working programs in a computer language that can solve computational problems and finds and fixes bugs that appear in them in a way that addresses the relevant problems or goals and demonstrates a deep grasp of the skill or concept by analyzing, explaining, or justifying the application in a way appropriate to the given context. Distributions
Identify different types of distributions and make inferences based on samples from distributions appropriately.The distribution of a variable is a function that maps all possible values of the variable to how often they occur (the frequency of occurrence). There are several types of distributions, but the most commonly used in statistics is the familiar bell curve, called a “normal” or “gaussian” distribution. When distributions are used to describe data, one must distinguish between population data (all possible values or elements of interest) and sample data (a subset of the population). Drawing statistical inferences about a population from sample data often requires a “sampling distribution,” a theoretical distribution of the values of a specified statistic (i.e., mean, proportion) over all possible random samples of a specific size that can be made from a given population. The shape of the sampling distribution is ensured by the Central Limit Theorem when certain criteria are met. However, statistical analyses often require assumptions about the underlying distributions, requiring one to state those assumptions and analyze their limitations. Considering distributions can help one recognize phenomena such as regression to the mean.ExampleYou are an engineer, hired to construct elevators for a shopping mall. Your manager asks you to determine the probability that the elevator will be over the threshold safety weight (700 kg) when there are 9 people in it. Past research shows that the shoppers at the mall have a mean weight of 55 kg and a standard deviation of 12 kg. While you don’t know the shape of this underlying distribution of weights, you realize that as long as it is somewhat symmetric, you can employ the Central Limit Theorem to conclude that the sampling distribution of the sample mean (with sample size 9) is approximately normal, with a mean of 55 kg and a standard error of 4 kg (SE = SD/sqrt(n)). This allows you to accurately compute, using z-scores, the probability that the mean weight of 9 randomly selected mall shoppers is over 78 kg (700 kg / 9) as the area under the upper tail of the sampling distribution. However, you realize that perhaps the original distribution is very skewed to the left, with mostly adults at the mall but a few babies as well. In this case, since the small sample size of interest is only 9, the sampling distribution would still be skewed to the left, meaning that the probability you computed under the assumption of normality is an underestimate. You deliver these results to your manager with all caveats laid out.
Rubric
0
Did not submit the assigned item, or submitted something that does not meet the minimum standard.
1
Does not demonstrate knowledge of distributions when prompted; fails to identify an appropriate distribution in the given context; describes or calculates features of a distribution mostly or entirely inaccurately; (when applicable) does not consider the effects of the underlying distribution when making inferences from samples, failing to address the required assumptions; (when applicable) does not recognize when regression to the mean applies in a given context.
2
Identifies an appropriate distribution in the given context only somewhat accurately; describes or calculates features of a distribution only somewhat accurately; (when applicable) considers the effects of the underlying distribution when making inferences from samples, including the required assumptions, only somewhat accurately; (when applicable) recognizes regression to the mean only somewhat accurately.
3
Identifies an appropriate distribution in the given context mostly or entirely accurately; describes features of a distribution, using calculations if appropriate, mostly or entirely accurately; (when applicable) considers the effects of the underlying distribution when making inferences from samples, including the required assumptions, mostly or entirely accurately; (when applicable) recognizes when regression to the mean applies in a given context mostly or entirely accurately.
4
Accurately identifies an appropriate distribution in the given context and justifies the choice; accurately describes features or characteristics of a distribution, including relevant calculations, with an appropriate level of detail; (when applicable) correctly considers the effects of the underlying distribution when making inferences from samples, including required assumptions, and clearly explains the consequences and limitations; (when applicable) accurately recognizes when regression to the mean applies in a given context and describes measures to mitigate the effect.
#Visualizations
Interpret, analyze, and create data visualizations.
Typically, one can understand data best by looking at it from multiple different perspectives, which can be facilitated by data visualization techniques—this is because the human brain did not evolve to be able rapidly to scan and make sense of columns of data. Histograms, cumulative histograms, difference histograms, bar graphs, line graphs, scatter plots, and many other types of graphics can provide insights into how to ask and answer questions. Different data visualization techniques have different strengths and weaknesses, and one must consider properties of the data and the questions of interest when deciding how best to use these tools.
Example
Your marketing firm is working with a company that is trying to increase product sales from existing customers. You have access to a range of data and create a few graphs to highlight potential areas of opportunity. You wonder if there are seasonal trends, so you construct a line graph of monthly sales, which show that sales substantially dip during winter months. A line graph is appropriate because of the temporal relationship the time series data have. Next, you wonder if particular demographic groups drive sales, so you create a bar graph to compare those groups. A bar graph is appropriate because the demographic groups membership is a categorical variable. Last, you wonder if income levels may relate to product sales, so you construct a scatter plot with income levels on the x axis and product sales per customer on the y axis. A scatter plot is appropriate because the two variables are continuous variables and you want to evaluate the relationship between them. For each graph, you ensure that you have proper axis labels with units, and a brief but informative caption. The insights the data visualizations provide help guide the marketing strategy.
Rubric
0Did not submit the assigned item, or submitted something that does not meet the minimum standard.
1
Does not critique or apply tools for generating data visualizations when prompted, or does so entirely or mostly ineffectively; does not analyze or interpret data visualizations when prompted, or does so mostly or entirely inaccurately.
2
Critiques a data visualization only somewhat accurately; generates a somewhat effective data visualization; analyzes or interprets a data visualization only somewhat accurately.
3
Effectively generates a data visualization; effectively analyzes or interprets a data visualization; (when applicable) effectively critiques a data visualization.
4
Effectively generates a detailed data visualization appropriate for the data; effectively analyzes or interprets a data visualization and provide appropriate justification and details; (when applicable) effectively critiques a data visualization and clearly explains why certain methods are better for distinct data sets.
#InferentialStats
Apply and interpret confidence intervals, statistical significance, and regression.
Measurements drawn from samples are used to make inferences about broader populations, often for the purpose of evaluating whether patterns in the population are “significant.” Tests of statistical significance tell us when a difference or pattern in a sample is likely due to chance or instead reflects strong evidence for a genuine difference or pattern in the population. Setting up, executing, and interpreting the results of significance tests requires one to identify and contemplate the unavoidable statistical errors that are possible. Statistical significance, however, says nothing about whether the size of the difference or pattern matters practically. That is the domain of practical significance, which is evaluated by measuring the effect size. One must take care to distinguish these two very different concepts, using appropriate calculations and interpretations.
Example
You read a journal article that reports on a large clinical trial carried out to compare a new medical treatment for hangover headaches with standard paracetamol. The analysis shows a statistically significant difference in the average duration of a headache when using the new treatment compared to the standard treatment, reported with a standard significance level (p<0.05). This means that, assuming there is zero difference in effect, it is quite unlikely (less than 5% chance) to observe results this extreme or more. However, given the very large sample size, you know that the statistical power is high, so even small practical differences would be deemed statistically significant. Upon closely examining the size of the effect, you note that the decrease in headache duration is only 15 minutes compared to the standard treatment group. Recognizing that a typical hangover headache lasts from 5-12 hours, you become more reserved about the practical implications of the clinical trial. Your reservation is supported by the small Cohen’s d value of 0.1, a standardized measure of practical significance (effect size) when comparing two means, which in this case indicates that the effect of the new treatment is small relative to the existing one.
Rubric
0
Did not submit the assigned item, or submitted something that does not meet the minimum standard.
1
Does not demonstrate knowledge of significance when prompted; applies or interprets an appropriate significance test or measure of significance mostly or entirely inaccurately; (when applicable) distinguishes between type I and type II errors mostly or entirely inaccurately; (when applicable) distinguishes between practical and statistical significance mostly or entirely inaccurately.
2
Applies or interprets an appropriate significance test or measure of significance only somewhat accurately; (when applicable) distinguishes between type I and type II errors only somewhat accurately; (when applicable) distinguishes between practical and statistical significance only somewhat accurately.
3
Applies or interprets an appropriate significance test or measure of significance mostly or entirely accurately; (when applicable) distinguishes between type I and type II errors mostly or entirely accurately; (when applicable) distinguishes between practical and statistical significance mostly or entirely accurately.
4
Accurately applies or interprets an appropriate significance test or measure of significance with well justified reasoning; (when applicable) accurately distinguishes between type I and type II errors and explains the potential consequences; (when applicable) accurately distinguishes between practical and statistical significance and explains the difference within a given context; (when applicable) accurately and effectively calculates or interprets the results of a significance test or measure in a subtle or complex scenario.
#Probability
Apply and interpret fundamental concepts of probability, including conditional and bayesian probabilities.
A probability specifies how likely it is that a specific event will occur. There are different interpretations of probability, which provide different frameworks for understanding claims about the probability of various events. In addition, a conditional probability is the probability of an event occurring given the occurrence of another event. One important type of conditional probability is the "posterior probability" (the degree of belief, conditional on new data, in a hypothesis) based on a prior probability (the baseline or the degree of belief in the hypothesis prior to the new data). The posterior probability is updated as new data are obtained. One must take care to identify the appropriate method to calculate the probability of an event and to properly interpret probabilities.
Example
You have your annual physical, and they take some blood. They have randomly been testing people for meningitis. They believe that 1 in 10,000 people in your city has this disease. The sensitivity (true positive) and specificity (true negative) of the blood test are both 99%. Your test comes back positive. Before you panic, you start analyzing the situation. Assuming that the probability of you catching meningitis is the same as the city population, the prior probability is P(Sick) = 0.0001. The posterior probability is calculated based on the prior probability and the sensitivity and specificity of the test: P(Sick|+) = P(+|Sick)*P(Sick) / [P(+|Sick) + P(+|¬Sick)] = 0.99*0.0001 / (0.99*0.0001 + 0.01*0.9999) — which is just under 1%. The low prior probability of being sick and the possibility of a false positive explain this relatively low chance of having the disease even in light of its high accuracy. It is also noted that the calculation depends on the assumption of prior probability. If instead you worked in a clinic and know that one out of ten providers contract the disease, you would use P(Sick) = 0.1 in the probability calculation, yielding a much higher probability. Luckily, you work a corporate job so you don’t stress about it too much.
Rubric
0
Did not submit the assigned item, or submitted something that does not meet the minimum standard.
1
Does not demonstrate knowledge of probability when prompted; applies or interprets probabilities mostly or entirely inaccurately; (when applicable) distinguishes between different interpretations of probability mostly or entirely inaccurately; (when applicable) calculates a probability mostly or entirely inaccurately.
2
Applies or interprets probabilities only somewhat accurately; (when applicable) distinguishes between different interpretations of probability only somewhat accurately; (when applicable) calculates a probability only somewhat accurately; (when applicable) applies a probability type inappropriate for the given context.
3
Applies an appropriate probability mostly or entirely accurately; interprets probabilities mostly or entirely accurately; (when applicable) distinguishes between different interpretations of probability mostly or entirely accurately; (when applicable) calculates an appropriate probability mostly or entirely accurately.
4
Accurately applies appropriate probabilities with well justified reasoning; accurately interprets probabilities with well justified reasoning; (when applicable) accurately distinguishes between different interpretations of probability and provides a detailed explanation; (when applicable) accurately calculates an appropriate probability with clear detailed steps; (when applicable) accurately and effectively applies an appropriate probability or interprets a probability in a complex or sophisticated context.