Hypothesis Testing Practice Quiz
(The text in the scenario below is copied from the following website: http://www.psychstat.missouristate.edu/introbook/sbk26m.htm
Please feel free to visit this site for additional information on hypothesis testing)

 

Section I: Scenario

A superintendent in a medium size school has a problem. The mathematical scores on nationally standardized achievement tests such as the SAT and ACT of the students attending her school are lower than the national average. The school board members, who don't care whether the football or basketball teams win or not, is greatly concerned about this deficiency. The superintendent fears that if it is not corrected, she will lose her job before long.

As the superintendent was sitting in her office wondering what to do, a salesperson approached with a briefcase and a sales pitch. The salesperson had heard about the problem of the mathematics scores and was prepared to offer the superintendent a "deal she couldn't refuse." The deal was teaching machines to teach mathematics, guaranteed to increase the mathematics scores of the students. In addition, the machines never take breaks or demand a pay increase.

The superintendent agreed that the machines might work, but was concerned about the cost. The salesperson finally wrote some figures. Since there were about 1000 students in the school and one machine was needed for every ten students, the school would need about one hundred machines. At a cost of $10,000 per machine, the total cost to the school would be about $1,000,000. As the superintendent picked herself up off the floor, she said she would consider the offer, but didn't think the school board would go for such a big expenditure without prior evidence that the machines actually worked. Besides, how did she know that the company that manufactures the machines might not go bankrupt in the next year, meaning the school would be stuck with a million dollars worth of useless electronic junk.

The salesperson was prepared, because an offer to lease ten machines for testing purposes to the school for one year at a cost of $500 each was made. At the end of a year the superintendent would make a decision about the effectiveness of the machines. If they worked, she would pitch them to the school board; if not, then she would return the machines with no further obligation.

An experimental design was agreed upon. One hundred students would be randomly selected from the student population and taught using the machines for one year. At the end of the year, the mean mathematics scores of those students would be compared to the mean scores of the students who did not use the machine. If the means were different enough, the machines would be purchased.


Section II: Questions
(I recommend answering these questions first before looking at the answers in Section III)

1.      What is the null hypothesis?

2.      What is the alternative hypothesis?

3.      What are the two possible realities (states of the world) regarding the machines’ effectiveness?

4.      The study obtained different math improvement scores for the two groups.  What are the two relevant reasons for why this difference is there?

5.      If we decide to use the machines but they don’t really work, is this a good decision?  What name do we give decision in terms of the hypothesis testing table?

6.      If we decide not to use the machines but they really do work, is this a good decision?  What name do we give to this decision in terms of the hypothesis testing table?

7.      What two choices can we make about the null hypothesis?  What determines which of these two choices we make?

8.      If the reality is the machines don’t work, and we repeated this experiment 100 times, about what percentage of the time would we decide to use the machines anyway?

 

 

 

Section III: Answers

1.      The teaching machines don’t work; the teaching machines will have no effect on students’ math scores

2.      The teaching machines do work; the teaching machines will have an effect on students’ math scores

3.      Possibility #1: the teaching machines really do work; Possibility #2: the teaching machines really do not work

4.      Relevant reason #1: the scores are different because of chance or random variation; Relevant reason #2: the scores are different because the teaching machines caused a change in student math scores.

5.      No.  Deciding that the teaching machines work when in reality they really don’t (rejecting Ho when Ho is really true) is an example of a Type I Error.

6.      No.  Deciding that the teaching machines do not work when in reality they really do (failing to reject Ho when Ho is really false) is an example of a Type II Error.

7.      Choice #1: reject Ho ; Choice # 2: fail to reject Ho ; Our choice is made by whether the p-value linked with our statistic (t, F, chi-square, etc…) is above or below the alpha level (usually set at .05).  If the p-value is below alpha, we reject Ho and if the p-value is equal to or above alpha, we fail to reject Ho.

8.      5%.  This question assumes that in reality the machines don’t work.  Our decision about the machines isn’t based on this reality.  It is based on the p-value we get with our statistic and if it is less than alpha (.05), then we reject Ho and decide that the machines do work.  The percentage of times we will make this particular mistake (rejecting Ho when Ho is really true, a Type I Error) is completely dependent on the value we choose for alpha.  If alpha were lowered to .01, then 1% we would make this error.  If alpha were moved to .1, then 10% of the time we would make this particular error.