Table 15.5 (reproduced): A random selection of data from Zhang et al. (2013). Scores show the percentage correct on a maths test when males and females take the test either using their own name, or a fake name.

Whether someone conducted the maths test using their own name or a fake name is represented by the variable Name, and we want to predict this variable from Score, which is how well the person scored on the maths test. The model can be expressed as follows:

Because the experiment in this puzzle uses different entities in each group (i.e. different women took part in the fake-name and own-name conditions), the design is known as an independent design and therefore requires us to conduct an independent t-test. We could assign participants who took the maths test under their own name a value of 0 for the variable Name; this is the baseline group. The ‘experimental’ group was made up of participants who took the test under a fake name, and we could assign these participants a value of 1 for the variable Name.

Table 1: Calculating cross-product deviations

We need to calculate estimates for b₀ and b₁ in the model. To compute b₁, we calculate the total relationship between the predictor and outcome, the sum of cross-products (SCP) and divide this value by the total deviation of the predictor from its mean, SS_x (you can find these values in Table 1):

We could also calculate b₁ by taking the difference between the two group means (you can find the means in Table 15.5 (reproduced)):

Calculating b₀ is easy because it is equal to the mean of the baseline group (the group that we coded as 0), which in this case was the own-name group:

or:

Next we need to calculate the sum of squared residuals, which I have done in Table 2. I calculated the sum of squared residuals to be 4317.00 and we can turn this total error into an average by dividing by the degrees of freedom. The degrees of freedom will be N-p, where p is the number of parameters. There were 20 participants, N, and two parameters (b₀ and b₁), so we divide the residual sum of squares, 4317.00, by 20-2, which is 18. The resulting mean squared error in the model is 239.83.

The standard error of the model will then be the square root of this value:

Table 2: Calculating the sum of squared residuals

We can then use the standard error in the model to calculate the standard error for the b by dividing by the square root of the sum of squares for the predictor (sum of cross-products) (see Table 1):

We can now calculate the t-value:

Now we need to look up the critical value for t (see the table ‘Critical values of the t-distribution’ at the back of the main textbook) at the 0.05 significance level with 18 degrees of freedom, which is 2.10. Our observed value of 3.44 is larger than the critical value, indicating that we have a significant result. In other words, we can conclude that scores on the maths test were significantly different in females who used their own name compared to females who used a fake name, and the means tell us that women who used a fake name scored significantly higher on the maths test than women who used their own name.

For this puzzle we will follow the same procedure as in puzzle 1, only using the data for the male participants in Table 15.5 rather than the data for the female participants. The model will therefore be the same:

The experiment in this puzzle again uses different entities in each group (i.e. different men took part in the fake-name and own-name conditions), so again we have an independent design, which requires us to conduct an independent t-test.

When we conducted this test on the female participants (puzzle 1) we assigned participants who took the maths test under their own name a value of 0 for the variable Name (the baseline group), and participants who took the test under a fake name (the experimental group) a value of 1 for the variable Name, and we can do the same for the male participants.

Table 3: Calculating cross-product deviations

As in the previous puzzle, we first need to calculate estimates for b₀ and b₁ in the model. To compute b₁, we calculate the total relationship between the predictor and outcome, the sum of cross products (SCP) and divide this value by the total deviation of the predictor from its mean, SS_x (you can find these values in Table 3):

A far easier way of calculating b₁ would be to calculate the difference between the two group means (you can find the means in Table 15.5 (reproduced)):

Calculating b₀ is easy because it is equal to the mean of the baseline group (the group that we coded as zero), which in this case was the own-name group:

or

Next we need to calculate the sum of squared residuals, which I have done in Table 4. I calculated the sum of squared residuals to be 7726.40, and we can turn this total error into an average by dividing by the degrees of freedom. The degrees of freedom will be N-p, where p is the number of parameters. There were 20 participants, N, and two parameters (b₀ and b₁), so we divide the residual sum of squares, 7726.40, by 20-2, which is 18. The resulting mean squared error in the model is 429.24.

The standard error of the model will then be the square root of this value:

Table 4: Calculating the sum of squared residuals

We can then use the standard error in the model to calculate the standard error for the b by dividing by the square root of the sum of squares for the predictor (sum of cross-products) (see Table 3):

We can now calculate the t-value:

Now we need to look up the critical value for t (see the table ‘Critical values of the t-distribution’ at the back of the main textbook) at the 0.05 significance level with 18 degrees of freedom, which is 2.10. Our observed value of −0.15 is smaller than the critical value, indicating that we do not have a significant result. In other words, scores on the maths test were not significantly different in men who used their own name compared to men who used a fake name.

Let’s start with calculating Cohen’s d for the female data. I will calculate Cohen’s d using the pooled standard deviation to give you some practice. The two groups both contained 10 participants so the Ns will both be 10. The variances for the fake-name and own-name groups are in Tables 5 and 6 respectively, and if you work all of that out the pooled standard deviation is 14.69.

Table 5: Calculating the standard deviation of the fake-name group

Table 6: Calculating the standard deviation of the own-name group

We then use the pooled standard deviation (s_p) to calculate d using the following equation:

So we end up with an effect size of 1.62. In other words, in the female data, scores in the fake-name condition were 1.62 standard deviations higher than the own-name condition, which is a large effect.

Next we can calculate Cohen’s d for the male data in exactly the same way. I will use the pooled standard deviation (s_p) again and so we need to calculate this first, but before we can do that we need to calculate the variances of both the fake-name and own-name groups for the male data which I have done in Tables 7 and 8, respectively.

Table 7: Calculating the standard deviation of the fake-name group

Table 8: Calculating the standard deviation of the own-name group

We can now calculate d using the equation:

So we end up with an effect size of −0.07. In other words, in the male data, scores in the fake-name condition were 0.07 of a standard deviation lower than the own-name condition, although this is a very small effect.

For the female data (Output 15.7) the Bayes factor is 12.92, which means that the data are 12.92 times more likely given the alternative hypothesis than they are given the null hypothesis, which would be regarded by Jeffreys as ‘strong evidence’ for the alternative hypothesis, i.e., that women score higher on a maths test when using a fake name compared to their own name. Output 15.7 also shows that the Bayesian estimate, assuming that the alternative hypothesis is true, of the difference between means (beta) is 19.836 with a standard error of 0.248. You can also use the 2.5% and 97.5% quantiles as the limits of the 95% credible interval for that difference. Again, assuming the alternative hypothesis is true, there is a 95% probability that the difference between means is somewhere between 2.44 and 34.21.

For the male data (Output 15.8) the Bayes factor is 0.40, which is less than 1 and therefore supports the null hypothesis (that there is no difference in test scores in men who used their own name compared to those who used a fake name) by suggesting that the probability of the data given the null is higher than the probability of the data given the alternative hypothesis. The output also shows that the Bayesian estimate, assuming that the alternative hypothesis is true, of the difference between means (beta) is −0.561 with a standard error of 0.243. You can also use the 2.5% and 97.5% quantiles as the limits of the 95% credible interval for that difference. Again, assuming the alternative hypothesis is true, there is a 95% probability that the difference between means is somewhere between −16.79 and 15.44.

Bayes factor analysis
--------------
[1] Alt., r=0.707 : 12.9241 ±0%

Against denominator:
 Null, mu1-mu2 = 0
---
Bayes factor type: BFindepSample, JZS

1. Empirical mean and standard deviation for each variable,
   plus standard error of the mean:

                                   Mean           SD   Naive SE   Time-series SE
mu                            54.838    3.7842    0.11967              0.11967
beta (Fake - Own)   19.836    7.8554    0.24841              0.29969
2. Quantiles for each variable:

                                      2.5%         25%         50%        75%    97.5%
mu                          46.7436   52.5273    54.902    57.282     61.96
beta (Fake - Own)   2.4448   15.1047    20.140    24.759     34.21

Output 15.7: Abridged Bayes factor output for the female data in Table 15.5

Bayes factor analysis
--------------
[1] Alt., r=0.707 : 0.4006056 ±0%

Against denominator:
 Null, mu1-mu2 = 0
---
Bayes factor type: BFindepSample, JZS

1. Empirical mean and standard deviation for each variable,
   plus standard error of the mean:

                                        Mean          SD  Naive SE  Time-series SE
mu                            60.59589   4.7344    0.14971             0.14971
beta (Fake - Own)    -0.56118   7.6947    0.24333             0.24333

2. Quantiles for each variable:

                                           2.5%       25%          50%        75%     97.5%
mu                           51.20770  57.8040  60.55002  63.5746  70.3617
beta (Fake - Own) -16.78788   -5.2981   -0.28806   4.0389   15.4411

Output 15.8: Abridged Bayes factor output for the male data in Table 15.5

The analyses conducted in puzzles 1–4 (t-tests, Cohen’s ds and Bayes factor analyses) all indicate that women using a fake name tend to score higher on a maths test than those using their own name, but men achieve similar scores whether they use a fake name or their own name.

Let’s start with the females. As the question states, we are asking how well we can predict accuracy scores on a maths test based on whether participants used their own name or a fake name. This is a linear model with one dichotomous predictor:

We can replace the outcome, Y, with what we measured, accuracy scores (I called this variable Score) and replace the predictor, X, with the group to which a person belonged (I called this variable Name). The variable Name is a dichotomous variable – in other words, a nominal variable with two categories. We can’t put words into a statistical model so we must convert these two categories into numbers. In puzzle 1 I used dummy coding and coded women who used their own name as 0 for the variable Name and women who used a fake name as 1 for the variable Name:

In our model, b₀ tells us the accuracy score when the predictor is 0, i.e., it tells us the accuracy score when a woman used her own name (because I coded ‘own name’ as 0), and b₁ shows the relationship between the predictor (type of name used) and outcome (accuracy scores). We can replace the values of b₀ and b₁ with those that we calculated in puzzle 1:

We can then use the model to predict accuracy scores in women who used their own name by replacing Name in the equation with 0. The answer is 42.90, and if you look back at puzzle 1 you will see that this was also the mean score of the own-name group:

To predict accuracy scores in women who used a fake name we replace Name in the equation with 1. The answer is 66.70, and if you look back at puzzle 1 you will see that this was also the mean score of the fake-name group:

We can then do the same for the male participant data. The model would be the same as for females:

In the model above, b₀ tells us the accuracy score when the predictor is 0, i.e., it tells us the accuracy score when a man used his own name (because I coded ‘own name’ as zero), and b₁ shows the relationship between the predictor (type of name used) and outcome (accuracy scores). We can replace the values of b₀ and b₁ with those that we calculated in puzzle 2:

We can then use the model to predict accuracy scores in men who used their own name by replacing Name in the equation with 0 (because I coded ‘own name’ as 0). The answer is 61.40, and if you look back at puzzle 2 you will see that this was also the mean score of the own-name group:

To predict accuracy scores in men who used a fake name we can replace Name in the equation with 1 (because I coded ‘fake name’ as 1). The answer is 60.00, and if you look back at puzzle 2 you will see that this was also the mean score of the fake-name group.

Table 15.6 (reproduced): Data for 10 participants in Hogarth et al.’s (2104) experiment.

To answer this puzzle we need to conduct a paired-samples t-test because the same participants took part in both conditions of the experiment (control cloak and calcite cloak). 

First we need to calculate the difference scores, the mean difference score and the standard deviation, which I have done in Table 9.

Table 9: Calculating difference scores

We can see from Table 9 that the mean is −6.80 and the standard deviation is 6.63. This suggests that, on average, people recognize fewer objects when they are obscured by a transparent cloak containing calcite than when they are obscured by a transparent cloak that does not contain calcite.

Now we need to calculate the standard error of differences, which will tell us how widely we can expect difference scores to be across samples. We can calculate the standard error of differences using the following equation:

We can now calculate the t-value by dividing the mean difference by the standard error of differences as I have done in the equation below:

If we look up the critical value for t at the 0.05 significance level with 9 (N – 1) degrees of freedom, we can see that it is 2.262. Our observed t of −3.24 is bigger than the critical value (we can ignore the minus sign as that just tells us the direction of the effect), indicating a significant result. In other words, people recognized significantly fewer objects when they were concealed by a transparent cloak containing calcite than when they were concealed by a similar transparent cloak that did not contain calcite.

Remember that we are dealing with difference scores when we compute the test statistic. One very simple way to standardize the difference between means is to use the difference scores instead of the raw scores:

So we end up with an effect size of −1.45. In other words, recognition in the calcite condition was −1.45 standard deviations higher than in the control condition, or you might say recognition in the control condition was 1.45 standard deviations higher than in the calcite condition.

The Bayes factor is 6.15, which means that the data are 6.15 times more likely under the alternative hypothesis compared to under the null, which would be regarded by Jeffreys as ‘evidence with substance’ for the alternative hypothesis, i.e., that a cloak containing calcite decreases recognition scores when compared to a control cloak. The difference between means is estimated as 5.87, and there is a 95% probability that the effect lies between 1.41 and 10.63, assuming the alternative hypothesis is true.

Bayes factor analysis
--------------
[1] Alt., r=0.707 : 6.148254 ±0%

Against denominator:
  Null, mu = 0
---
Bayes factor type: BFoneSample, JZS

1. Empirical mean and standard deviation for each variable,
   plus standard error of the mean:

                  Mean          SD   Naive SE  Time-series SE
mu          5.8705    2.3628    0.07472              0.08214

2. Quantiles for each variable:

                    2.5%       25%        50%       75%      97.5%
mu          1.4121   4.3430    5.8204     7.300     10.631

Output 15.9: Abridged Bayes factor output for the data in Table 15.6

Looking at Outputs 15.7, 15.8, and 15.9, you can use the 2.5% and 97.5% quantiles as the limits of the 95% credible interval for the difference between means. So assuming the alternative hypothesis is true, there is a 95% probability that the difference between means is somewhere between 2.44 and 34.21 in Output 15.7, a 95% probability that the difference between means is somewhere between −16.79 and 15.44 in Output 15.8, and a 95% probability that the difference between means is somewhere between 1.41 and 10.63 in Output 15.9.

A confidence interval is set so that before the data are collected there is a 95% chance that the interval will contain the true value of the parameter. Once the data are collected, your sample is either one of the 95% that produce an interval containing the true value, or it is one of the 5% that do not. In other words, having collected the data, the probability of the interval containing the true value of the parameter is either 0 (it does not contain it) or 1 (it does contain it), but you do not know which. A credible interval is different in that it reflects the plausible probability that the interval contains the true value. For example, a 95% credible interval has a 0.95 probability of containing the true value; this is not true of a 95% confidence interval (the probability of it containing the true value is either 0 or 1, but you don’t know which).

You cannot use a credible interval to test hypotheses because it is constructed assuming that the alternative hypothesis is true. It tells you the interval within which the effect will fall with 95% probability, assuming that the effect exists. It tells you nothing about the null hypothesis.