Introduction

This worksheet builds on three previous worksheets: within-subject differences, understanding interactions, and factorial differences. We therefore assume you are familiar with the concepts and commands in those worksheets. If that is not the case, read (or re-read) those worksheets before proceeding.

In the factorial differences worksheet, we did a two-factor Bayesian ANOVA where one factor was within-subject and the other was between-subjects. In this worksheet, we’ll use the same data frame to illustrate another couple of two-factor ANOVAs.

Getting started

Return to the project that you used for the factorial differences worksheet, which you named rminr-data.

Ensure you have the files required for this worksheet, by asking git to “pull” the repository. Select the Git tab, which is located in the row of tabs which includes the Environment tab. Click the Pull button with a downward pointing arrow. A window will open showing the files which have been pulled from the repository. Close the Git pull window.

Now, open a new blank script and save it as anova4.R. Put all comments and commands in that script, and save the script regularly.

First, you’ll need to load some standard packages, load the data we used in the previous worksheet, and set the appropriate columns up as factors.

Enter these comments and commands into your script, and run them:

# Clear workspace
rm(list = ls())
# Load packages
library(tidyverse)
library(BayesFactor, quietly = TRUE)
# Load data
words <- read_csv("wordnaming2.csv")
# Set columns as factors
words$sex <- factor(words$sex)
words$subj <- factor(words$subj)
words$medit <- factor(words$medit)
words$congru <- factor(words$congru)
words$block <- factor(words$block)

Two between-subject factors

We can use anovaBF to do a factorial Bayesian ANOVA where both factors are between-subjects. It works exactly the same way as before. For example, we can do a factorial Bayesian ANOVA for sex and medit like this:

First we create a subject-level summary of the data - this involves averaging across the 90 data points that contribute to the reaction time (RT) measure for each participant. We do this using the now-familiar group_by and summarise commands.

Enter this comment and commands into your script, and run them:

# Create subject-level summary
wordsum <- words %>% group_by(subj, medit, sex) %>% summarise(rt = mean(rt))

Now we run the Bayesian ANOVA, as follows; enter these commands into your script, and run them:

# Calculate Bayes Factors
bf2bs <- anovaBF(
    formula = rt ~ medit*sex,
    data = data.frame(wordsum))

Note that, unlike the Bayesian ANOVAs we previously did that included a within-subjects factor, we do not include + subj or whichRandom = 'subj' in this entirely between-subjects ANOVA. This is because we have created a subject-level summary of the data in which each participant only contributes one data point.

Explanation of output

The output is interpreted the same way as before; enter this comment and command into your script, and run it:

# Show results of analysis
bf2bs

Bayes factor analysis
--------------
[1] medit                   : 718.3828 ±0.01%
[2] sex                     : 1.446498 ±0.01%
[3] medit + sex             : 1160.597 ±1.38%
[4] medit + sex + medit:sex : 83.20431 ±2.23%

Against denominator:
  Intercept only 
---
Bayes factor type: BFlinearModel, JZS

`[1] medit`

This is the main effect of training type, medit. More specifically, it is a test of the hypothesis that medit, affects rt.

The Bayes Factor for this hypothesis is around 718, which is strong evidence that there is a main effect of meditation training. If you’ve got a particularly good memory, you’ll have noticed this isn’t quite the same Bayes Factor as we saw for the main effect of meditation in this analysis of the same data. As with many forms of multi-factor ANOVA (including traditional, p-value, ones) the evidence terms you get depend somewhat on the formula you specify. Here, you included sex but not congru as factors; in the previous analysis, it was the other way around. This can affect the results you get.

If the type of analysis you choose to conduct affects the central conclusions of your study, then that is normally an indication that you need to replicate your result with a larger sample. Conclusions should be robust to variations in analysis technique.

`[2] sex`

This is the main effect of participant sex. More specifically, it is the hypothesis that sex affects rt. This is compared against the hypothesis that there is no effect of sex.

The Bayes Factor is about 1.4, meaning the evidence for a main effect of sex is inconclusive.

The interaction

As before, anovaBF does not directly give us a Bayes Factor for the interaction of the two main effects. Instead, it gives us two Bayes Factors for things that we can use to work out the interaction BF. These are:

[3] `medit + sex`

This the hypothesis that there is a main effect of both factors. There is no assumption that the two main effects are of the same size. This ‘main effects’ hypothesis is compared against the null hypothesis, i.e. that neither medit nor sex affect rt.

The BF for this hypothesis is large (about 1200). We’d expect this, given that there was substantial evidence for medit alone.

[4] `medit + sex + medit:sex`

This is the Bayes Factor for the hypothesis that there are main effects for both factors (medit + sex) and that the two factors interact (+ medit:sex). This is again compared against the null hypothesis that neither medit nor sex have any effect.

The BF for this hypothesis is also large (about 85). We’d expect this, because there was substantial evidence for the ‘main effects’ hypothesis medit + congru. However, the BF is considerably lower than for the main effects hypothesis.

Interaction BF

The Bayes Factor for the interaction is given, as before, by entering this comment and command into your script, and running it:

# Calculate Bayes Factor for interaction
bf2bs[4] / bf2bs[3]

Bayes factor analysis
--------------
[1] medit + sex + medit:sex : 0.07169096 ±2.63%

Against denominator:
  rt ~ medit + sex 
---
Bayes factor type: BFlinearModel, JZS

This gives us a Bayes Factor for the interaction around 0.07. So, there’s substantial evidence for the absence of an interaction.

Two within-subject factors

NOTE: In this section, we present the simplest way to perform this calculation. There are better ways. For further discussion, see Bayesian ANOVA for Pros

We can also use anovaBF to look at two within-subjects factors. Again, this works just the same way as before. For example, we can do a factorial Bayesian ANOVA for congru and block like this:

First, we create the appropriate subject-level summary for our analysis.

Enter this comment and commands into your script, and run them:

# Create subject-level summary
wordsum2 <- words %>%
  group_by(subj, congru,block) %>% 
  summarise(rt = mean(rt))

Next, for the purposes of this worksheet, we’re also going to reduce the sample size from 420 participants to 60 participants. This is something you’d never do in a real analysis, but here we need to address the issue that within-subject Bayesian ANOVAs can take a very long time to run when we have, as in this case, several hundred participants (in this case, it would take around 15 minutes). The correct approach when analysing your own data is, of course, to just wait for the analysis to finish.

Enter this comment and command into your script, and run it:

# Reduce sample size (FOR DEMONSTRATION PURPOSES ONLY !!)
wordsum2 <- wordsum2 %>% filter(as.numeric(subj) < 61)

Now we run the Bayesian analysis as before.

Enter this comment and command into your script, and run them:

# Calculate Bayes Factors
bf2ws <- anovaBF(
    formula = rt ~ congru*block + subj,
    data = data.frame(wordsum2),
    whichRandom = 'subj')

Explanation of output

The output is interpreted the same way as before; enter this comment command into your script, and run it:

# Display results
bf2ws

Bayes factor analysis
--------------
[1] congru + subj                        : 3830.96    ±0.76%
[2] block + subj                         : 0.03441736 ±0.81%
[3] congru + block + subj                : 136.0741   ±1.25%
[4] congru + block + congru:block + subj : 1.543383   ±2.04%

Against denominator:
  rt ~ subj 
---
Bayes factor type: BFlinearModel, JZS

`[1] congru`

The is the main effect of congruence. More specifically, it is the hypothesis that congru affects rt. This is compared against the hypothesis that there is no effect of congruence.

The Bayes Factor is very large (about 3,800), meaning the evidence for a main effect of congruence is conclusive.

`[2] block`

This is the main effect of test block. More specifically, it is a test of the hypothesis that block, affects rt.

The Bayes Factor for this hypothesis is around 0.04, which is strong evidence for the absence of an effect of block.

The interaction

[3] `congru + block`

The BF for this hypothesis is large - about 135. We’d expect this, given that there was substantial evidence for congru alone.

[4] `congru + block + congru:block`

This is the Bayes Factor for the hypothesis that there are main effects for both factors (congru + block) and that the two factors interact (+ congru:block). This is again compared against the null hypothesis that neither congru nor block have any effect.

The BF for this hypothesis is considerably lower than for the main effects hypothesis.

Interaction BF

The Bayes Factor for the interaction is given, as before. Enter this command into your script, and run it:

# Calculate interaction Bayes Factor
bf2ws[4] / bf2ws[3]

Bayes factor analysis
--------------
[1] congru + block + congru:block + subj : 0.01134222 ±2.39%

Against denominator:
  rt ~ congru + block + subj 
---
Bayes factor type: BFlinearModel, JZS

This gives us a Bayes Factor for the interaction is around 0.01. So, there’s substantial evidence for the absence of an interaction.

Single between-subjects factor

It is also possible to use anovaBF to analyse a single factor, between-subjects design. Where that factor has two levels, you could alternatively use ttestBF. One of the useful things about anovaBF is that, unlike ttestBF, you can use it in a one-factor design that has more than two levels. For example: (enter these comments and commands into your script, and run them:)

# Create subject-level summary
wordsum3 <- words %>%
  group_by(subj, medit) %>% 
  summarise(rt = mean(rt))
# Calculate Bayes Factor
bf1bs <- anovaBF(
    formula = rt ~ medit,
    data = data.frame(wordsum3))
# Display Bayes Factor
bf1bs

Bayes factor analysis
--------------
[1] medit : 718.3828 ±0.01%

Against denominator:
  Intercept only 
---
Bayes factor type: BFlinearModel, JZS

More than two factors?

Although in principle it is possible to run Bayesian ANOVAs with more than two factors, they can take hours or days to run with realistically large data sets. They also produce output that is considerably harder to interpret than the output of two-factor analyses. For these reasons, we only look at up to two-factor Bayesian ANOVA in these worksheets. If you need to conduct an ANOVA with more than two factors, consider using traditional ANOVA techniques (bearing in mind, of course, the interpretative difficulties surrounding p values - and that any kind of analysis with more than two factors is often going to be pretty difficult to interpret).

Exercise 1

The data for this first exercise comes from a between-subjects experiment with two factors. Start by loading the data and removing any missing data (NA), using the following command.

Enter this comment and commands into your script, and run them:

# EXERCISE
# Load data
cf_raw <- read_csv('going-further/counterfactual.csv') %>% drop_na()

The first few rows of data look like this:

subj	perspective	predictability	blame
136	first	Yes	6
137	first	No	6
138	first	Yes	4
139	first	Yes	6
140	first	Yes	4
141	first	Yes	4

In this experiment, participants read a scenario describing some events leading up to an accident. The scenario was written from either a first-person or a third-person perspective. Next, they generated some counterfactual thoughts in relation to the scenario. For example, “If only I hadn’t been looking at my phone, then …”. These thoughts were coded as either predictive, or not predictive of whether the accident occurred. Participants rated how much the protagonist was to blame for the accident, on a scale of 1 (strongly disagree) to 7 (strongly agree).

It’s crucial to look at your data before analysing it. Start by calculating the mean blame score in each of the four conditions of this experiment. You can do this using the group_by and summarise commands that we have covered in many previous worksheets; for example, in the group differences worksheet.

If you’ve done it correctly, your means should look like this:

# A tibble: 4 × 3
# Groups:   perspective [2]
  perspective predictability blame
  <fct>       <fct>          <dbl>
1 first       No              4.89
2 first       Yes             4.70
3 third       No              4.06
4 third       Yes             4.38

Next, plot those means in a line graph - this helps you visualize the patterns in these data. Plotting of line graphs was covered in the understanding interactions worksheet. Your graph should look like this:

Now, do your Bayesian ANOVA. Set each of the independent variables as a factor and then use anovaBF to conduct the Bayesian ANOVA. There is only one observation for each participant in this dataset, so don’t include subjor whichRandom=subj.

Your results should look similar to this:

Bayes factor analysis
--------------
[1] perspective                                               : 3.252175  ±0.01%
[2] predictability                                            : 0.2035932 ±0.03%
[3] perspective + predictability                              : 0.6617437 ±2.26%
[4] perspective + predictability + perspective:predictability : 0.3010063 ±4.16%

Against denominator:
  Intercept only 
---
Bayes factor type: BFlinearModel, JZS

Bayes factor analysis
--------------
[1] perspective + predictability + perspective:predictability : 0.4548684 ±4.74%

Against denominator:
  blame ~ perspective + predictability 
---
Bayes factor type: BFlinearModel, JZS

Write a short paragraph explaining what, on the basis of your analysis, you can conclude from these data. Copy this paragraph, and the code you used to do your analysis, into PsycEL.

Exercise 2

The data for this second exercise come from a within-subjects experiment with two factors. Start by loading the data, selecting the columns and rows we’ll need for this analysis, and removing missing data (NA).

Enter these comments and commands into your script, and run them:

# EXERCISE 2
# Load data, select Test phase and appropriate columns; drop NA entries
raw <- read_csv('case-studies/chris-mitchell/priming-faces.csv') %>%
  filter(Running == "Test") %>%
  select(Subject, Congruency, Load, RT) %>%
  drop_na()

The first few rows of data look like this:

Subject	Congruency	Load	RT
1	Congruent	NoLoad	684
1	Congruent	Load	371
1	Incongruent	NoLoad	907
1	Incongruent	Load	659
1	Congruent	NoLoad	453
1	Congruent	Load	969

Participants were trained to associate two screen colours with pictures of two different food rewards. At test, they saw pairs of screen colours and food pictures. Their reaction time (RT) in milliseconds was measured in cases where the pairs matched the associations they’d previously learned (congruent trials), and in cases where the pairs did not match what they had previously learned (incongruent trials). The participants experienced this test both with, and without, a secondary task. In the secondary task, the participant had to verbally rate how much they liked pictures of faces. As all participants completed all conditions, we have a fully within-subjects design, with factors congruency (congruent, incongruent) and load (load, no load).

Start by calculating the mean RT for each of the four conditions, and plotting these in a line graph. You can use the commands from Exercise 1, slightly modified, to do this. If you get it right, your output should look like this:

# A tibble: 4 × 3
# Groups:   Congruency [2]
  Congruency  Load      RT
  <chr>       <chr>  <dbl>
1 Congruent   Load    672.
2 Congruent   NoLoad  575.
3 Incongruent Load    665.
4 Incongruent NoLoad  616.

Next, calculate Bayes Factors for the main effects and the interaction. In order to do this, you will first have to create a subject-level summary of your data - you can then pass this your your Bayesian ANOVA.

Your results should look similar to this:

Bayes factor analysis
--------------
[1] Congruency + Subject                          : 0.3059866 ±1.85%
[2] Load + Subject                                : 19235.62  ±1.11%
[3] Congruency + Load + Subject                   : 6457.079  ±1.88%
[4] Congruency + Load + Congruency:Load + Subject : 2191.695  ±5.73%

Against denominator:
  RT ~ Subject 
---
Bayes factor type: BFlinearModel, JZS

Bayes factor analysis
--------------
[1] Congruency + Load + Congruency:Load + Subject : 0.3394252 ±6.04%

Against denominator:
  RT ~ Congruency + Load + Subject 
---
Bayes factor type: BFlinearModel, JZS

Write a short paragraph explaining what, on the basis of your analysis, you can conclude from these data. Copy this paragraph, and the code you used to do your analysis, into PsycEL.

This material is distributed under a Creative Commons licence. CC-BY-SA 4.0.

More on Bayesian ANOVA

Andy Wills, Clare Walsh, Paul Sharpe, Chris Mitchell

Contents

Introduction

Getting started

Two between-subject factors

Explanation of output

`[1] medit`

`[2] sex`

The interaction

[3] `medit + sex`

[4] `medit + sex + medit:sex`

Interaction BF

Two within-subject factors

Explanation of output

`[1] congru`

`[2] block`

The interaction

[3] `congru + block`

[4] `congru + block + congru:block`

Interaction BF

Single between-subjects factor

More than two factors?

Exercise 1

Exercise 2