Contents

Introduction

This worksheet describes analyses you are likely to need when working with pscyhometric scales.

Getting started

To prepare for this worksheet:

  1. Open the project you used to complete the Preprocessing scales worksheet.
  2. Add the code below to the end of scales.R.

Testing scale reliability using Cronbach’s alpha

Asking people questions is one way of measuring psychological constructs. For example, if we wanted to measure people’s stress levels over the past week, we could ask them to rate how much they agree with the statement “I found it hard to wind down”. Because language is complex, questionnaires require careful design to give us confidence that the answers to our questions provide a reliable measure of the underlying construct we want to measure. A questionnaire would normally use multiple questions to produce a reliable measure of a construct like stress. These groups of questions are often referred to as a ‘scale’. One way to measure the reliability of a scale is to test the extent to which responses to different questions on the same scale co-vary (or more informally, how they ‘correlate’). If the questions measure the same thing, we would expect the answers to co-vary. Cronbach’s alpha is a test which uses the covariances between questions to provide a measure of scale reliability.

Cronbach’s alpha is a value from 0-1, with higher numbers often considered to indicate a more reliable scale (although this is debatable). In general, values of alpha between 0.7-0.8 indicate an acceptable level of reliability, and values substantially below this are suggestive of an unreliable scale. However, these values vary depending on the construct being measured, so it’s advisable to check the acceptable alpha range for the specific scale you are using.

A few cautionary points relate to the way that Cronbach’s alpha is calculated. First, alpha increases as more questions are added to a scale, so it would artificially report higher levels of reliability in a scale with lots of questions. Second, if your scale consists of subscales (for example it measures depression and anxiety, as well as stress) you should calculate alpha for the subscales and not the whole scale. This is because subscales measure different things, and hence we would not expect them to strongly co-vary. For example, although depression and anxiety co-vary to some extent, we can imagine someone who is depressed but not anxious, or anxious but not depressed. So, we expect that our anxiety and depression subscales are measuring different things, which means even if our questionnaire was a good one, we would not necessarily expect high co-variance between these two scales. Finally, reverse-coded items will artificially decrease the value of alpha, so ensure you have reversed any reverse-coded items before calculating alpha. Reverse coding was covered in the Cleaning up questionnaire data worksheet.

It’s advisable to ensure that Cronbach’s alpha in your sample data is similar to the value reported in the research paper which describes the scale you are using. We’ll demonstrate this for the State Self-Esteem Scale (SSES), which was introduced in the Preprocessing scales worksheet. The SSES contains some reverse-coded items, but these have already been adjusted in the raw data, so we don’t need to do that step in this worksheet.

We’ll use the cronbach() function from the psy package to calculate Cronbach’s alpha for the SSES.

Enter these comments and commands into your script, and run them:

# Analyzing scales
# Load 'psy' library, for 'cronbach' command
library(psy)
# Calculate Cronbach's alpha on Q1 -> Q20
sses_cronbach_total <- sses_pre_raw %>%
  select(q1:q20) %>%
  cronbach()
sses_cronbach_total$alpha
[1] 0.6862132

Explanation of commands:

  1. The cronbach() function expects a wide data frame (one row per subject), with columns containing the scale scores. We supply this using the function select(q1:q20). Notice how we can use the column names to specify the range of columns to select().

  2. In addition to the value of alpha, cronbach() returns the number of subjects and the number of items. Here were only interested in alpha, which we print with sses_cronbach$alpha.

This gives us a value of 0.69, which is much lower than the alpha of 0.92 reported by the authors who developed the SSES (Heatherton & Polivy, 1991).

Exercise

Calculate Cronbach’s alpha for the depression, anxiety and stress subscales in the DASS-21 data introduced in the Preprocessing scales worksheet. In each case, round alpha to 2 decimal places. The command select(dass21, c('DASS3', 'DASS5', 'DASS10', 'DASS13', 'DASS16', 'DASS17', 'DASS21')) selects the columns for the depression subscale. Use similar commands to select the correct columns for the anxiety and stress subscales. You can find the items for these subscales in this article. Cronbach’s alpha should be 0.92 for depression, 0.84 for anxiety, and 0.86 for stress.

Copy the R code you used for this exercise, including appropriate comments, into PsycEL.

References

Heatherton, T. F., & Polivy, J. (1991). Development and validation of a scale for measuring state self-esteem Journal of Personality and Social Psychology, 60(6), 895.

This material is distributed under a Creative Commons licence. CC-BY-SA 4.0.