View on GitHub

rmip

Research Methods in Practice

Research Methods in Practice (Part 2) - Teachers’ notes

2025 edition.

Contents

Overview

References to ‘week X’ mean the Xth teaching week of the semester. For the 2024/25 Plymouth University timetable, teaching week 1 of semester 2 begins 20/01/2025.

Course organisation and staffing

Students work in groups of ~8 within workshop classes of up to 118, so there are three workshop groups in order to accommodate a cohort of 331 students. The course has 12 weeks of teaching. Throughout that time, there is a specified teaching assistant (Ph.D student or TARA) who is the supervisor of each group and is responsible for monitoring the progress of their groups.

Each week, students attend a two-hour early-week workshop, and a two-hour late-week workshop. The early-week workshops have a 20:1 student:staff ratio. One of those staff in the early-week session must be a member of academic staff (Lecturer / AP / Prof), who is expected to be present throughout the session. The late-week workshops are staffed solely by teaching assistants, at a 24:1 ratio.

In addition to these sources of support, a subset of the TAs run ‘hatch times’ on a rota (one TA staffs the hatch at any given time). These are drop-in times, run every week day from week 2 of term, until the last day of workshops (Friday of Week 15). The hatch is not open during vacation breaks, nor during public holidays. Opening hours vary:

After the main report deadline, support for students with Extenuating Circumstances is, until the EC deadline, via the office hours of the TA who worked with them throughout the semester. For TAs paid by the hour, these support hours are claimable.

Assessment

The module’s mark is based on 20% Practical and 80% Coursework, with no exams, but with pass/fail competency components.

20% - Group mark for a 10-min group oral presentation. The mark is weighted mainly on the quality of the research presented, rather than audio-visual slickness. There is a detailed quantitative mark scheme. Occurs during class in Week 10 (24-28 March 2025)

80% - Individual mark for an individually-written report. The mark is derived from a detailed quantitative mark scheme. Due Friday of Week 12, noon (2nd May 2025). Return to students Tuesday of Week 20 (3rd June 2025).

Pass/fail - There are a set of pass/fail competency PsyLab components, which also act as a lab book. In order to keep on track students should complete them in the week indicated (see below). Each activity closes on the Friday of the week following its introduction in class. There’s an exact timetable here. In order to complete it after that time, students must request their personal tutor to re-open it. Some of these are group components. In these cases, every member of the group still has to submit it individually, but they can all submit the same thing without penalty.

They have to pass at least 11 of the 13 activities to pass the module.

List of assessments

A full list of assessments, with proformas and model answers in most cases, are available here.

Resources

Each student will need a proper computer (PC, Mac, or Linux - not a phone, tablet or chromebook) in all sessions. There are a number of laptops in Link 301 that students can borrow.

For activities based around R, students will continue to use the RStudio server account they were allocated at Stage 1. They should be prompted a week before the module starts to request login details if they do not have them.

There is an online lab book system (see above).

For activities using Open Sesame, students will need to install this on their machine. There is time in the schedule for them to do this in class.

For online data collection, the students will need access to a JATOS server, so they can upload their converted OpenSesame experiment.

Employability skills

Communication skills. Group working to deadline. Project planning / timetabling. Critical thinking / analysis.

Ethics

Students write a detailed description of their methods; the discussion and approval of this document by staff provides a practical insight into ethics in these fairly innocuous studies.

Week 0: Pre-class preparation

One week before the first class, students must be emailed and requested to:

  1. Check they can log in to RStudio Server, and request login details from the Tech Office if they cannot.

  2. If they did not complete our Stage 1 for any reason, they must complete the Very Brief Guide to R, unless they have done so already.

  3. Complete the R revision worksheet, as the module assumes this information will be fresh in their mind. It should take approximately 30 minutes to complete for most students. MSc/direct entry students will need a bit more time (perhaps 15 more minutes) as they’ll need to read some additional material on effect size, which was not covered in the materials they have received to date.

  4. Ensure they have access to a proper computer (see above) for each session, and that they have at least a few GB of space left on their hard drive.

Week 1: Select topic and learn about statistical power

Resources for teachers

Early-week session:

min Contents
005 Teacher gets students to start installing Open Sesame
010 Teacher gives general introduction to PSYC520
015 Teacher gives a short introduction to each of the three topics and sub-topics.
030 Students select topic and organise into groups. Each TA allocated to 3 groups. Any given TA does one / two topics maximum.
055 Break
065 Students do the Introduction to OpenSesame worksheet, with support
115 Finish

Note 1: When introducing the topic, teachers should lay the groundwork for students to understand the concept of an interaction.

Note 2: This is probably Plymouth-specific, but note that there is one restriction to the principle that students can pick their own groups – group members must all be from the same exit-level of qualification (i.e. all B.Sc., or all M.Sc.). This is because M.Sc. work is marked to a higher standard, and the presentation is group marked.

Late-week session

Homework

Week 2: Define question, outline design and learn about data preprocessing

Teachers’ resources

Here are some slides on experimental design written about a decade ago for the Exeter Stage 2 practicals. Chris, Clare, and Michael have developed more recent versions for this module, available on the PSYC520/720 DLE.

Early-week session

min Contents
000 Teacher introduces getting from sub-topic to broad experimental design
030 With TA support, groups define question and plan broad design
050 BREAK
060 Start Data preprocessing worksheet. If they have not already tried out the facial prototypes experiment in OpenSesame, they should skip this part, otherwise they will run out of time
110 Finish

Note: In the Teacher’s presentation, there should be some focus of understanding the concept of an interaction.

Late-week session, and homework

Week 3: Finalise experimental design

High intensity session

min Contents
000 Teacher introduces getting from broad to specific design
020 With TA support, groups work towards a specific protocol
050 BREAK
060 Work continues towards a specific protocol
110 FINISH

All groups need to bear in mind the limitations of the testing time and space they have. In particular:

Late-week session and homework

Week 4: Build experiment

Resources

Students may find this library of OpenSesame scripts useful.

IMPORTANT: When collecting data online using OSWeb/JATOS, the data may not correctly save if the participant does not press the space bar at the end of the experiment. Students should mention in their instructions that the participant must press the space bar at the end to complete their participation.

Advice: how to implement between-subject experimental manipulations

There are (at least) two ways to implement between-subject experimental manipulations in OpenSesame:

The second method is preferable because:

  1. the same JATOS link can be given to all participants
  2. it will ensure that the experiment is otherwise identical for all participants

When using the second method, or for any experiment that uses separate blocks, make sure the same data logger is used in both blocks (they should have the same name). This will ensure the log file has a single header line at the top.

Both sessions

Both sessions are dedicated to achieving the following goal: Have a working, bug-checked OpenSesame script ready to go.

If they forgot to watch the homework video (see above), they have to start by doing that. Then the tasks, which students need to divide between members of the group include:

Each project should meet all the items on this checklist by the first session next week:

Homework

Week 5: Upload experiment and begin to learn about ANOVA

Early-week session

Session is dedicated to getting their experiment working on JATOS. If they’ve forgotten to watch the homework video (see above), they need to start there.

Late-week session

min Contents
000 Upload of experiment to JATOS
050 Complete within-subjects differences ANOVA worksheet
110 Finish

Advice: Tips for conversion to OSWeb/JATOS experiment

Instructions on converting the OpenSesame experiment script for use in JATOS are given in this document

Here are a few things to watch out for when converting to OSWeb:

Advice: “Last working version” debugging of R code

If a student’s command in R is not working but is individually correct, then they have likely made an error at a previous step. This might have involved ignoring an earlier error message, or it might have involved doing something that was wrong but didn’t immediately generate an error (like mis-naming a new variable).

In these circumstances, you do ‘last working version’ debugging. In other words, go back through their previous commands until you find the error - the worksheets generally tell you what is expected as output at each step.

Homework

Week 6: Collect data and finish learning about ANOVA

Early-week session

min Contents
000 Complete Understanding interactions worksheet
060 BREAK
070 Data collection support
110 Finish

Data must be collected only from other students enrolled on the module. Students can act as participants at any point across weeks 6 and 7, during or outside class. However, there are 10 hours of class time scheduled for ‘data collection support’ to facilitate this. Data collection support, from the teachers’ perspective, involves two activities:

  1. Giving each group a list of JATOS links for experiments that each member of the group should participate in (usually through a shared spreadsheet linked on DLE).

  2. Troubleshooting/facilitating the process of data collection.

For online data collection, students should be reminded of the following:

TAs should also check that experimental data are being saved correctly in JATOS. This should be done as soon as data collection has started to avoid students being left without any data at the end of week 7. One way to check this is to attempt to export the JATOS data of the first few participants, convert them to a csv file using OpenSesame (see Chris Longmore’s “Exporting JATOS results” video) and check that the relevant data are present.

Late-week session

min Contents
000 Complete Factorial differences worksheet
040 BREAK
050 Data collection support
110 Finish

Homework

Week 7: Finish data collection

Both sessions are dedicated to the completion of data collection, plus any support students need for any R worksheet.

Again, TAs should make sure experimental data are being correctly saved on JATOS.

NOTE FOR 2025: In 2025, the Easter break will be between weeks 11 and 12. To give enough time to students to prepare their presentation (week 10), it may be wise to make them download/convert their data at the end of week 7 instead of the first session of week 8, so they can focus on analysing their data in week 8.

Homework

Week 8: Analyse data

Both sessions are dedicated to analysing and interpreting the collected data. Data analysis should ideally be submitted to PsyLab by the end of the week.

Week 9: Write a presentation

During this week the teacher must establish a schedule of presentation times for week 10 (see Week 10 below for details).

Early-week session

There are two components to this session:

  1. Supporting analysis (if not yet completed, see above), and interpretation of that analysis
  2. TA-led support for presentation planning (see Notes).

The order and relative emphasis of these activities will likely differ between groups.

Note 1:

TA and group reprise what they did, what they have found, and discuss any problems/limitations of what they have done. TA then leads discussion of examples of very clear, and very unclear, lectures they have encountered since joining Stage 1. What do these good/bad experiences have in common? Possible issues - Rate of presentation; incorrectly assumed knowledge; over-crowded slides; a million facts but no big picture; a big picture but no facts; conclusions do not clearly follow from facts. Then, TA supports group to sub-divide the task of writing a presentation between them. The sections would be something like: (A) Big picture overview, (B) Introducing the specific question, (C) Methodology, (D) Results, (E) Interpretation, (F) problems and next steps, (G) writing an abstract. Agree an allocation of time and number of slides each.

Note 2:

Each group must be allocated a one-hour presentation slot for week 10. Half of groups will present in their early-week session, and half in their late-week session. With table group size of 7, each Workshop Group will have 16 Table Groups to distribute across the four available one-hour slots. Sort which groups go when during week 10, and post the allocations to the DLE before the week 9 late-week session. Use the flexibility of having two sessions (from the student’s perspective) to accommodate groups who have serious reasons for not being able to make one of the sessions (e.g. funeral of a close relative). If we can’t accommodate this, then absent members have to do the Extenuating Circumstances assessment instead.

Note 3:

Students should be expected to produce graphs that have proper labels that are of a readable size. Some of the things they need to know about this are covered in the Absolute Beginners’ Guide, but the best source is the Better Graphs page.

Late-week session

This session is dedicated to ensuring all groups are ready for the week 10 presentation. This may include:

Students should submit presentation title, author names, and abstract by end of the late-week session.

Week 10: Give presentation

All the sessions this week are identical – each is staffed by one TA and one Teacher (who double mark). In each Workshop, there are a maximum of 15 groups; with 220 minutes of usable time across two two-hour sessions, that is less than 15 minutes per presentation. The presentations are 10 minutes, plus 2 minutes for questions. This allows around 160 seconds for a handover to the next group, so one cannot allow students to over-run their alloted time. Give them 5-minute, 1-minute, and STOP signs. Don’t let them talk more than 1 minute beyond ‘STOP’, and reduce their question time accordingly. Anything else is unfair on those students who planned appropriately and kept to the time limit.

Students must hand in their presentation slide in advance of the workshop in which they will present so that all slides are loaded on the presentation computer before each session. Handing in the presentation time on time is one of the marking criteria.

Marking process for group presentations

During the session itself, focus on giving frank feedback that will help them with the next stage of this module. You can also give presentational feedback, of course.

Immediately after the end of each session, the Teacher and TA agree component scores and feedback for each group. The Teacher then immediately emails the agreed scores/feedback CSV file, and the groups/attendance CSV file to the module leader. Note that you do not assign an overall grade, this is done during moderation, although if you have a suggested grade for the groups, feel free to put those in the body of the email. It is important that correctly formatted CSV files are used because much of the subsequent process is automated. The links give examples of the format needed. In particular:

Group_ID: Use their self-assigned group name, but it must be unique for each group and consistent within a group. If they didn’t give themselves a name, assign a bland one (e.g. Workshop3Group4). It is absolutely VITAL that you use the SAME Group_ID in the two CSV files (feedback.csv and groups.csv). For example do not use, “W3G2” in one file, but “3.02” in the other file.

PU_email: Give their full Plymouth University email address. Do not give their name (names can be non-unique, and emails make it easy to email feedback). You can find the email address for any student enrolled on the module using the classlist system (accessible via S4).

present: 1 = present, 0 = absent. It’s important to explicitly record the absent, as well as the present, students in each group.

Component: Use these exact words to describe the mark components. It’s really important that you do not mis-spell the Components, e.g. “Clairty of speech”, because the script will treat this as a different component. Also: USE ALL COMPONENTS FOR ALL GROUPS. It is not OK to, for example, drop the “advice on next steps” row if you don’t have any advice.

Score: Use 1, 2, 3, 4, or 5. 1 = Poor, 2 = Patchy, 3 = OK, 4 = Good, 5 = Excellent. Zero is not a valid score.

Feedback: Write what and as much as you like, but don’t use formatting (bold etc) and don’t use carriage returns - just keep typing on a single line. Apologies for the following basic advice, but it’s all based on errors from the first time we ran this: (1) Start all your sentences with a capital letter, (2) avoid spelling and grammatical errors, (3) avoid abbreviations, (4) avoid note form, (5) if it’s a question, use a question mark – these comments are passed directly to students without editing!

Empty rows: There should be no empty rows anywhere in the completed files (e.g. do not separate groups with empty rows).

NA: The only place “NA” should appear is for the score for “advice on next steps”. It is not a valid feedback comment, for example. Also note that N/A is not the same as NA. Use only the latter.

Once the Module Leader receives files from all sessions, they moderate, and return overall grades and feedback to the students via email. The ML also uploads overall grade to the DLE. These processes are largely automated by R scripts. The final step for the module team is to release the marks to the students (again done by the ML). This done via Moodle, folllowing these instructions. From this point, the marks are imported into the university’s record system (“Unit-E”) by Faculty administrative staff, for preparation of exam board materials.

Students are not required to upload their presentations to the DLE.

Marking PSYC720

Although PSYC520 (a B.Sc. module) and PSYC720 (a M.Sc. module) were co-taught, the marking of the presentation is done entirely separately, with different marking criteria. Given you do not award a grade, much of this process is handled during moderation. However, there are still some important differences:

Week 11: Write a report plan

Early-week session

min Contents
000 Presentation feedback and reflection
050 BREAK
060 Talk on report writing
110 Finish

In the first hour, the group and TA reflect on their results, and the feedback received on their presentation. They discuss what they would have done differently if they were starting again. In the second hour, the Teacher gives a talk on report writing.

Here are some slides on report writing I put together for a similar course at Exeter about a decade ago. There is an updated version of these slides on the course DLE.

Late-week session and homework

Students write and submit a bullet-point plan for their report, with support.

Homework

EASTER BREAK

There is a now a three-week hiatus for Easter Vacation. The module is paused, and no support is available; questions must be held over until the beginning of term.

Week 15: Critique and support

The end of the late-week session in Week 12 is the last opportunity for students to receive support on this module (unless they have Extenuating Circumstances, in which case support continues through the office hours of their TA). It’s important to make clear to students that this means all forms of support, including email, are over by the end of this session. Between then and the Friday Week 12 deadline, they are on their own.

Early-week session

min Content
000 TA-led good-and-bad reports activity (see Note 1)
025 TA-led discussion of bullet point plans (see Note 2)
050 Open Q & A, and support (see Note 3)
110 Finish

Note 1: Groups are given one example 1st class and one example 2ii report. Can they tell why those two reports differed in mark? What are the good points of the 1st class report? Where did the 2ii report go wrong?

Note 2: All groups working with a TA get together. Each group, with an audience of that TA’s other groups, talks through their bullet point plan. TA and other groups provide feedback on what they found clear / unclear.

Note 3: The second half of the session is an open Q&A, and support session, so students are best prepared to write their report. In there is time, or a lack of queries, support them in the following:

APA referencing

Common issues of grammar and style

Late-week session

This session falls the day before the report deadline. It’s an open Q&A / support session.

Week 16-20: After the workshops end…

TA guide to marking the main report

TAs mark the reports of the groups they led (normally around 20 reports). The total time this takes should not exceed one hour per report (excluding breaks, which you should take regularly). It’s likely you will get faster with practice, but if your average over the first five reports is substantially more than an hour, speed up (e.g. write briefer comments). For TAs on an hourly rate, report marking is paid at one report per hour.

In addition, allow two hours for the initial markers’ meeting and the preparation required for it.

Timetable

Friday 2nd May 2025, mid-day (Week 15) - Submission deadline passes. Module leader sends marking sample to markers. Markers must not mark any further reports until after initial markers’ meeting (below)

Thursday 8th May 2025, mid-day (Week 16) - Markers return marking sample with scores, but without feedback, to module leader.

Friday 9th May 2025, 1pm (Week 16) - Initial markers’ meeting between all markers and module leader. Immediately after the meeting, you may start marking.

Monday 12th May 2025, mid-day (Week 17) - Submission grace period passes, no further submissions permitted.

Monday 2nd June 2025, mid-day (Week 20) (HARD DEADLINE!) - All reports commented in Turnitin. All mark spreadsheets emailed to Module Leader. Process of moderation begins.

Tuesday 3rd June 2025 (Week 20) - Reports and marks are returned to students via DLE.

Marking process

The marking process is as follows:

  1. On the day of the submission deadline, every marker receives the same four reports to mark. Unlike all subsequent marking, you do not write feedback on these reports. You just read the report and allocate the 25 scores using the score spreadsheet (for details, see below). This should not take you more than about 20 minutes per report.

  2. Within one week, email your marks for these four reports to the module leader.

  3. You’ll then have a single, 30-minute, Zoom meeting with all markers and the module leader to discuss how we all have marked these four reports. The module leader will have also marked the reports. The purpose of this meeting is to ensure you are all marking to the same standard.

  4. If you worked in Clare Walsh’s workshop, some of your students were registered for PSYC720 (M.Sc. Psychology), not PSYC520 (B.Sc. Psychology). So, you’ll have to access both the PSYC520 and PSYC720 DLE pages.

  5. Mark the report in the usual way:

    5.1. Use the Turnitin tool to add helpful, specific, supportive comments as you go.

    5.2. Do not use sarcasm. Do not belittle. Don’t write anything you wouldn’t want to receive on your work. If something is wrong, say so, but also say what is right.

    5.3. The mark scheme indicates the sorts of things you should be giving feedback on.

    5.4. When you have finished reading and commenting on the report, fill in the ‘overall comments’ section on the cover sheet. You should aim to say three ways in which the report was good, and three ways in which it could have been improved. Your scores CSV file should help you decide what to write here.

    5.5. Note that you do not assign an overall grade, this is done during moderation, although if you have suggested grades, feel free to tell me what they are (in an email, not on this sheet). If you are curious about the process of moderation, including how number marks are mapped to letter grades, the R script is available in the github repository for this site - see scripts/report-marks.R.

  6. As you mark, fill in the scores CSV file:

    6.1. This contains an example student, obviously remove this before returning the CSV file to the Module Leader. More generally, this file is automatically read, so it’s really important you get the format exactly right. So:

    6.2. Use a single file for all your marking within the module (not one file per student).

    6.3. It is vital that the Identifier column contains the student identifier exactly as shown on the DLE. So put e.g. “Participant 11396739”, not “11396739” or “Participant_11396739”. Do NOT put the Student Reference Number in the Identifier column. Do NOT type in these numbers by hand, ALWAYS copy and paste from the DLE. Typing in by hand leads, across 300+ reports, to various copy errors, which are devilish to fix, particuarly if your error is actually also a valid Identifier.

    6.4. In the Score column, put a number from 1 to 5. Only use these numbers. Do not use fractional numbers. Do not use zero. Do not use NA. The numbers have the following meanings: 1 = Poor, 2 = Patchy, 3 = OK, 4 = Good, 5 = Excellent

    6.5. In the Component and Description columns, use the descriptions exactly as they appear in the example. Use copy-and-paste to ensure you do not make mistakes.

    6.6. Do not leave any blank cells. So, for example, the Identifier should appear on every line for that student. Use copy down in your spreadsheet application to ensure you don’t make errors. Exactly how this works depends on your application, but for LibreOffice Calc and Microsoft Excel, highlight the cells you want to fill with the contents of the first row and press Ctrl+D.

    6.7 Do not leave any blank lines.

  7. When the EC deadline passes, check to see if you have received any new reports, and mark those. Also check to see if anything you have already marked has been updated since you marked - if it has, you will have to re-mark it.

  8. When you have marked ALL your reports, send your marks CSV file to the module leader.

  9. If any matter arises that is not covered by the marksheet - for example evidence of plagiarism, collusion, or violation of formatting or page length rules - let the module leader know via email of your concerns, then mark as if you had not noticed these issues. The module leader will consider each of these exceptional cases and take appropriate action.

Marking PSYC720

Although PSYC520 (a B.Sc. module) and PSYC720 (a M.Sc. module) were co-taught, the marking of the main report is done entirely separately, with different marking criteria. Given you do not award a grade, much of this process is handled during moderation. However, there are still some important differences:

  1. Mark PSYC520 first, so you have a clear picture of the average standard of B.Sc. work.

  2. Now mark PSYC720. Relative to your B.Sc. marking you should:

    11.1. Give less ‘benefit of the doubt’: So, in an UG report you might find something unclear but be able to work it out eventually (perhaps because you are familiar with the project). In these cases we tend to give some ‘benefit of the doubt’ in that we assume the student meant the right thing and just didn’t express it that well. For an M.Sc. student, don’t try so hard to work out what they are saying. If it’s unclear, treat it as if it is wrong.

    11.2. We expect a higher standard of written (and graphical) expression at M.Sc. level. At B.Sc. level, in some weaker cases the quality of English is not that great. And, at B.Sc. level, we would try quite hard to see beyond that to the substantive content. For an M.Sc. piece of work, we expect consistently good-quality writing that shouldn’t need to be ‘decoded’ into a clear expression by the reader.

After marking

After the marking is complete, the ML runs a statistical moderation process (for details, see the script folder in the github repository for these pages), and then uploads the marks to Moodle (the DLE). The final step for the module team is to release the marks to the students (again done by the ML). This done via Moodle, folllowing these instructions. From this point, the marks are imported into the university’s record system (“Unit-E”) by Faculty administrative staff, for preparation of exam board materials.