Lenard Dome, currently a graduate student in my lab, has just released an R package for parameter space partitioning onto CRAN (psp). I helped a little bit. But, what is parameter space partitoning?
It’s easiest to start by talking about what it is not. It’s not model fitting, at least in the sense that is usually meant. One common way to evaluate computational models in cognitive science is to vary their free parameters until the difference between the model’s output and the observed data is minimized - model fitting. This tells us how quantitatively close a model can get to the observations.
What model fitting doesn’t tell us is how many other, unobserved, results the model could have also fitted (Roberts & Pashler, 2000). Metrics such as the Bayesian Information Criterion (BIC) attempt to correct fits for model flexibility, such that models varying in flexibility can be compared. This approach has some limitations (see Wills & Pothos, 2012, p. 120). BIC also doesn’t provide a direct answer to the question of what other unobserved results the model could have fitted.
Parameter space partitioning (Pitt, Kim, Navarro & Myung, 2006) provides a method for looking at model flexibility in this way. In brief, one specifies model outputs as a set of equalities and inequalities. For example, accuracy of response might be higher for stimulus A than stimulus B and C, which are the same: A > [B, C]
. For three cues, there are 30 different patterns. In any particular experiment, only one of those 30 patterns is observed at the group level. How many of the other 29 patterns can the model produce? Parameter space partitioning allows us to answer this question.