Experimental materials, data, and code for the paper Tracking the Eye of the Beholder

Authors: Andrew Stewart, Henrik Singmann, Matthew Haigh, Jeffrey Wood, and Igor Douven. Division of Neuroscience and Experimental Psychology and Department of Computer Science.

This paper reports the results of two studies examining the comprehension of explanations during reading. There is much recent evidence showing that explanation is central to various cognitive processes. On the other hand, philosophers have argued that the notions of explanation and explanation quality are too subjective to play any role in science: what may be an adequate explanation for one person may fail to be so for another. We compare the results of a study tasking participants with rating explanation quality with those of an eye-tracking study, finding that ratings of explanation quality from participants in the former study were strongly predictive of the ease with which participants in the latter study processed text fragments presenting the same explanations that were used in the rating study. This finding undermines the thought that explanation is only in the eye of the beholder. 

Applying open research practices

Alongside this paper, we published the experimental materials used during the comprehension tasks, the offline rating and eye-tracking data from the tasks, and the analysis code. The analysis was conducted in R and we linked the paper to the .Rmd script and the resulting .html knitted output which contains the R code chunks, the associated output, and substantial narrative explaining the purpose of each code chunk as well as interpreting the output generated by that chunk.  

Overcoming challenges

The main challenge was associated with capturing the version of R and the packages used. In future, we will package up the full computational environment using Binder or Docker.  

Benefits of using these open research practices

Providing both the data and analysis script in an annotated and useable format meant the reviewers could inspect the code and run it on their own machines to see how the various statistical models we used fitted our data.  

Top tip

Ensure that analysis scripts are sufficiently well documented so that it is clear to a reader what the purpose is of each component of the script. 

Find out more

The experimental materials, data, analysis scripts and paper can be found below.

Related links