Have you ever wanted to start a study with a new piece of software or something outside your normal method, and wished there was some existing experiment code that you could adapt for your needs?
A couple of people on the PsychoPy users list have suggested that it would be good to have a place to upload experimental code and materials to share.
It would serve a few purposes:
- makes a study genuinely replicable, because you would be able to fetch the actual experiment as the authors used.
- publicises an experiment that you've run because people could browse the repository looking for experiments they found interesting
- provides a starting point for new users of a piece of software to build an experiment
The repository would be agnostic to the subject of the study, and to the software used to run it. You would upload all the materials needed to run it (code, image files etc), tag which software package it was written for (PsychoPy, E-Prime, Presentation, Psychtoolbox etc...), provide a summary of what results should be expected and a reference to the paper showing the original (if published). Then you provide keywords about the topic that the experiment addresses so that people can browse or search for the experiment. Users might search by topic, keyword or software package to find experiments to learn from or replicate.
Potential issues
A few people have raised concerns about the idea:- Will it lead people to run studies that they didn't actually understand? For example, see this post on eagle-eyed-autism describing a study going badly wrong because the authors had borrowed code and hadn't really understood it. Is the answer to make sure it's very difficult to run studies, so that the scientist has to really know what you're doing in order to manage? That seems more than a little arrogant.
- Will errors in studies propagate more? If a study has an error, when another lab writes it from scratch the error will likely not be made, but if they borrow and tweak the bug could propagate. I think the benefit that more eyes potentially examine the experiment and reduce the propagation of bugs.
- Why should someone else simply take the experiment that I spent hours writing? To me this one just seems blatantly at odds with the aims and philosophy of science. But I guess some people will feel territorial like that.
- People would never use such a site (unless forced) because they will be too embarrassed by the quality of their code, which was, after all designed to work without necessarily being elegant. I'm fairly sympathetic to this (although I've obviously shared many thousands of lines of my own code). But some people will be brave enough to expose their work fully, especially if it was generated by something like E-Prime or PsychoPy Builder, where the need actually to write code is reduced.
The idea is definitely growing on me, although I don't currently have the time to build the site, nor the funding to pay someone to build it.
I'm keen to hear more views. So feel free to comment below. Hopefully the idea will also be discussed as part of a satellite event on open-science at the Vision Sciences Society conference this May.
