We moved :)

Please find the updated FORRT Website here where you can find what we are up to these days. Check out the self-assessment learning tool, FORRT’s Manuscript, FORRT’s curated resources, and FORRT Code of Conduct.








A Framework for Open and Reproducible Research Training1

Introduction


Abstract

Current norms for the teaching and mentoring of higher education are rooted in anachronistic practices of bygone eras. Improving the transparency and rigor of science is the responsibility of all who engage in it. Ongoing attempts to improve research credibility have, however, neglected an essential aspect of the academic cycle: the training of researchers and consumers of research. Principled teaching and mentoring involve imparting students with an understanding of research findings in light of probabilistic uncertainty, and moreover, an understanding of best practices in knowledge advancement. We introduce a Framework for Open and Reproducible Research Training (FORRT). Its main goal is to provide educators with a pathway towards the incremental adoption of principled teaching and mentoring practices, including open and reproducible research. FORRT will act as a resource to support instructors, collating existing teaching pedagogies and materials that may be reused and adapted for use within existing courses. Moreover, FORRT can be used as a tool to benchmark the current level of training students receive across six clusters of open and reproducible research practices: ‘reproducibility and replicability knowledge’, ‘conceptual and statistical knowledge’, ‘reproducible analyses’, ‘preregistration’, ‘open data and materials’, and ‘replication research’. FORRT will strive to be an advocate for the establishment of principled teaching and mentorship as a fourth pillar of a true scientific utopia.

Above is the abstract for our manuscript introducing FORRT https://osf.io/bnh7p. We welcome input on FORRT itself, and on the manuscript; you can access the working document here (please select suggested changes): tinyURL.com/FORRTworkingDOC.

We are actively seeking help to develop FORRT into the pedagogical resource we envisage. If you would like to help us support and advocate for teachers, please get in touch here. Please also see the ‘implementation plan’ tab for more information on our current plans.



Principles


FORRT is centred around a framework comprising six clusters of open and reproducible research practices. Each cluster has six sub-clusters.


Cluster 1: Reproducibility crisis and Credibility Revolution

Summary: Attainment of a grounding in the motivations and theoretical underpinnings of reproducible and open research. Integration with field specific content (i.e., or grounded in the history of replicability);

  • Replication crisis and credibility revolution
  • Exploratory and confirmatory analyses
  • Questionable research practices (their ‘theory’) and prevalence
  • Proposed improvement science initiatives on statistics, measurement, teaching, data sharing, code sharing, pre-registration, replication.
  • Ongoing debates, (e.g. incentives for and against open science).
  • Ethical considerations for improved practices.


Cluster 2: Conceptual and Statistical Knowledge

Summary: Enacting this principle indicates that students attain a grounding in fundamental statistics, measurement, and its implications.

  • The logic of null hypothesis testing, p-values, Type I and II errors (and when and why they might happen).
  • Limitations and benefits of NHST, Bayesian and Likelihood approaches.
  • Effect sizes, Statistical power, Confidence Intervals.
  • Research Design, Sample Methods, and its implications for inferences.
  • Questionable research (QRPs) & measurement practices (QMPs).
  • Understand the relationship between all of the above.


Cluster 3: Reproducible analyses

Summary: Reproducible analyses allow the checking of analytic pipelines and facilitate error correction. Enacting this principle requires students to move towards transparent and scripted analysis practices.

  • Strengths of reproducible pipelines
  • Scripted analyses compared with GUI
  • Data wrangling
  • Programming reproducible data analyses
  • Open source and free softwares
  • Tools to check yourself and others; statcheck, GRIM, and SPRITE


Cluster 4: Open data and materials

Summary: Enacting this principle indicates that students have attained a grounding in open data and materials in both; using and sharing.

  • Knowledge of traditional publication models. Open access publishing, preprints
  • Reasons to share; for science, and for one’s own practices
  • Repositories; e.g. OSF, FigShare, GitHub
  • Accessing/sharing others data, code, and materials
  • Ethical considerations
  • Examples and consequences of accessing un/open data


Cluster 5: Preregistration

Summary: Preregistration entails laying out a complete methodology and analysis before a study has been undertaken. This facilitates transparency and removes several potential QRPs.

  • Purpose of preregistration - distinguishing exploratory and confirmatory analyses, transparency measures
  • Preregistration and registered reports - strengths and differences
  • When can you preregister? Can you preregister secondary data?
  • Writing a preregistration
  • Comparing a preregistration to a final study manuscript
  • Conducting a preregistered study


Cluster 6: Replication research

Summary: Replication research takes a variety of forms, each with a different purpose and contribution. Reproducible science requires replication research.

  • Purposes of replication attempts - what is a ‘failed’ replication?
  • Large scale replication attempts
  • Distinguishing conceptual and direct replications
  • Conducting replication studies; challenges, limitations, and comparisons with the original study
  • Registered Replication Reports
  • The politics of replicating famous studies



Although not exhaustive, these concepts provide a broad coverage of each FORRT cluster.

Implementation Plan


Our developing implementation plan is as follows (we are also open to feedback and suggestions):

  • Continue to develop FORRT. The ‘introduction to FORRT’ manuscript is intended as a starting point to elicit widespread feedback about the framework and its implementation.
  • Check with the community whether the pilot for the evaluation and its preliminary rankings can be improved upon.
  • Invite course leaders to become the first adopters of FORRT; to submit their materials, to complete a pedagogical assessment of their courses, and to join us in FORRT
  • To invite diverse feedback on the optimal implemention of the educational NEXUS. Including assistance with website development.

We are actively seeking help to develop FORRT into the pedagogical resource we envisage. If you would like to help us support and advocate for teachers, please get in touch here.



  1. Authors note. This project was initiated at the 2018 meeting of the Society for the Improvement of Psychological Science in the “Teaching replicable and reproducible science” hackathon led by Kristen Lane and Heather Urry. The initial framework was developed in a subsequent working group consisting of: Sam Parsons, Flavio Azevedo, Carl Michael Galang, Kristin Lane, Lisa DeBruine, Benjamin Le, Donald Tellinghuisen, and Madeline Harms (we apologise if we have missed anybody - please let us know). The current version of the paper introducing FORRT was drafted by Sam Parsons & Flavio Azevedo, with integral feedback and support from Carl Michael Galang. The FORRT website has been designed - and is maintained - by Flavio Azevedo. To further develop FORRT we are seeking additional contributors and will endeavor to acknowledge all that provided feedback on this project.↩︎