Skip to content

09 - Open Peer Review#

This course will introduce you to Open Peer Review on the OEP. At the moment it provides basic insights into Open Peer Review.

You can provide feedback to this course on GitHub: https://github.com/OpenEnergyPlatform/academy/issues/189

Note

The Open Peer Review on the OEP is currently under development!

For whom is this training and what can you learn?#

This course is aimed at researchers, who

  • publish data and seek a quality check for the metadata to ensure accuracy and completeness
  • want to review and enhance an existing dataset, ensuring its reliability, relevance, and usability

After reading the sections of this training course you will

  • understand the developed open peer review for open data
  • be able to start and complete an open peer review on the OEP
  • contribute to and improve the implemented review process

Introduction: What is Open Peer Review?#

In the realm of open science, Open Peer Review (OPR) is gaining increasing significance as an alternative approach to evaluating scholarly works. Unlike traditional peer review processes, which often lack transparency and clear criteria, OPR distinguishes itself through specific characteristics. This chapter delves into the features and benefits of OPR, focusing on the criticisms of conventional peer reviews and highlighting the importance of transparency, open participation, and identifiable contributors in the evaluation process. Additionally, we explore how OPR facilitates broader involvement from experts and interest groups, leading to improved quality and validation of research findings.

Ross-Hellauer's 2017 study identified four key features that differentiate an open peer review from conventional peer review.

Open Identities#

Open Peer Review (OPR) adopts an unblinded review approach, where in contrast to traditional peer review processes, both reviewers and authors are openly acknowledged and documented. In a single-blind review, the identities of the reviewers are known to the authors (or contributors), while the authors' identities remain concealed. This aims to mitigate biases that could arise from authors' reputations or affiliations. A downside is the lack of incentive for the extra work and time. In contrast, the open identities in OPR serve as a motivating factor for conducting accurate work and ensure visibility of contributions within the community.

Open Reports#

Open reports entail the inclusion of the reviewer's report along with their name. This introduces transparency to what was previously an opaque and boolean process. Like open identities, this heightened transparency is anticipated to enhance the quality of reviews, as the report becomes accessible to the public. It serves as an additional incentive or reward for reviewers.

Open Participation#

Open participation expands the review process beyond a single person or a small group of experts to include a larger community or group. While traditional peer review often involves inviting a select few individuals to participate, open participation allows interested parties to provide structured feedback or comments. By increasing the number of reviewers, the dominance of individual interests can be avoided. Embracing multiple perspectives and diverse knowledge enhances the accuracy of the review, which may be lacking when conducted by an individual alone. Additionally, previously overlooked groups have the opportunity to contribute to the process. Therefore, not only selected individuals from academia, but also groups from industry or members of interest groups can actively participate.

Open Platforms#

Open Platforms play a crucial role in facilitating the implementation of open peer review and fostering collaboration within the scholarly community. It is a digital space where researchers can engage in transparent and interactive review processes. Needed functionalities are data or document sharing, version control, commenting, and collaborative editing, enabling reviewers and authors to exchange feedback and ideas seamlessly. The platform needs to incorporate features that support open identities, open reports, and open participation, ensuring a transparent and inclusive environment for the review process.

The Open Energy Platform (OEP) serves as a community platform for conducting Open Peer Review (OPR) in the field of energy research. It provides a robust infrastructure that enables researchers to openly share and review energy-related datasets, methodologies, and findings. With its features designed for transparency, collaboration, and open participation, the OEP empowers the energy research community to engage in rigorous and inclusive peer review, driving the advancement of knowledge and fostering innovation in the field of energy.

How do you review open data?#

[Tutorial on Open Peer Review] (link to tutorial once its published) This tutorial gives a detailed step by step description of the Open Peer Review Process on the OEP.

Adapting an open peer review process to open data and its metadata involves a structured approach to ensure the quality and reliability of the information. The current development on the OEP provides the functionality to review and comment on every metadata key of the published data. The reviewer can either Accept, Suggest, or Deny an entry and can provide a value suggestion and a comment. The review is structured after the main sections of the Open Energy Metadata standard. It offers a progress indicator and a summary page.

How to evaluate and present the review? What are badges?#

Based on a scientific survey, the individual metadata keys were evaluated and categorized into groups. By accepting all entries in this section, a corresponding badge is awarded. These badges are named after precious metals such as "iron," "bronze," "silver," "gold," and "platinum."

  • Iron – Technically required for data structure
  • Bronze – Basic description of the data
  • Silver – Supplement description of the data
  • Gold – Extended description of the context
  • Platinum – Ontological annotation

Who is involved in a review?#

Data Contributor#

The original creator of a table has all permissions and is the owner.

Reviewer#

Every registered member of the OEP can start a OPR of every table. You cannot start the review of the tables if you are the owner.

Editor / Community Manager#

The editor asks members to start and submit a review for specific tables. When a review started and is not completed the editor can close the review to proceed. In case the contributor and reviewer do not come to an agreement, the editor decides. The editor is not implemented yet.

Supplementary material and further readings#

  • Journal of Open Source Software (JOSS): A developer friendly, open access journal for research software packages.
  • Ross-Hellauer T, Deppe A, Schmidt B (2017) Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE 12(12): e0189311. https://doi.org/10.1371/journal.pone.0189311

About this course#

logo