The following are the concepts that you should keep in mind while writing your review.
Getting the best of the paper for serving the community.
In a full Open Access publication, as in the Insight Journal, the purpose of the review is not to judge the quality of the paper with the attitude of accepting it or rejecting it. Instead, the position of the reviewer should be to help the authors clarify points that are not already clear, with the end goal of serving the community by helping readers get maximal benefit from the paper's content. Reviews are intended to be honest requests for additional information and/or attempts to provide the authors with background information that would make the publication more useful to the community.
Claims should be supported by evidence.
As with any technical publication, claims must be supported. This rule applies both to the paper and to the reviewer comments.
For example, if a paper claims that method A is faster than method B, this claim should be followed by the full description of the experiment which the authors used to arrived at this conclusion. The experiment should be described in enough detail to allow the reader to repeat the experience and validate the claim.
The same goes for reviewers comments. Only claims that are supported by experimental evidence should be mentioned in the reviews.
Opinions of the reviwers that are not supported by evidence should be labeled as "opinion" or "speculation" in the review. The nature of research is such that both authors and reviewers may sometimes need to speculate on some issues (such as "is validation really the most important problem in medical computation?"), and it is helpful, especially to the introductory reader to be able to clearly differentiate reviewer speculation vs. evidence based facts.
The Insight Journal provides the infrastructure for having an open dialog on line where authors can reply and make clarifications to the reviewers. You should rather post your comments, even your questions, as part of your review. Please do not contact the authors directly about the papers.
Note that after posting a review you can still add more content to it, so you have a chance for expanding on your comments if your interpretation of the paper changes over time.
Contribution to the community
Papers AND reviews should make a contribution to the community. Papers are encouraged to include Open Source code, or Open Access data, or descriptions on how to better use them. Reviews are encouraged to consider how effectively the authors meet those goals and similarly to also include code, data, and parameter settings that were used when doing the evaluation.
Criteria for evaluation
We recommend the following criteria and weightings when formulating a review and an overall score.
- (2 points) Does the paper follow the standards of open-science by including the relevant code, data, and parameters needed to replicate the work described in the paper? Can the work be duplicated? How well does the work (when appropriate) build upon (instead of duplicating) existing open-source efforts?
- (1 point) Does the work address a (knowledge or software-based) need within the community? Where do you see it having the most impact?
- (1 point) How well is the work described in the paper? Is sufficient background material provided in the paper itself, or easily located using pointers in the paper?
- (1 point) Can you name one or two research projects that you think would benefit from the use of this work?
Your final score (1-5 stars) for the paper should reflect the point sum for the paper, e.g., 0-1 = 1 star ... 4-5 = 5 stars.
Please note that unlike for IEEE TMI, MedIA, and other such journals, reproducibility of results is the central review criterion for this journal, rather than its algorithmic originality. In particular, a submission describing an open-source implementation of an existing method is highly appropriate for this journal, if an open-source implementation of that method does not already exist.
- Please be sure to answer the above questions in your review. If a particular question does not apply, please note that explicitly.
- In addition to the answers to the above questions, it might be useful to the reader if a coarse-to-fine review was provided that starts with (a) the overall recommendation of the reviewer to the community (e.g., "Download this!" or "Has great potential, but needs some additional work" or "Work is interesting, but not (yet?) appropriate for the open-source community for the following general reasons..."), (b) General comments on the technology involved, (c) General comments on the contribution (introduction, background, method description, evaluations, followed by (d) Detailed comments on the contribution (grammar, spelling, data formats).