1. What is the function of a review?

View section

A review plays two important roles in the publication process:

Quality control
The review should help the editor to make decisions aimed towards preserving the level of quality of the targeted journal (or conference). The review achieves this by providing information about the relevance, originality and validity of the manuscript’s contents.
Constructive criticism
The review should help the authors to improve their work (upon revision of their paper, for instance) by providing constructive comments and clearly formulated suggestions.

The expert who agrees to review a manuscript and does a good job of providing these two things is helpful both to the journal and the authors of the manuscript, but also to his scientific community at large (since it will have access to better articles, thereby also promoting better science to improve on existing work).

Caution

A review is not the place to:

  • Display the great knowledge, cleverness or personality of the reviewer
  • Fuel a debate regarding whether any given field of study is interesting or not
  • Take revenge for any personal or professional prejudice, real or imagined, whether or not it was caused by one of the authors
  • Etc.

2. Committing to review an article

View section

It can be flattering to receive a request for a review from a prestigious conference or journal in your field of research, especially the first few times. Sometimes, the first requests of this kind come from a thesis advisor, who delegates her own reviewing tasks. In both cases, this demonstrates recognition of the expertise you have accumulated over the years, which is not unpleasant.

In academia, writing reviews is part of most researchers’ lives. It is a privilege, in the sense that the reviewer has the opportunity to learn about new ideas in his field before they are even published, but it is first and foremost a responsibility. Generally, it is also unpaid work. Researchers who agree to review manuscripts generally do so because they consider that they contribute to a collective effort that is crucial to preserve the general quality level of the articles they read and use in their research. It is also a way to give back to the community what they receive every time they submit a paper to a journal or a conference.

Agreeing to review a manuscript thus represents a certain commitment towards the scientific community in general, and towards the authors in particular. For this reason, when committing to review a manuscript, it is important to:

Make sure you are sufficiently competent
The contents of the manuscript should be close to your own research expertise, either because it addresses similar topics or because it uses methods closely related to those you understand well. You do not need to be an expert on all topics covered by the paper, but you should feel relatively comfortable when reading the abstract and already have a rough idea of what the rest of the paper might present.
Make sure you have enough time and meet the deadline
The editorial board generally asks for the referee report within a few weeks of the request for a review. Out of respect for the authors who want to publish, it is important that you meet the specified deadline. Reviewing a manuscript and writing a good review can take several hours. We have all received a few hastily written reviews of only a few lines. This type of review helps no one.
Respect confidentiality rules
The paper that you are to review has not yet been published and constitutes a confidential document. You have access to it only for reviewing purposes. Therefore, it is important that you do not share the contents of the manuscript with anyone and that you do not exploit the new information that you may find in it for your own purposes before it is published.
Remain anonymous
Generally, the identity of reviewers is also confidential data and is hidden from the authors. Remaining anonymous is important so as to avoid introducing additional bias in your relationship to the authors (whom you may actually know).
Avoid conflicts of interests
Avoid evaluating manuscripts by authors with whom you have recently collaborated, or had an employer-employee or close personal relationship. This would undermine your credibility. It is better to decline the request, stating the conflict of interests.

If, for whatever reason, you are not able to commit to writing a review, it is good practice to decline politely and promptly and, if possible, to suggest a colleague (to whom you have preferably spoken to about this beforehand) to replace you.

3. How to evaluate a manuscript

View section

Evaluating a research paper requires time and thought. The first step is, of course, to read the paper. While you read the paper for the first time, it is useful to make a few notes regarding:

  • The key ideas of the paper
  • Some points that are obscure or seem suspicious to you
  • The key bibliographical references mentioned in the paper, especially if you are not familiar with them

It is likely that the next step in the process will be to read the paper again, carefully. This time, keep making notes but be more critical of what you read and try to learn more about the points that are obscure to you; there might be gaps to fill in your knowledge.

Caution Do not get impatient if the process seems lengthy and labor intensive the first few times you review an article. It will get easier and a bit shorter with experience.

Your repeated readings of the article should allow you to:

Determine whether the manuscript is relevant to the targeted venue
If the paper tackles an important or new problem in the field and if it is aimed towards the intended audience of the venue, then it is probably relevant. If the venue seems to be a poor choice, try to think of a better alternative.
Identify the main contributions of the paper and evaluate their originality
Ideally, the authors should have clearly identified the contributions of their work in the manuscript. Otherwise, use your knowledge of the field and the contents of the manuscript to express it more clearly. Establishing the degree of originality of a scientific contribution requires a comparison to existing work. This means you may have to read other papers (at least in part) to make your evaluation. These papers may or may not be cited in the manuscript. If you are not completely aware of the literature on the topic, briefly search the literature for yourself. Do not forget to look up prior publications by the authors of the manuscript to understand the true extent of the progress made since. It is quite common to publish an extended version of a conference paper in a journal, but this practice should be properly recognized in the journal article and the extension should in principle add some value to the original.
Evaluate the completeness of the literature review
Having evaluated the originality of the manuscript, you will have quickly identified the important articles on the topic. Are these articles discussed in the literature review? Are the contents of this discussion roughly accurate?
Evaluate the methodological contents
In engineering, the main contributions of a research paper often consist of new methods to solve a given problem. The new methods must be well defined and well justified. The following questions might be helpful in evaluating the quality of the proposed approach:

  • What assumptions (explicit or hidden) does the proposed approach rely on and can they be justified?
  • Are there practical circumstances where you anticipate that the proposed methods would fail?
  • Are the mathematical developments correct and are all variables properly defined?
  • Are the results plausible, given what is proposed in the manuscript?
  • Is there enough information to reproduce the results?
  • How were the various adjustable parameter values chosen and is the process well justified? Are these choices likely to generalize to slightly different contexts?
Evaluate the experimental methodology

In the scientific method, experiments are meant to validate a research hypothesis. In the case of an engineering research paper, this hypothesis might be that the approach proposed in the paper to solve a given problem works well, or that it offers an important advantage over the state-of-the-art. To evaluate the experimental methodology, ask yourself the following questions:

  • Do the experiments really challenge/demonstrate the contribution claimed by the authors?
  • Are the results quantitative or qualitative? Are they convincing?
  • Do the experimental conditions reflect the assumptions explicitly made by the authors? Are they too simplistic?
  • Should there have been more experiments?
  • Where independent measurements made sufficiently accurately and precisely when needed?
  • Are the experimental conditions reproducible?
Evaluate the analysis of the results

It is not enough to present results in a table or chart to draw scientifically valid conclusions. The results must be analyzed and interpreted. A better interpretation often leads to richer and more insightful conclusions. When evaluating this aspect of the manuscript, pay attention to the following questions:

  • Do the results come with sufficiently detailed analyses and interpretations?
  • If some of the results are in appearance surprising or very different from the others, are they explained?
  • Is there sufficient detail regarding the statistical analysis of the results (for instance, the names and parameters of the statistical tests that were used) and if so, was the analysis carried out appropriately?
  • Do the authors compare their results to those of other researchers (preferably to the state-of-the-art)?
  • If so, is the comparison fair and based on valid or commonly used criteria?
Evaluate the quality of the writing

The quality of the writing should not be the main determining factor in accepting or rejecting the manuscript unless it is really incomprehensible. However, since revisions are almost always required (whether or not the manuscript is likely to be accepted by this journal), it is useful to identify potential writing problems and to mention them in the review:

  • Is the manuscript relatively easy to read (for a research paper), given the usual audience of the journal?
  • Are there any important ideas that are not expressed sufficiently clearly?
  • Are ideas presented in the right order?
  • Does the abstract adequately reflect the paper’s contents?
  • Are there major problems with the English?
  • Are there superfluous sections?
  • Are the drawings and charts readable and do they really help the reader understand the paper?
  • Are there figures missing that might help the reader understand the paper?
Tip Think about each of these evaluation criteria and keep making notes. You may want to take a break after your first read and/or before writing your review. Taking a step back sometimes helps detect subtleties (often important ones) that would otherwise escape one’s attention.

4. How to write a review

View section

Nowadays, most conferences and journals use on-line forms to manage reviews. The degree of structure in the required information varies substantially between venues. Most of the time, the form includes parts that are visible by the authors of the manuscript (and the editor) and a part that will be seen by the editor only. Most of your review will be visible by the authors.

4.1. Factual summary

A complete review begins with a factual summary of the manuscript. This summary should briefly and rather neutrally describe the contents of the paper: the context and specific problems addressed, the main contributions (according to the authors), the methods used, the results obtained and the conclusions drawn. The role of the factual summary is to:

  • Provide an alternative point of view to the editor, who may only have had the time to read the authors’ abstract
  • Show the authors that you understood their paper and its contributions correctly (if this is not the case, the authors might then need to make clarifications to the manuscript)

4.2. Critical summary

After the factual summary, it is important to also provide a critical summary of the manuscript that will help guide the editor’s decision. This is where to first emphasize the strengths of the manuscript (there are almost always some), for instance:

  • The paper tackles a particularly difficult, novel and/or important problem
  • The paper proposes particularly novel methods
  • The paper presents highly interesting or surprising results, or results that are full of practical implications
  • The paper describes rigorous experimentation and/or analysis of the results
  • The paper is very clearly written
  • The paper presents a very thorough and useful review of the literature
  • Etc.

The critical summery then identifies the weaknesses of the manuscript (there are almost always some) in a global fashion. For instance:

  • The paper tackles a problem that is no longer relevant
  • The paper proposes methods that represent only a small increment over previous work
  • The results presented in the manuscript are not very convincing
  • There are errors or gaps in the proposed methods, the experimental methodology or the analysis of the results
  • The manuscript is difficult to read or poorly structure
  • The paper does not refer sufficiently to the state-of-the-art
  • Etc.

4.3. Detailed list of comments

Following the critical summary, a good review typically provides a detailed and numbered list of the most problematic points. This list is particularly important if the manuscript contains enough good ideas to warrant eventual publication but requires substantial revision. The list should clearly identify and detail each problematic point, that is:

  • Refer to the page, section or figure number where the problem occurs whenever possible
  • Identify the problem and explain why it is a problem
  • Ask specific questions if the manuscript was not sufficiently clear
  • Suggest corrective measures when possible
Tip When addressing criticisms to the authors of the manuscript, it is important to remain constructive, polite and as anonymous as possible. A reviewer should feel almost comfortable enough to defend her opinion in person.

If the manuscript seems almost ready for publication, one may also include a list of minor problems, including typographical errors (there is no need to be overzealous) or problems with the colors of the charts, for example.

4.4. Final recommendation

The part of the form that is hidden from the authors is the place to provide your final recommendation to the editor and to justify it briefly (in general, the rest of the review already achieves this). This is also where you can indicate your level of confidence in your own review. For instance, it can be useful to indicate that one of the topics touched upon in the manuscript is further away from your expertise, or that you did not scrupulously check every equation or mathematical proof provided in the appendix. When needed, this section can also be used to remind the editor of possible conflicts of interests or to signal a case of plagiarism.

Caution In case of plagiarism, thorough verifications are required: it is a serious accusation that can undermine the authors’ reputations. Thus, one must make sure that the accusation is founded (i.e. it is really plagiarism) and provide all necessary proofs to the editor, clearly identifying the documents that were plagiarized (with complete bibliographical references) and, more specifically, the parts that were copied and the places where they appear in the manuscript to be reviewed. The reviewer’s role does not extend beyond that. The editorial board is in charge of applying the journal or conference’s policy with respect to plagiarism.