Download The Task of the Referee - Lecture Notes | CMPS 290 and more Papers Computer Graphics in PDF only on Docsity!
T
here is an endless stream of research papers submitted to con- ferences, journals, newsletters, anthologies, annuals, trade journals, news- papers, and other periodicals. Many such publications use impartial, external experts to evaluate papers. This approach is often called peer review, and the reviewers are called referees. Refereeing is a public service, one of the professional obligations of a computer science and engineering professional. Unfortunately, referees typically learn to produce referee reports without any formal instruction; they learn by practice, by feedback from editors, by seeing referee reports for their own papers, and by reading referee reports written by others. This article tells you how to evaluate a paper, write a referee report, and apply common standards and procedures. It is intended to replace Forscherís rules, 1 which are distributed by some editors but do not reflect the procedures used in com- puter science and engineering. This article focuses on research papers in applied areas of computer science and engineer- ing, such as systems, architecture, hard- ware, communications, and performance evaluation, but most of the discussion is
generally applicable; separate sections consider research proposals and survey and tutorial papers. Authors might find this material useful for preparing papers for publication. Another recent paper dis- cusses refereeing in theoretical computer sciences; there are some differences between theory and the applied areas con- sidered here.
The refereeís task
Your role as referee is to decide whether a paper makes a sufficient con- tribution to the field. The contribution can be new and interesting research results, a new and insightful synthesis of existing results, a useful survey of or tutorial on a field, or a combination of those types. To quote a referee for this article:
Small results which are surprising and might spark new research should be published; papers which are mostly repetitions of other papers should not; papers which have good ideas badly expressed should not be pub- lished but the authors should be encouraged to rewrite them in a better, more comprehen- sible fashion.
The Task of the Referee
Alan Jay Smith
University of California at Berkeley
Computer
researchers have
a professional
obligation
to referee the work
of others. This
article tells you
how to evaluate a
paper and write a
report using
common
standards and
procedures.
Reading a paper as a referee is closer to what a teacher or professor does when grading a paper than what a scientist or engineer does when reading a published work. In the latter case, the reader pre- sumes that the paper has been checked (refereed) and is thus correct, novel, and worthwhile. As a referee, on the other hand, you must read the paper carefully and with an open mind, checking and evaluating the material with no presump- tion as to its quality or accuracy. The result of your reading should be a referee report that recommends for or against accepting the paper and lists necessary and suggested changes. It is important that you walk the uncer- tain line between being too permissive (ìpublish everythingî) and being too restrictive (ìnothing is good enough to publishî). If you are not critical enough, you encourage poor research, recognize and honor those who donít deserve it, mis- lead naive and inexperienced readers, mis- lead the author as to what is publishable, encourage disrespect for the field, distort commercial development, hiring, promo- tion, and tenure decisions, and perhaps actually subtract from the general store of knowledge; consider the Piltdown man fraud, which misled anthropologists for years. As has been noted by Thompson 3 and others, unrestrained publication buries the professional under mounds of paper, only a very small fraction of which can be examined, let alone read. If you are too critical, you block or delay good research from publication, waste the time of authors, damage careers, and perhaps leave journals with nothing to publish and conferences with nothing to present. It is particularly important not to reject new and significant work that runs counter to the prevailing wisdom or cur- rent fashion. If you want to be taken seriously as a referee, you must have a middle-of-the- road viewóyou must be able to distin- guish good from bad work, major from minor research, and positive from nega- tive contributions to the literature. A refer- ee who always says ìyesî or always says ìnoî is not helpful.
The referee report
A good referee report should have sev- eral parts. First, you should briefly state
your recommendation and the reasons for it. Second, you should summarize the point of the paper in one to five sentences, both for the editorís use and to ensure that you actually understand the paper. Third, you should evaluate the validity and sig- nificance of the research goal. Fourth, you
should evaluate the quality of the work (methodology, techniques, accuracy, and presentation). Finally, you must provide an overall recommendation for or against publication. If you recommend against publication, you should clearly state why. You should also make the strength of your opinions clear; an equivocal (ìmaybeî) recommendation is acceptable if your rea- sons for it are clearly documented. In any case, your report must contain enough dis- cussion and information to justify your recommendation. If your recommendation is favorable, you must list both necessary and suggest- ed changes. If your recommendation is negative, but you think the paper can be
salvaged and either submitted elsewhere or resubmitted, then you should provide a similar (but perhaps less detailed) list. Suggestions for alternate places to publish are always welcome. Refereeing a paper can require consid- erable time and effort; donít waste that effort on a detailed critique of a badly flawed paper that can never be made pub- lishable. Finding one or more fatal and uncorrectable flaws excuses the referee from checking all subsequent details. Typically, the author receives the text of the referee report stripped of all material identifying the referee. Thus, while it is important to be clear and explicit, your report should not be insulting. Donít refer to the author as a ìfoolî or an ìidiotî nor to the paper as ìtrash.î Your review should be directed at the paper, not the author. The review of a proposal, though, is also a review of the investigator. In this case, it is appropriate to evaluate the authorís abilities as well as the research proposed. In all cases, however, the evalu- ation should be objective and fair. The more psychologically acceptable the review, the more useful it will be.
Evaluating a research
paper
As a referee, you must evaluate a paperís novelty, significance, correctness, and readability. This general set of goals can be broken down into a much more specific series of questions.
What is the purpose of the paper? What is the problem? Is it clearly stated? Does the author make the important issues clear? Does the author tell you early in the paper what he or she has accomplished? For example, if the paper is a system description, has the system been implemented or is it just a design?
Is the paper appropriate? Does this paper have anything to do with computer science or engineering? If so, is the research appropriate for this forum? (Authors should not submit papers on queueing theory to Datamation or market analyses of the latest MVS release to the Journal of the ACM or the Proceedings of the IEEE .)
How to become
a referee
Editors are always looking for qualified and responsible referees. The easiest way to become a referee is to write a paper, thus bringing your name and expertise to the attention of the community. You can also become active in professional activities, such as local IEEE or ACM groups, IEEE Technical Committees, ACM Special Interest groups, and conference- organizing committees. Participating in these activities will help you meet editors and program chairs. Also, editors sometimes actively solicit referees.
(1) Major results; very significant (fewer than 1 percent of all papers). (2) Good, solid, interesting work; a defi- nite contribution (fewer than 10 per- cent). (3) Minor, but positive, contribution to knowledge (perhaps 10-30 percent). (4) Elegant and technically correct but useless. This category includes sophisticated analyses of flying pigs. (5) Neither elegant nor useful, but not actually wrong. (6) Wrong and misleading. (7) So badly written that technical eval- uation is impossible.
The next question is: What are the stan- dards of this journal or conference? Is this the Proceedings of the IEEE , ACM Transactions on Computer Systems , or the ACM Symposium on Operating Systems Principles (all quite selective), or is it the Tahiti Conference on Beach Ball and Computer Systems (fictional, but a pre- sumed boondoggle)? You should compare the paper with the average paper in that specific journal or conference, not with the best or worst. Of course, in some cases the average is too low and needs to be raised by critical refereeing. Note that you cannot tell how selective a conference or journal is by the percentage of papers it accepts; far fewer bad papers are submit- ted to the best conferences and journals. You should then make a recommenda- tion, whether favorable (ìpublishî) or unfavorable (ìrejectî). Your recommenda- tion is your opinion as to whether the paper makes a sufficient contribution. Generally, this will include all papers in Categories 1 and 2 above and some in Category 3. The strength of your recom- mendation should be clear to the editor (ìwonderful paper, definitely acceptî; ìuseful paper, probably acceptî; ìmarginal paper, see how many better ones have been submittedî; or ìwrong and mislead- ing, definitely rejectî). If you feel that the paper has something worthwhile to say, but youíre not sure it is good enough for this journal or conference, you can say ìmaybe.î You can also recommend that a paper be rejected as inappropriate for the journal or conference. If the paper is inap- propriate or marginal in quality for the forum but is suitable elsewhere, you can suggest other places to submit the paper. In any case, you must discuss and justify your recommendation. A recommendation without sufficient justification will carry very little weight.
If the author is asked to prepare a revised version, the revision will usually be sent to the same referees for further review. It is important to ensure that the revisions are satisfactory, but you should avoid comments inconsistent with your first review. You should also avoid harass- ing the author by unnecessarily recom- mending revision after revision. It is pos- sible, however, that a revised manuscript still contains serious problems due to things overlooked in the first review, prob- lems that have only become apparent after revision, or new errors introduced in the revision. Such problems must be addressed. The presence of serious prob- lems after a second revision suggests that the author is incapable of fixing the prob- lems, in which case it is often appropriate to recommend final rejection. For a conference, a paper that requires substantial revision generally cannot be accepted due to the short time available for revisions and the difficulty of arrang- ing for additional rounds of revisions. For journal publication, however, the extent of necessary revisions is a separate issue from the recommendation for (eventual) publication.
Surveys and tutorials
Surveys and tutorials differ from research papers in that most or all of the work reported is not new. Such a paper, however, might include a variety of minor research results that would not stand on their own in separate papers. Surveys and tutorials are similar but not identical. A pure tutorial explains some body of material to nonexperts, usually novices. It might not cover the entire field, and it might have a specific point of view. A survey provides broad and thorough coverage of some field or body of knowl- edge. It can be aimed at readers ranging from the novice to the near-expert. In reviewing a tutorial, there are specif- ic issues to address: Does the paper cover the material promised by the title or abstract? Is this a reasonable body of knowledge to cover in a tutorial? Is the scope too wide, too narrow, or too bizarre to be useful? Does the paper have a con- sistent theme? Is the material correct? Is the level of coverage too simple-minded
or sophisticated, given the likely audi- ence? Is the paper well written and clear? This last is crucial for tutorials, but jour- nals that publish tutorials, such as Computer and ACM Computing Surveys , often have editors and a professional staff to help with revisions. For a survey, many of the same ques- tions apply. Does the paper cover the material promised by the title or abstract, and is this a reasonable body of knowl- edge to survey at one time? Is the material correct, and is the author sufficiently expert on the subject to interpret results correctly and provide perspective on the field? Has the author integrated the mater- ial in a consistent manner, or is this just an annotated bibliography? Is the authorís coverage balanced and thorough? Does he or she cite all important relevant literature, or is the presentation biased, slanted, or unevenly selective? Controversial opin- ions and evaluations should be identified as such. If the survey includes new research results, do they meet the validity and correctness criteria given above for research papers? (A survey does not have to stand on its own as a research paper, so the research does not have to be so signifi- cant as to justify publication as a research paper.) Finally, is the paper well written and clear?
Proposals
A proposal is a request to a funding agency, company, or foundation for finan- cial support, supposedly to do the research described in the proposal. Reviewing pro- posals is quite different from reviewing papers, and some special considerations apply. Reviews of papers address only the science; reviews of proposals must also consider the investigator. The primary difficulty with reviewing a proposal is that the investigator is sup- posed to tell you what he or she plans to do, in addition to what has been done already. The questions you must ask, then, are: Is the research topic significant? Is the method of approach described, and is it reasonable? Do the investigators and assistants appear to have sufficient exper- tise to produce useful results? Is the bud- get reasonable given the proposed research, the qualifications of the investi-
gators, and the typical level of funding provided by the agency in question? Are the necessary facilities available? The easiest way to write a detailed and specific proposal is to propose research that is already complete or at least sub- stantially underway; this approach is quite common among established researchers. Unfortunately, that isnít the purpose of a research proposal, and requiring a high level of detail and specificity in the pro- posal discriminates against newcomers and those who propose new work. Also, a proposal might include a larger scope of work than can be reasonably accom- plished with the time and effort specified. This is not a negative factor if the investi- gators clearly recognize it and indicate that they will choose subtopics, depending on their interest and the availability of assistants to work on them. A major difference between a research proposal and a paper is that a proposal is speculative, so you must evaluate what is likely to result. Therefore, when you eval- uate a proposal by a well-known investi- gator, a substantial fraction of that evalua- tion should depend on the investigatorís reputation. People with a consistent histo- ry of good research will probably do good work, no matter how sloppy or brief their proposal. People with a consistent history of low-quality research will probably con- tinue in the same manner, no matter how exciting the proposal, how voluminous their research, or how hot the topic. However, you must also consider the pos- sibility that a well-regarded researcher may propose poor research or that a researcher noted for poor-quality work has decided to do better work. It is important that you do not discrimi- nate against newcomers who have no rep- utation, either good or bad. In this case, you must rely much more heavily on the text of the proposal and such information as the investigatorís PhD institution and dissertation, academic record, host institu- tion, and comments by his or her advisor or others. Reviewers are asked to comment on the proposed budget. Keep in mind that many factors affect the size of the budget other than the proposed scope of research, such as the agency providing the funding and the availability of facilities and staff. Also note that for a new investigator, there is a major difference between no funding and minimal funding (two months summer salary plus amounts for travel, supplies,
and computer time). Funding a new inves- tigator at a low level is often a good gam- ble; two or three years later the investiga- tor will have a track record and, if it is a good record, higher levels of funding can be justified. Such small grants are often called ìinitiation grantsî and should be much easier to get than regular grants.
Other issues
Simultaneous submission, prior pub- lication, and unrevised retries. If an author submits a paper simultaneously to two or more places, he or she must have the approval of all editors or program chairs. All referees should also be notified. Submitting a paper simultaneously with- out notification is unethical and a suffi- cient basis for rejection. There is a good chance that a simultaneous submission will be detected through the review process. If a paper has already been published (in conference proceedings, for example) and is submitted for republication (per- haps in an archival journal), the editor and referees must be notified. Some associa- tions such as the IEEE and ACM permit republication in their journals, but the paper generally must meet a higher stan- dard than if it had never been published. Significant extensions or major revisions are often a sufficient reason for republica- tion. As a reviewer, you should be alert to the author who tries to publish the same work in all its various combinations, per- mutations, and subsets, and to the author who adds the ìleast publishable unitî of new material to each paper. Also note that if the first version of the paper was pub- lished elsewhere, copyright restrictions might require the first publisherís explicit permission to republish the paper.
You will sometimes receive a paper to referee that you have previously recom- mended for rejection by some other publi- cation. If the paper has not been rewritten to comply with your previous review, it is appropriate to return a copy of that review along with a blunt note suggesting that the author comply with referee reports.
Acknowledgments and plagiarism. Authors must not plagiarize, and they must fully acknowledge joint work and the contributions of others. You should explicitly point out any such problems.
Timely response and returning a paper. You should return your report promptly. For a conference, referee reports must reach the program chair well before the program committee meeting so that the material can be assembled and pre- pared for discussion. Reports received after the program committee has met are useless. Journals do not generally have firm deadlines, but preventing consideration of a paper by taking a long time to review it is unethical. Computer science journals are notorious for long delays between sub- mission and publication; the two major bottlenecks are the referees and the publi- cation queue for the journal itself. Imagine if it were your paper. If you canít read the paper in a reasonable amount of time, typ- ically four to eight weeks, send it back to the editor or at least get the editorís agree- ment to the delay. Dante probably had a place for referees who promise to do reports and then donít do so.^6 Keep in mind that if you expect to have your own papers published, you have a responsibility to referee a reasonable num- ber of papers. It is part of your job as a researcher. The option of sending a paper back to the editor should not be abused. Editors can choose not to handle papers by authors who donít fulfill their referee- ing responsibilities. If you are sent a paper that you are not qualified to referee, you can send it back to the editor. However, if the editor has selected you to provide an ìoutsideî view of the field, you should provide a limited opinion (see the section on the editorís role). If you are going to send a paper back without refereeing it, do so immediately. Be sure to return the manuscript.
The primary difficulty with reviewing a propos- al is that the investigator is supposed to tell you what he or she plans to do.
Keep in mind that a good referee report is immensely valuable, even if it tears your paper apart. Remember, each report was prepared without charge by someone whose time you could not buy. All the errors found are things you can correct before publication. All the mistaken inter- pretations could have been made by the final readers. Appreciate referee reports, and make use of them. An author who feels insulted and ignores referee reports wastes an invaluable resource and the ref- ereesí time. Some authors suspect that a negative referee report indicates that the editor, program committee, program chair, and referees are incompetent, biased, or other- wise unfair. While this is sometimes the case, it is the exception. There is seldom a single correct evaluation of a paper, and equally skilled and unbiased readers will differ. However, a set of negative referee reports is an accurate indication that your paper should be rewritten or reworked before resubmission or discarded as unpublishable or embarrassing. A reader forms an opinion of you based on your paper; if your paper ís quality would reflect badly on you, you should not even submit it for publication. Authors should note that Day, 5 Levin and Redell, 10 Manola, 11 and Wegman 12 discuss how to write technical papers. Refereeing is also a good way to learn to write better papers; evaluating the work of others will give you insight into your own. Scientific progress relies heavily on the peer review processóthe evaluation of research for publication and funding by researchers qualified to evaluate the work. Referee reports are essential to that process. The refereeís task is necessarily a matter of opinion; as a referee gains expe- rience, the quality of the evaluation should improve. The guidelines and instructions presented here should be particularly use- ful in training and instructing novice refer- ees.
Acknowledgments
Iíd like to thank Peter Denning, Domenico Ferrari, Susan Graham, Anita Jones, Edward Lazowska, and Ken Sevcik for their comments on drafts of this article. The opinions expressed here are, however, my own. A number of the referees also made useful suggestions, many of
which have been incorporated. My research (regarding which I have received many referee reports) is supported in part by the National Science Foundation under grant MIP-8713274, by NASA under consortium agreement NCA2128, by the State of California under the Micro program and by IBM, Digital Equipment Corporation, Apple Computer, and Signetics/Philips Research Laboratories.
References
- B. Forscher, ìRules for Referees,î Science , Oct. 15, 1965, pp. 319-321.
- I. Parberry, ìA Guide for New Referees in Theoretical Computer Science,î SIGACT News , Vol. 20, No. 4, Apr. 1989, pp. 92-109.
- K.S. Thompson, ìMarginalia-The Literature of Science,î American Scientist , Vol. 72, Mar.-Apr. 1984, pp. 185-187.
- L Carroll, Alice Through the Looking Glass ,
- R. Day, ìHow to Write a Scientific Paper,î IEEE Trans. Professional Communication , Vol. PC-20, June 1977, pp. 32-37.
- D. Alighieri, The Divine Comedy, Cantica 1: Lílnferno , 1314, translated by D. Sayers, Penguin Books, Baltimore, Md., 1949.
- C.T. Bishop, How to Edit a Scientific Journal , ISI Press, Philadelphia, Pa., 1984.
- ìInformation for Authors,î Comm. ACM , Vol. 32, No. 3, Mar. 1989, pp. 411-414.
- ìGuidelines for Authors,î IEEE Software , Vol. 1, No. 1, Jan. 1984, pp. 7-8.
- R. Levin and D. Redell, ìAn Evaluation of the Ninth SOSP Submissions,î ACM Operating Systems Review , Vol. 17, No. 3, July 1983, pp. 35-40.
- F. Manola, ìHow to Get Even with Database Conference Program Committees,î IEEE TC Database Engineering newsletter , Vol. 4, No. 1, Sept. 1981, pp. 30-36.
- M.N. Wegman, ìWhat Itís Like to Be a POPL Referee, or How to Write an Extended Abstract so that It Is More Likely to Be Accepted,î SIGAct News , Vol. 17, No. 4. Spring 1986, pp. 50-51.
Alan Jay Smith is a professor in the Computer Science Division of the Department of Electrical Engineering and Computer
Sciences at the University of California at Berkeley, where he has been on the faculty since 1974 and was vice chair of the depart- ment from July 1982 to June 1984. His research interests include the performance analysis of computer systems and devices, computer architecture, and operating systems. He received the IEEE Best Paper Award for the best paper in the IEEE Transactions on Computers in 1979. Smith is an associate editor of the ACM Transactions on Computer Systems , is a sub- ject-area editor of the Journal of Parallel and Distributed Computing , and is on the editorial board of the Journal of Microprocessors and Microsystems. He was program chair for the Sigmetrics 89/Performance 89 Conference and is program cochair for the 1990 Hot Chips Symposium. Smith received the BS degree in electrical engineering from the Massachusetts Institute of Technology, and the MS and PhD degrees in computer science from Stanford University. He is an IEEE fellow and a member of the ACM, SIAM, the Computer Measurement Group, Eta Kappa Nu, Tau Beta Pi, and Sigma Xi.
∗Readers may write to Smith at the Computer Science Division, EECS Department, University of California, Berkeley, CA 94720.
