Review process
CompEd papers are reviewed using a dual-anonymous process managed through EasyChair. The reviewing process has three phases: bidding, reviewing, and discussion and metareviewing.
Reviewers are expected to provide high-quality reviews to provide authors with feedback that will help to improve the work for publication or for future submission. Associate program chairs (APCs) lead the discussion among reviewers and metareview each paper, providing a recommendation and feedback to the program chairs.
Each paper submission is expected to receive three or four reviews and a metareview, but this number is subject to change depending on the number of submissions and the number of reviewers and APCs.
All reviews are submitted through EasyChair. In EasyChair, reviewers are called ‘PC members’ and APCs are called ‘senior PC members’.
All paper submissions are anonymous. Reviewers and APCs are and should remain anonymous to one another and to the authors of the submissions they review. When writing comments on a review during a discussion, please refer to yourself and other reviewers by the number of their review (R1, R2, etc).
Reviewer and APC timeline
The following dates describe the timeline for reviewer & APC work on CompEd 2023. Note that the review period is just two and a half weeks.
All reviewers and APCs are expected to engage with the process throughout the review period. Please consider your workload and other commitments around these dates before offering your services as a reviewer or APC. Those who cannot commit to engagement throughout this period should not volunteer to be reviewers or APCs for CompEd 2023.
| Activity | Start date | End date (Anywhere on Earth, UTC-12) |
| Bidding (1 week) | Mon 8 May | Sun 14 May |
| Reviewing (2.5 weeks) | Thurs 18 May | Sun 4 June |
| Discussion and metareviewing (10 days) | Mon 5 June | Wed 14 June |
Expected workload
- Reviewers: 4-6 full papers (each up to 6 pages + references)
- APCs: 8-12 full papers (each up to 6 pages + references)
These estimates are based on the workloads over the past few years. Of course, they are subject to the number of submissions, so actual workloads for CompEd 2023 might be a little higher or lower.
The program chairs will generally not accommodate requests for lighter review loads. If you are not able to commit to the full load, please do not volunteer your services.
Bidding
Reviewers and APCs will bid for papers that they are interested in reviewing or metareviewing during the week between abstract submission and full paper submission. If reviewers take the bidding process seriously, this reduces the chance of weak reviews arising from a lack of familiarity with the content of a submission. As part of the bidding process, reviewers will be required to note papers with which they have a conflict of interest (papers that they recognise are from colleagues or project partners). Separately from the bidding, reviewers will be shown a list of authors and asked to indicate which of those they have a conflict of interest with.
It is important to understand that bidding does not mean choosing the papers you will review. That choice will be made by EasyChair, using an algorithm that tries to allocate the papers reasonably fairly. Ideally, bidding is a process of identifying all of the papers that you would like to review or are willing to review. The more papers you bid for, the more likely you are to be allocated papers from your list.
Bidding on just a small number of papers influences the algorithm unfairly in your favour. To counter this, anyone who bids on too few papers will have their bids supplemented with an arbitrary selection of papers. If you do not wish this to happen to you, please ensure that you bid ‘yes’ for at least 12 papers and ‘yes’ or ‘maybe’ for at least 12 more.
Once you are notified that bidding is open, you should log in to your EasyChair account and access the CompEd 2023 conference with the appropriate role (‘PC member‘ or ‘senior PC member‘).
You are likely to find yourself on a page listing the Submissions to the conference, but this is not the page you want. Instead, click on the Paper Bidding menu. Again you will see a list of the submissions, but for each one, you can now click links to indicate the level of your interest in reviewing them. To see the abstracts as well as the titles, click either Show full abstracts or Show abstract summaries at the upper right of the screen.
There is no need to click No for papers you do not wish to review; No is the default option. You should click Conflict for a paper if you believe you know who wrote it, and thus cannot be sure of reviewing it impartially.
The papers are presented in a random order for each bidder, to help ensure a good spread of bids. However, this order is preserved for each bidder, so if you spread your bidding over several sessions, the papers will appear in the same order each time you access them.
The bids are colour-coded, but the selected yes/maybe/no/conflict link is also indicated in bold face, which will help those who have trouble distinguishing the colours.
Reviewing
Reviewer responsibilities
A few days after the close of bidding, reviewers and APCs will be notified which papers they have been assigned, and paper reviewing will begin.
Reviewers are urged to download their assigned papers immediately and take a first look at them. If, for example, you discover that you cannot review a paper because you have discovered a conflict of interest, we would far rather be informed in the first few days of reviewing than on the last day.
When you log in to the appropriate role on EasyChair you will see a Reviews menu. On this page, you can download the papers assigned to you. You should also click Add Review for one of the papers, to get a first look at the review form, but don’t submit the review until you’ve written it!
Please aim to complete your reviews before the deadline; it is a deadline, not a target submission time. When reviewers leave the task till the last minute, things can go wrong (illness, work emergency, family emergency, reviewing taking longer than you anticipated), and then the reviews are not done by the deadline, which means that we are left asking for volunteers to review extra papers at very short notice.
There will be periodic reminders during the review period. If at any time during this period you become aware that you will not have your reviews written by the deadline, please contact us immediately so that we can decide how to deal with it.
Once you have submitted your review for a paper, you will be able to see the other reviews already submitted for that paper, and EasyChair will notify you by email when further reviews are submitted. Please consider these other reviews, in preparation for the discussion period.
Here are some points that you should consider when reviewing.
1. Is the paper within the scope of CompEd? Is it about the education of students who are studying computing? Is it pertinent to the category for which it has been submitted?
2. Your review should be written by you; please do not farm your reviews out to other people. If you believe that other people should be reviewing for CompEd, please encourage them to do so in their own right. If you do not have the time or the inclination to do your own reviewing for CompEd, please do not volunteer to do so.
3. Your review should indicate whether the paper as submitted is acceptable for CompEd. There is no review-revise-review cycle, so papers should be marked as accept if they are currently acceptable, not if they have the potential to become acceptable after substantial revision. In particular, CompEd does not have a conditional acceptance. If a paper cannot be made acceptable with grammatical corrections and/or minor revisions of content, you should recommend that it be rejected.
4. Whether or not a paper is accepted, the authors will want to know why. Please ensure that your review gives reasons, explanations, and examples of the points you make. Reviews just a few lines long are generally not helpful to the authors, the APCs, or the program chairs. Authors should be able to see the text parts of the reviews as a constructive critique of their submission.
5. Be sure that your overall recommendation tallies with your impression of the paper. If you do not believe that the paper should be accepted, please do not give it a 4 just to be nice, or to err on the side of generosity. It is more helpful to the program chairs, the APCs, and the authors if your review is honest and transparent and your overall recommendation reflects your impression.
6. CompEd reviewing is dual-anonymous. You should not know who wrote the papers that you review, and the authors should not know who reviewed it. Please, therefore, take care not to divulge your identity in your review, for example by saying ‘This paper that I wrote might be helpful to you’ or by recommending the addition of references to a number of papers all of which you have (co-)authored.
7. Please appreciate that authors sometimes have to omit or disguise references to their own work in order for their paper to be anonymous. As a reviewer, you may give the authors the benefit of the doubt. Use the ‘Confidential comments to the committee’ box to indicate this to the program chairs; for example ‘This paper should reference Kim Lee’s work, unless those references were removed for anonymity’.
8. If you find that a paper you are reviewing is not completely anonymous, you may point this out to the authors. However, please do not divulge the authors’ identity to the other reviewers or the APC, and please continue to review the paper as though it were anonymous. If you feel that you cannot review it objectively, please notify the program chairs immediately.
9. Most papers have some flaws in their writing. The reviewers are not required to be editors, pointing out every flaw that requires correction. However, some reviewers choose to do this, and many authors appear to appreciate the additional service. The review form has a specific field for such suggestions, keeping them distinct from the strengths and weaknesses and the overall evaluation of the paper. It is also worth remembering while offering suggestions that English is not a first language for all our authors. We receive submissions from all over the world from authors belonging to different social, cultural, and linguistic groups.
10. It is a good idea for reviews to be grammatically correct, especially if they criticise the grammar of the paper. Just as a poorly written paper gives the reader a poor first impression, a poorly written review can make the authors less inclined to accept its content. Please read your own reviews carefully, and, if necessary, submit revised versions with the spelling and grammar errors corrected. If you choose to draw attention to spelling issues in a paper, please be aware that many words have different spellings in different parts of the English-speaking world. Neither ‘analyze’ nor ‘analyse’ is wrong: they are different acceptable spellings; likewise ‘program’ and ‘programme’, ‘behavior’ and ‘behaviour’, and many more such pairs.
11. When reviewing a paper, please be aware of the category in which it was submitted. There are three categories: computing education research; experience reports and tools; and curriculum and position. Different guidelines apply to each category, and it would be unreasonable, for example, to review an experience report as though it were a research paper. See Paper category guidelines below for the guidelines for each category.
12. Good papers will be relevant to an international audience. CompEd certainly accepts papers based on work from specific countries, but such papers are more valuable if they have clear insights for people from other countries. For example, a paper that writes about making it easier for people from the mountain regions to succeed in a computing degree would be better received if it made a clear generalisation to other underrepresented minorities, or perhaps to people with better lung capacity, regardless of country.
13. The limit of six pages (with references possibly extending onto a seventh page) is a maximum, not a minimum. There is nothing wrong with shorter papers if they are good and pertinent; they do not need to be padded out to six pages. At the same time, if a shorter paper clearly lacks something that would have strengthened it, feel free to point that out to the authors.
14. Authors are not permitted to provide links to supplementary materials in a review submission. Author guidelines make it clear that the paper should still stand on its own merits. If authors overlook this guideline and provide links to supplementary material, reviewers are not expected to access those links and examine the material.
15. It is acceptable to point out formatting issues, but not to make them the deciding factor in a recommendation to reject. Some authors are still struggling to come to grips with the ACM templates. One version of the template even results in a single-column pdf. If in doubt, leave it to the proceedings chair to decide on a paper’s compatibility with the template.
16. Please be aware that different versions of the ACM template have different placeholders for the title and the authors’ names. There is nothing insidious about a paper that says it is being submitted to WOODSTOCK’18, June, 2018, El Paso, Texas USA, or that its authors are F. Surname et al. or G. Gubbiotti et al.; these are all defaults in different versions of the template. As with other aspects of the formatting, assume that it will be fixed if and when the paper is accepted for publication and revised.
APC responsibilities
Each submission will be assigned to one APC. As an APC, please carefully read each submission at the start of the review period so that you will be able to relate the reviews to it as they are submitted. You are expected to watch the reviews for your assigned papers as they come in; to do this you will need to explicitly add them to your watchlist so that you will receive email notifications. From the CompEd 2023 menu choose My watchlist, choose the Select all option, then Save watchlist. As the reviews arrive, please keep an eye open for obvious problems: a review might have been submitted for the wrong paper, or might identify the author or the reviewer, or might be so brief as to be unhelpful. We will do our best to notice such problems, but your attention will also help.
Unlike some other conferences, CompEd keeps track of review completion, sends a number of review reminders, and chases unresponsive reviewers. Reviewers can become confused if they receive multiple reminders from the same EasyChair email address, for example, they might respond to a reminder from the chairs, and then a few hours later receive another reminder from an APC. Therefore we explicitly ask APCs not to send reminders to reviewers, and indeed not to send any emails to reviewers, but to limit their communications to the comment feature of EasyChair and their eventual metareviews.
Emergency reviewing (optional)
Despite the best of intentions, some reviewers find that they are not able to review the papers assigned to them and others simply drop out of contact for various reasons. The program chairs often discover this only at the end of the reviewing period. They then need to find reviewers who have completed their assigned reviews and are able to review one or two more papers within a day or two.
The call for emergency reviewers generally goes to all reviewers, but there is no expectation that all reviewers will respond. We understand that most reviewers now have other calls on their time, and we are appreciative and grateful that a few are able to step in and take on this additional load.
Discussion and metareviewing
The discussion period facilitates communication among the reviewers and APCs to ensure that the APC makes the best recommendation to the program chairs and the submission is given full consideration in the review process.
Reviewer responsibilities during discussion and metareview
As a reviewer, you are expected to engage with the discussion on each of your papers during the discussion period. Read the reviews from the other reviewers and take part in the discussion using the Comments feature in EasyChair (beneath all the reviews). Your comments will not be sent to the paper’s authors, but the final version of your review will. If, after reading the other reviews and taking part in the discussion, you are inclined to revise your review, please do so. There is no obligation to revise your reviews, but your review should reflect your final impression of the paper, and compatible reviews from all reviewers make matters easier for the APCs, the program chairs, and the authors of the submission.
If you do revise your review, please do not refer explicitly to other reviews. It is not appropriate to write, for example, ‘I agree with Reviewer 3 about the paper’s shortcomings’. It should be possible to read each review independently of the others. If you do wish to explain why you have revised your review, please say it in the Confidential comments part, where it will be seen by the other reviewers, the APC, and the program chairs, but not by the paper’s authors.
APC responsibilities during discussion and metareview
As an APC, you will lead and encourage discussion among the reviewers, especially for submissions with divergent reviews. This discussion will help you to form an opinion on whether the paper should be accepted.
Ideally, the reviewers and the APC will come close to consensus on the quality of the submission. Different reviewers might see different strengths and weaknesses in the paper, or might weight those strengths and weaknesses differently. However, it would be good if all reviewers, after reading the other reviews and taking part in the discussion, have fairly similar opinions about the paper overall, and are willing to revise their reviews if necessary to reflect those opinions.
On the other hand, reviewers should never feel coerced to change their reviews, scores, or viewpoints in this process.
The APC will write a metareview, summarising the reviews, adding their own opinion if appropriate, and providing a recommendation for the program chairs. If the reviews are divergent, it is important for the metareview to summarise both positions. This metareview will be sent to the authors along with the regular reviews.
The text of the metareview should not explicitly include the APC’s accept/reject recommendation; there is a separate field for that. As an APC, you will only see a few of the submitted papers; please understand that your recommendation will be carefully considered, but that the program chairs might choose the opposite outcome when considering the full set of submissions and working to put together a broad yet coherent program. At the same time, please remember to select a recommendation. Recommendations to accept or reject are far more helpful to the program chairs than ‘undecided’, which should be reserved for the really difficult ones.
Please do not wait until the very end of the discussion period to write your metareviews.
APCs who are unsure what to put into a metareview might consider starting with a simple four-paragraph template: what the paper is about; what’s good about it; what’s not good about it; your overall opinion. An alternative approach would be to briefly note the key points made each reviewer, then to add your own opinions. Remember that you are a senior reviewer: your role is not to repeat what the reviewers have already said, but to coalesce it into a single overall impression, which can certainly include your own thoughts on the paper.
APCs should note that confidential comments in the metareview are visible to all of the paper’s reviewers as well as the program chairs. If you wish to communicate just with the program chairs, please do so by email.
Additionally, the program chairs will ask APCs for feedback on the quality of reviews, to help inform decisions about future invitations to review for CompEd.
Paper category guidelines
There are three categories/tracks for papers. Reviewers and APCs will generally be assigned papers across all three categories.
Authors are asked to choose the track that they feel best fits their submission. Review the submission using the guidelines for the track the submission is in; not the track you would prefer it to be in.
Please note that we are considering the third category as a catch-all for any papers that don’t fit well into the other two. It is not our intention that papers be rejected solely because we can’t find the right category for them.
Computing education research category
Papers submitted to the computing education research category typically describe an empirical computing education project.
Computing education research papers should adhere to rigorous standards, describing research questions or hypotheses, methods, and results as is typical for research studies. These will normally focus on topics relevant to computing education with emphasis on educational goals and knowledge units or topics relevant to computing education with statistical rigour; methods or techniques in computing education; evaluation of pedagogical approaches; and studies of the many different populations that are engaged in computing education, including (but not limited to) students, instructors, and issues of gender, diversity, and underrepresentation. Papers need not present novel research and positive results to be accepted at CompEd: we also welcome replication papers and papers that present null or negative results, so long as they meet the criteria below.
For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):
- Are there one or more clearly stated research questions? Since the rest of the paper will be organised around these, it’s often good to put them in the abstract and early in the body of the paper.
- Are the questions of interest to the CompEd audience? Is the focus on the education of computing students, as opposed to education in general?
- Related work in computing education
- Is the relevant work in computing education included? If not, a good review will give references to missing material. It is not enough to write ‘The related work section is incomplete’.
- Do the authors clearly describe the relationship between prior work and the current research questions? In what ways does the current project build on prior work, and how is it different?
- Related work in educational theory
- Is the project based on educational theory? If not, should it be and what are some theories the authors should consider?
- Is the theory described clearly, with appropriate citations?
- Is the theory’s relationship to the current project clearly described?
- Is the data gathering described clearly enough that the reader could reproduce it? Some key information to include:
- About the data: why this particular type of data is relevant to your research questions.
- About the participants: how many, what was their background (are they instructors, students, graduates, etc.); what if any formal coursework have they had in computing; the gender distribution; and any other factors that are relevant to the author’s project.
- About the people gathering the data: what is their relationship to the participants? For example, if the data were collected from students in a class, was the instructor one of the researchers?
- About the data-gathering process: did the project use surveys, interviews, samples of student work, something different? If surveys or interviews, exactly what questions were asked?
- About ethical approval: was appropriate ethical approval sought and granted for the data gathering?
- Is the data analysis process/methodology sufficiently described that the reader could reproduce it?
- What methodology was used?
- Is the method described, with an appropriate citation?
- Is the implementation of the method described clearly? How many people were involved? What process was used to resolve any disagreements?
- Is the analysis process/method appropriate for answering the research questions?
- Is the analysis method something new to computing education research that might itself be a contribution?
- Are the results of the analysis clearly summarised?
- Are the results thoroughly discussed, including:
- Their relationship to the research questions
- Their relationship to prior work
- The implications of the results for teaching
- The implications of the results for future research
- Is there a discussion of threats to validity?
Experience reports and tools category
This category accepts classroom experience reports, descriptions of teaching techniques, and presentations of pedagogical tools and tools that assist in computing education research. Experience reports should carefully describe a computing education intervention and its context, and provide a rich reflection on what worked, what didn’t, and why. Tools papers should describe aspects of the tool, its purpose, its implementation, and, if possible, an indication of its impact when used. All papers in this category should provide enough detail to enable others to adopt the new innovation.
For a typical paper in this category, here are some key factors to include (as an author) and to look for (as a reviewer):
- Are there one or more clearly stated goals in this paper? Since the rest of the paper will be organised around these, it’s often good to put them in the abstract and early in the body of the paper.
- Is the experience or tool of interest to the CompEd audience? Is the focus on the education of computing students, as opposed to education in general or in other disciplines?
- Related work in computing education
- Is the relevant work in computing education included? If not, a good review will give references to missing material. It is not enough to write ‘The related work section is incomplete’.
- Do the authors clearly describe the relationship between prior work and the experience being reported? In what ways does the current project build on prior work, and how is it different?
- Are the observations and/or findings from the experience or the use of a tool clearly summarised?
- Are the findings thoroughly discussed, including:
- Their relationship to prior work
- The implications of the results for future use
- The implications of the results for teaching or computing education research
- Information on how to adopt or adapt teaching techniques and/or tools in other contexts or institutions.
Curriculum and Position papers
Please note that despite its name, we are considering this category as a catch-all for any papers that don’t fit well into the other two. It is not our intention that papers be rejected solely because we can’t find the right category for them.
Papers in this category include all computing education papers that do not fit well into one of the other two categories. Papers may describe new curricula, programs, or degrees, the motivating context for the new initiative, what it took to put the initiative into place, what the impact has been, and suggestions for others wishing to adopt it. This category also includes position papers, which are meant to engender fruitful academic discussion by presenting a defensible opinion about a computing education topic, substantiated with evidence. Proposals for future work are also included in this category.
- Is the goal of the paper, or the innovation described by the paper, clearly stated? Since the rest of the paper will be organised around this, it’s often good to put it in the abstract and early in the body of the paper.
- Description of the problem or need being addressed.
- Is the main topic of the paper of interest to the CompEd audience? Is the focus on the education of computing students, as opposed to education in general or in other disciplines?
- Related work in computing education
- If a problem is described, what prior solutions to this problem exist?
- Is the relevant work in computing education included? If not, a good review will give references to missing material. It is not enough to write ‘The related work section is incomplete’.
- Do the authors clearly describe the relationship between prior work and the material presented in this paper? In what ways does the current project build on prior work, and how is it different?
- Presentation and discussion
- If a problem is described, how effectively does the solution address the problem?
- How is the idea advanced in this paper different from previous ideas?
- Has the paper discussed the implications of the topic for the computing education community?
- Future success indicators
- If an innovation is proposed, how widely could it contribute to the computing education community, what barriers might prevent widespread adoption, and how would the success of the innovation be measured?
- How might the paper contribute to the computing education community? Who might it impact, and how?
- Value of proposal
- If the paper is presenting a proposal for work yet to be done, does the proposal itself bring value to the computing education community – or would it be better to wait until the work has been done and there are results and findings to present?