The following review is an example from a paper that was rejected (see below for one from a paper that was accepted).
— Summary —
This paper makes two contributions. First, an instrument used to collect data about mobile learning and second, the results of applying that questionnaire to students.
— Strengths —
The authors adapt other questionnaires used to evaluate web-based learning to a mobile environment. There are few research papers that investigate mobile learning rigorously, so it is good to see work in this area.
— Weaknesses —
The paper reports on what looks to be part of a much bigger research project, and one that appears to be worthwhile. However, this particular research paper has some significant weaknesses:
1. The research questions/goals of the paper are not clearly identified anywhere. I struggle to identify what the authors think the major contribution of the paper is. Is the intention that the instrument itself is the contribution, or is the data collected from students the contribution?
2. The purpose of the study is described in the first paragraph of the methodology as “investigate how the learning context could be improved by using mobile technology…”, however, this paper is about an instrument and a survey that asks students if they like learning using mobile devices – not at all about *how* learning could be improved.
3. The background / related work is unfocused and wide-ranging. For example, the second paragraph of section 2 focuses on different generations, such as the x-gen, which seems almost completely irrelevant to the instrument presented in the paper, or to the results of the questionnaire (although there might be some difference between students of different ages, it doesn’t seem to be necessary to delve into the “generation” literature).
4. The questionnaire is presented in detail, but there is no detail about the context in which the questionnaire is applied – which students were surveyed and what was their background? In other words, what was the context in which this instrument was applied? The focus of the survey has been mobile phones, but there is a substantial difference between smartphones and “dumb” phones that provide little more than txt messaging capability. Was any attempt made to distinguish between different kinds of phones? One plausible explanation for the difference between ages is that older people have fewer smartphones, so the phones themselves have less capability for learning.
5. The meaning of “preferred” and “actual” questionnaire results is never discussed.
6. Several scales ([…], […], […]) are presented in detail, but the […] scales are not – why not?
7. The conclusions contain a lot of speculation that is not well supported by the data. For example, the conclusion claims that this study provided evidence of the benefits of using mobile technologies — since this is a survey of students, the best that could be claimed is that students believe that there are benefits.
8. Many of the paragraphs are very long and unfocused. It would be beneficial to identify the main topic of each paragraph.
— Overall evaluation (2: Clear Reject)—
The paper appears to be related to a larger body of work which likely has merit. However, this paper does not have a clear research focus, which makes the narrative of the paper meandering. I have serious concerns about the conclusions, which make claims that are not supported by the data. Overall, the lack of coherence and concerns about the validity of the conclusions suggest that the paper would benefit from significant revisions before publication.
The following review is an example for a paper that was accepted:
This is an interesting paper that reports on changes to the structure of an introductory programming course and reflects on the perceived impact of those changes.
— Strengths —
* The course changes are justified with reference to the education literature on active learning, automated feedback, and program visualization.
*Due to the substantial interest in first year programming, it is valuable to see different approaches, particularly those that appear to be grounded in literature. Although this paper is relatively light on robust research, it does make a useful contribution by reporting on changes in teaching that practitioners might find inspirational.
* The paper was well written and enjoyable to read. A few minor types were apparent, so a final proof-read would be beneficial.
— Weaknesses —
* Given the focus of the paper is about the redesign of a course, the literature review has a relatively narrow focus and would be improved with a paragraph or two with a broader focus on instructional design for first-year programming courses. The following paper might be a good starting point here:
[paper reference removed]
* The results indicated that more students passed the course, but the overall average remained similar. I would have liked to have seen an acknowledgment that the results of a course are entirely dependent on the exam questions, and minor changes in the content of exam questions can have a significant impact on the results. It is therefore extremely difficult to determine if a change in a course is good or bad from exam scores. At the very least, a claim about an exam would be more believable if an independent researcher evaluated the difficulty of the exam (or alternatively, apply some of the various metrics that have recently been shown to correlate with exam difficulty as an independent measure). Essentially, the results boil down to “anecdotally, we think it worked better” since the exam scores are not an independent measure, and we are asked to take it on faith that the exam was equivalent (or even harder). This is not uncommon in the research literature, but it would have been better to see it appropriately acknowledged as a major threat to the validity/generalisability of the findings.
— Overall (5: Clear Accept) —
The justification of design choices are supported by the literature and although the evaluation is relatively small, it appears to have been well conducted. Although there are some improvements possible, the paper is engaging and contains several interesting points. Overall, this is a well-structured case study that will be of interest to a broad audience.