{"id":400,"date":"2018-11-16T18:57:34","date_gmt":"2018-11-16T18:57:34","guid":{"rendered":"http:\/\/www.acmcomped.org\/?page_id=400"},"modified":"2018-11-16T20:37:39","modified_gmt":"2018-11-16T20:37:39","slug":"review-samples","status":"publish","type":"page","link":"https:\/\/comped2019.next500.org\/review-samples\/","title":{"rendered":"Review samples"},"content":{"rendered":"

The following review is an example from a paper that was rejected (see below for one from a paper that was accepted).<\/strong><\/p>\n

— Summary —
\nThis paper makes two contributions.\u00a0 First, an instrument used to collect data about mobile learning and second, the results of applying that questionnaire to students.<\/p>\n

— Strengths —
\nThe authors adapt other questionnaires used to evaluate web-based learning to a mobile environment.\u00a0 There are few research papers that investigate mobile learning rigorously, so it is good to see work in this area.<\/p>\n

— Weaknesses —
\nThe paper reports on what looks to be part of a much bigger research project, and one that appears to be worthwhile.\u00a0 However, this particular research paper has some significant weaknesses:<\/p>\n

1.\u00a0 The research questions\/goals of the paper are not clearly identified anywhere.\u00a0 I struggle to identify what the authors think the major contribution of the paper is.\u00a0 Is the intention that the instrument itself is the contribution, or is the data collected from students the contribution?
\n2.\u00a0 The purpose of the study is described in the first paragraph of the methodology as “investigate how the learning context could be improved by using mobile technology…”, however, this paper is about an instrument and a survey that asks students if they like learning using mobile devices – not at all about *how* learning could be improved.
\n3.\u00a0 The background \/ related work is unfocused and wide-ranging.\u00a0 For example, the second paragraph of section 2 focuses on different generations, such as the x-gen, which seems almost completely irrelevant to the instrument presented in the paper, or to the results of the questionnaire (although there might be some difference between students of different ages, it doesn’t seem to be necessary to delve into the “generation” literature).
\n4.\u00a0 The questionnaire is presented in detail, but there is no detail about the context in which the questionnaire is applied – which students were surveyed and what was their background?\u00a0 In other words, what was the context in which this instrument was applied?\u00a0 The focus of the survey has been mobile phones, but there is a substantial difference between smartphones and “dumb” phones that provide little more than txt messaging capability.\u00a0 Was any attempt made to distinguish between different kinds of phones?\u00a0 One plausible explanation for the difference between ages is that older people have fewer smartphones, so the phones themselves have less capability for learning.
\n5.\u00a0 The meaning of “preferred” and “actual” questionnaire results is never discussed.
\n6.\u00a0 Several scales ([…], […], […]) are presented in detail, but the […] scales are not – why not?
\n7.\u00a0 The conclusions contain a lot of speculation that is not well supported by the data.\u00a0 For example, the conclusion claims that this study provided evidence of the benefits of using mobile technologies — since this is a survey of students, the best that could be claimed is that students believe that there are benefits.
\n8.\u00a0 Many of the paragraphs are very long and unfocused.\u00a0 It would be beneficial to identify the main topic of each paragraph.<\/p>\n

— Overall evaluation (2: Clear Reject)—
\nThe paper appears to be related to a larger body of work which likely has merit.\u00a0 However, this paper does not have a clear research focus, which makes the narrative of the paper meandering.\u00a0 I have serious concerns about the conclusions, which make claims that are not supported by the data.\u00a0 Overall, the lack of coherence and concerns about the validity of the conclusions suggest that the paper would benefit from significant revisions before publication.<\/p>\n

<\/div>\n
\n
\n

The following review is an example for a paper that was accepted:<\/p>\n<\/div>\n

—\u00a0Summary —<\/span><\/div>\n
\n

This is an interesting paper that reports on changes to the structure of an introductory programming course and reflects on the perceived impact of those changes.<\/span><\/p>\n

\n

—\u00a0Strengths —
\n*\u00a0The course\u00a0changes are justified with reference to the education literature on active learning, automated feedback, and program visualization.
\n*Due to the substantial interest in first year programming, it is valuable to see different approaches, particularly those that appear to be grounded in literature. Although this paper is relatively light on robust research, it does make a useful contribution by reporting on changes in teaching that practitioners might find inspirational.
\n*\u00a0The paper was well written and enjoyable to read. A few minor types were apparent, so a final proof-read would be beneficial.<\/p>\n

\n
\n

— Weaknesses —<\/span><\/p>\n<\/div>\n<\/div>\n

*\u00a0Given the focus of the paper is about the redesign of a course, the literature review has a relatively narrow focus and would be improved with a paragraph or two with a broader focus on instructional design for first-year programming courses. The following paper might be a good starting point here:<\/p>\n

[paper reference removed]<\/p>\n

*\u00a0The results indicated that more students passed the course, but the overall average remained similar. I would have liked to have seen an acknowledgment that the results of a course are entirely dependent\u00a0on the exam questions, and minor changes in the content of exam questions can have a significant impact on the results. It is therefore extremely difficult to determine if a change in a course is good or bad from exam scores. At the very least, a claim about an exam would be more believable if an independent researcher evaluated the difficulty of the exam (or alternatively, apply some of the various metrics that have recently been shown to correlate with exam difficulty as an independent\u00a0measure). Essentially, the results boil down to “anecdotally, we think it worked better” since the exam scores are not an independent measure, and we are asked to take it on faith that the exam was equivalent (or even harder). This is not uncommon in the research literature, but it would have been better to see it appropriately acknowledged as a major threat to the validity\/generalisability of the findings.<\/p>\n

—\u00a0Overall (5: Clear Accept) —
\nThe justification of\u00a0design choices are\u00a0supported by the literature and although the evaluation is relatively small, it appears to have been well conducted.\u00a0\u00a0Although there are some improvements possible, the paper is engaging and\u00a0contains several interesting points.\u00a0\u00a0Overall, this is\u00a0a well-structured case study that will be of interest to a broad audience.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

The following review is an example from a paper that was rejected (see below for one from a paper that was accepted). — Summary — This paper makes two contributions.\u00a0 First, an instrument used to collect data about mobile learning and second, the results of applying that questionnaire to students. — Strengths — The authors … <\/p>\n

Continue reading “Review samples”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"om_disable_all_campaigns":false,"jetpack_post_was_ever_published":false},"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/PaeCzN-6s","_links":{"self":[{"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/pages\/400"}],"collection":[{"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/comments?post=400"}],"version-history":[{"count":9,"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/pages\/400\/revisions"}],"predecessor-version":[{"id":413,"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/pages\/400\/revisions\/413"}],"wp:attachment":[{"href":"https:\/\/comped2019.next500.org\/wp-json\/wp\/v2\/media?parent=400"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}