There is a recent paper in the journal Neuroimage, titled “Ten ironic rules for non-statistical reviewers”. It is a sarcastic critique of the excuses that reviewers come up with to reject manuscripts, which are often just excuses used when legitimate statistical principles for rejection are lacking. It begins with the line: “As an expert reviewer, it is sometimes necessary to ensure a paper is rejected.” !!!
The following excerpt will give a good idea of the paper’s thrust.
“Rule number four: the under-sampled study
If you are lucky, the authors will have based their inference on less than 16 subjects. All that is now required is a statement along the following lines:
Reviewer: Unfortunately, this paper cannot be accepted due to the small number of subjects. The significant results reported by the authors are unsafe because the small sample size renders their design insufficiently powered. It may be appropriate to reconsider this work if the authors recruit more subjects.
...in the unhappy event the authors are allowed to respond – be prepared for something like:
“Response: We would like to thank the reviewer for his or her comments on sample size; however, his or her concerns are statistically misplaced. This is because a significant result (properly controlled for false positives), based on a small sample indicates the treatment effect is actually larger than the equivalent result with a large sample. In short, not only is our result statistically valid. It is quantitatively stronger than the same result with a larger number of subjects.”
… On the bright side, the authors did not resort to the usual anecdotes that beguile handling editors. Responses that one is in danger of eliciting include things like: “Response: We suspect the reviewer is one of those scientists who would reject our report of a talking dog because our sample size equals one!”
Bold and italics mine; hat tip to Jeremy Fox at Dynamic Ecology for making the paper known. And remember, always use a sample size of at least 16, even with talking dogs, just to be safe.