Double-blind reviewing in IACR conferences, revisited

16 Aug

We provide background on the practice of double-blind reviewing (aka “anonymous submissions”) and discuss its advantages, disadvantages, and side effects. While much of the discussion is general, we concentrate on the specific case of  the conferences run by the International Association for Cryptologic Research (IACR), namely Crypto, Eurocrypt and Asiacrypt. We conclude with an opinion and a suggested course of action. We hope that this post will prompt open discussion on double blind reviewing in general and in  IACR conferences in particular.

The full posting is here (in PDF)

Advertisements

14 Responses to “Double-blind reviewing in IACR conferences, revisited”

  1. Ran Canetti September 5, 2012 at 5:06 pm #

    A delayed response to Christian Cachin’s post:
    Many thanks for the reference to Richard Snodgrass’s writing the
    subject. He actually has a number of writings on this, can avoid the
    paywall at his hompage:

    http://www.cs.arizona.edu/people/rts/publications.html

    Indeed these are thorough articles. Would have made my life easier if I
    found them earlier 🙂

    However I still disagree with his conclusions:

    He quotes numbers of 55%-80% success in hiding the identity. My feeling is that these days the numbers are significantly lower; But regardless of the exact numbers, There is something inherently broken in imposing a mechanism that is known to work only some of the time, and furthermore is easy to subvert and manipulate.

    Furthermore, Snodgrass refers to a journal review situation, where a
    paper is reviewed by a handful of people, and reviews are individual. In a program committee setting, where reviews are meant to be collaborative, things are worse: With these numbers it is very likely that that for a vast majority of the papers we have that some of the PC members know the authors and some do not. This is an unhealthy situation which naturally leads to what I called “underhanded bias”.

    So I dont agree that DBR clearly results in overall better reviews. For some reviewers and some papers this might indeed be so, but in other cases DBR might (and does) actually lead to more bias. Furthermore, in these cases the veil of secrecy allows the
    bias to go uncontrolled.

    I think that in such murky situations it is best to put things on the table as much as possible: This “full information model” seems like the most effective way to avoid those uncontrollable side effects. It is also the best way to teach ourselves (as individuals and as a community) to put biases aside and concentrate on the science.

    BTW, regarding the DB community: there are a couple of other studies in that community regarding the effect of DBR, see:

    http://www.sigmod.org/record/issues/0606/p29-inv-article-madden.pdf
    http://www.comp.nus.edu.sg/~atung/DB_detail.pdf

    These two look at practically the same data and reach opposite conclusions. (They seem to predate Snodgrass’s notes and the TODS decision.)

    I also recently spoke with a number of colleagues in the DB community and there seems to be some major resentment there too to the DBR policy of TODS and SIGMOD. The arguments I heard are very similar to the ones in the IACR community – namely DBR doesnt really work, it just obfuscates things and causes confusion.

  2. stevengalbraith August 29, 2012 at 8:38 pm #

    I am currently co-chair of INDOCRYPT. We are using DBR. As chair, I know the author names and can see all reviews and discussions.

    Without mentioning any specifics, it is my impression that DBR is allowing papers from famous and non-famous authors to be considered on an equal footing. There are several cases where it is clear that the reviewers have not managed to deduce the author identity. On the other hand, it is very hard for me as chair to be un-biased about papers when I know who the authors are. It has been difficult for me to ensure that my judgement has not been influenced by this information — maybe I have failed. Hence, overall I am in favour of DBR.

    On the other hand, I was also on the CRYPTO and EUROCRYPT PC this year. For these conferences the success rate is very low and the competition is very tough. The conference programme is already dominated by researchers from a small number of leading institutions (as is the PC). It was usually obvious who the authors were of papers I was assigned to referee (at least, for the serious ones that had any chance of success). I agree that DBR is not very effective in this setting.

    The issue of authors referring to their own work is not so hard to deal with. Most people seem to have no trouble with this.

    The issue of questions from the PC about authorship can be handled by the chairs. For example, there is sometimes an issue of “If this paper is by X then it should be accepted, otherwise it is plagiarism and should be rejected.” I have seen this several times — and the chairs can easily handle it.

    If there was voluntary anonymity then I would be inclined to submit my papers anonymously.

    To conclude, I’m in favour of DBR overall, but I’m not sure it makes much difference for CRYPTO and EUROCRYPT.

  3. Ran Canetti August 29, 2012 at 12:26 am #

    I’d like to try to give another response to Manoj and Jens, complementary to that of Anonymous. I can certainly sympathize with the convenience of not having to deal with author names when reviewing papers. But this convenience has a price, and it’s not clear to me that it’s worth paying that price.Let me explain:

    My first reaction when reading your postings was “great, we can eat the cake and have it too: Why not allow reviewers to choose to not see author names. these reviewers will get anonymized versions of the submissions.” But at second thought it became clear that this is idea doesnt work. In fact, the reasons why it doesn’t work almost completely coincide with the drawbacks of DBR: PC member interaction will still be awkward (in fact, perhaps even more than with full DBR). Including relevant-but-identifying information (such as code, websites, public experiments) in submissions will still be problematic, Underhanded bias (the type that *shouldnt* happen, not the type that Manoj refers to) is still facilitated, again perhaps even more so.

    This “thought experiment” highlighted for me why DBR is a bad idea in the first place. In theory (ie, for ideal reviewers like Manoj and Jens 🙂 it’s great. But when you consider the side effects, it becomes clear that the price for this convenience is high.

    So it may be better to bite the bullet and learn to deal with author names being out there -ie learn to consciously keep author names out of the equation when evaluating the paper, or perhaps even learn to consciously exercise some extra lenience when reviewing a paper by an unknown author, knowing that they may have had a different training but still have something worthwhile to say.

  4. christian cachin August 28, 2012 at 3:30 am #

    I prefer anonymous submissions because

    (1) discussions in the PC appear less biased towards persons and authors, we talk about “paper A” and not about “X’s paper”;

    (2) the other leading conferences in computer science that have debated the issue openly concluded also to adopt anonymous submissions.

    Supporting (2), the editorial by McKinley (cited by Ran, available here: http://www.cs.utexas.edu/users/mckinley/notes/blind.html) is one example from programming languages. It contains also more references to the same debate in databases. In fact, the ACM TODS journal has changed to DBR in 2006, see http://doi.acm.org/10.1145/1206049.1206050

    I do not buy the argument that “speculation is harmful for the review process” because
    reasonable reviewers should not speculate about authorship but assess the paper’s merit. This is exactly the pedagogic objective that is being imposed by the policy.

  5. anonymous August 28, 2012 at 12:29 am #

    Reply to MP:

    1. This is precisely the issue: why restrict the reviewers from searching through the web to learn more about the work being reviewed, and be more confident about the work’s merits before submitting their reviews? It is an extremely unwise approach to forcefully limit the reviewer’s resources during the review, just to fulfill a narrow goal of the success of the DBR process. It is like winning the battle and losing the war!

    2. The opposite would be true for precisely the same reason. The reviewer knows the names of the authors – this possibility is getting higher and higher everyday with the onslaught of online publication and availability of advanced web-search techniques. He then deliberately gives low scores based on some non-academic reasons, and then successfully defends himself against “underhand bias” using the protection of anonymity. On the other hand, if the names of the authors are there on the paper, the reviewer will be more responsible, cautious, objective and professional with his review — negative or positive — in order to establish his professionalism, something he can ill afford to lose.

    Reply to JG:

    “Unconscious bias” … there you go. It is a dangerous thing. An anonymous submission most likely makes the reviewer speculate, guess who the authors could be; it is unavoidable. The cognitive bias arising due to this kind of (avoidable) confusion and speculation is very harmful for any objective scientific review. More so when a sizable part of the submitted papers is already online and therefore, non-anonymous by definition, when the other part is not. Now when you know the names of some of the authors, why not make the names of all the authors available to you to make the process completely independent of this bias.

    In my opinion, non-anonymous submission will make the process more open and transparent, and that it would make the reviewers more responsible and concrete with his review reports, be it bad or good. It is the widespread perception among the researchers these days that the conference review system — I talk with respect to major crypto conferences — is broken totally mainly due to cronyism (you can rather do a polling on this issue to check how far this is true.) Non-anonymous review system would not restore the system overnight, but it is a wise step towards that.

  6. Jens Groth August 27, 2012 at 12:22 am #

    As a reviewer I like DBR. I deliberately try not to find out who the authors are to ensure I treat each paper on an equal basis and avoid unconscious biases.

    In some cases I know a result already. In other cases I can identify authors based on writing style (although probably not the full author list). But my experience is that for the majority of papers I do not know whose paper I am reviewing.

  7. Manoj Prabhakaran August 26, 2012 at 1:35 pm #

    I’m voting for retaining DBR. Here’s my reasoning.

    1. I’m not sure that the DBR system is completely ineffective. I believe that a significant fraction of the PC/reviewers wouldn’t bother to find out the authors of a paper they are reviewing (even if it is only a web search away), especially if they have been instructed not to actively try and subvert the DBR process.

    2. Let me make a case for “underhanded bias.” Or rather, for using the pretense of anonymity, when the reviewer may actually know who the authors are. If everyone knows that the reviewer knows who the authors are, negative comments about the paper could be interpreted as negative comments about the authors. It would be easier for everyone if there is a chance of anonymity.

    3. Finally, we can (and do) have the best of both worlds, by having some conferences using DBR and some not (e.g., TCC, STOC/FOCS). If all our major conferences abandon DBR, I feel there is a real danger of the system becoming less open in the longer run. Why put all our eggs in the same basket? (We could experiment with ideas like what Ivan suggested below too — a first round of DBR followed by revealing the author names. That could be a third basket to put our eggs in.)

  8. Ran Canetti August 23, 2012 at 10:43 am #

    I agree with you that “mathematically” this phenomenon of underhanded bias doesn’t compute, for all the reasons that you mentioned and others. (For instance, even with DBR the chair(s) knows the identities and so the bias is not really concealed.) But the fact is that these things do happen, and they tend to happen more when author names are not explicitly on the table.

    I don’t think that those reviewers that do it are all necessarily “intentionally malicious”. Somehow we are all “semi honest” with varying degrees of bounds that we set for ourselves. Making author names explicitly known helps us make our bounds explicit (and therefore more rational/uniform) as well.

    It would have been easy to toss all this aside as crack hypotheses (at least until one sees it in ones own eyes) if not for this study in reference [2] in my initial posting:

    http://www.unifr.ch/wipol/assets/files/PhD%20Course/Blank1991.pdf

    In table 9 there they show how the dependence of the acceptance rates of papers on the ranking of the author university changes drastically between the case where the review is anonymous but the reviewer correctly guessed the author identity, and the case where review is not anonymous.

    BTW, regarding your comment on how to reprimand a reviewer that you think has written a biased review. It is always possible to send a gently-put private email on this. (In fact I’ve done this a couple of times.)
    As you say, this is always a very delicate letter to write – but when author names are public writing such a letter is much easier, since you’re not “accusing of cheating” but rather noting the obvious that the author and reviewer are close and therefore the review should perhaps be more toned down.

  9. ivandamgard August 23, 2012 at 9:29 am #

    Maybe there is a way to have the best of both worlds: one could have submissions be anonymous in the first round of reviewing, i.e., until everyone have submitted reviews. And then author names are revealed. Assuming that people cannot delete what they said in the first review, this would seem to give most of the advantages of anonymity, while solving at least some of the problems with it that Ran is pointing out.

  10. also anonymous August 23, 2012 at 2:12 am #

    Let me take on another argument which one can find in Ran’s slides:

    “Unleashes underhanded bias under veil of anonymity”

    Besides the problem pointed out also above that, if one argues that all submissions are already public anyway, and that there would be no such veil of anonymity then, let’s consider a simple example:

    Suppose that submissions are non-anomyous and that, in a PC discussion, I encounter a PC member supporting a paper of someone with whom this PC member is close friends, and I suspect this PC member to act so because of the friendship. Can I use this point, independently of whether the friendship actually biased the opinion or not, in the discussion? Probably not, at least not without starting a diplomatic crisis.

    It’s not quite clear to me how non-anonymous submissions help in this regard. It’s like getting to know some information, having the freedom to form an opinion about it, but not having the freedom of speech to talk about it. Also note that we de-anonymize the authors names of accepted papers afterwards anyway (after-the-fact leakage). It doesn’t seem to stop such things from happening now, why should it become better with non-anonymous submissions?

  11. also anonymous August 23, 2012 at 1:45 am #

    However, the lazy and casual authors would use the practice of anonymous submission as protective ring within which they can gamble.

    If the first point was true, that new results are immediately available, then this point above about lazy authors doesn’t make much sense, does it?

  12. also anonymous August 23, 2012 at 1:41 am #

    I’ve seen or heard this eprint argument a couple of times. For me, it doesn’t really support the idea of abandoning anonymous submissions, it only supports “optional anonymity”, or, rather, “optional de-anonymization”. Right now, an author can decide if he or she wants to put the paper on eprint, but this is not a must.

    As Ivan said, there are many good things about eprint publications, one being that the work is immediately available to the public. If we force submissions to be non-anonymous, someone strongly in favor of non-anonymous submissions can submit his or her paper carrying the authors name to the conference only, instead of submitting it to eprint to de-anonymize it. Then only PC members can see the paper, but not the public.

  13. anonymous August 21, 2012 at 5:46 pm #

    In the present age, where the new results are immediately put on the IACR-ePrint, where they are quickly presented in the rump sessions, where interesting results are discussed on weblogs, forums etc. without any delay, to me, it seems like a joke that the IACR conferences still demand ‘anonymous submissions’ which is a misnomer. All serious and honest authors who submit their work only after they develop full confidence in them, and who can appreciate good results produced by their peers, would always support the practice of ‘submission with full details of the authors’. However, the lazy and casual authors would use the practice of anonymous submission as protective ring within which they can gamble. It is my opinion that the number of ‘bad’ submissions to IACR conferences would be reduced, if this rule of anonymous submission is lifted. This would effectively allow the reviewers more time to read the serious submissions.
    On the other hand, this mockery of anonymous submission has some corrupting influence on the reviewers as you already pointed out that they can intentionally downplay good results (and exaggerate bad results) for many non-academic reasons (e.g. cronyism, fraternity) using the excuse of anonymity, when all details about the authors are practically known to them.

  14. Ivan Damgard August 19, 2012 at 10:21 am #

    I used to be a convinced supporter of anonymous submissions, some years back, but now I am no longer really convinced that it makes a difference to the quality of reviews. However, if you compare the situation now to a few years ago, the main new issue is that people put papers on eprint, regardless of which conference they are going to submit to. This is a good thing for tons of other reasons, but it means that to really have anonymity one has to trust referees not to look for the paper at eprint. It seems that if we trust the referees’ integrity enough that we think they will not do that, we could also trust them to do unbiased reviews even given the authors’ names.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: