Effectiveness of Anonymization in Double-Blind Review
by Claire Le Goues, Yuriy Brun, Sven Apel, Emery Berger, Sarfraz Khurshid, Yannis Smaragdakis
Abstract:
Double-blind review relies on the authors' ability and willingness to effectively anonymize their submissions. We explore anonymization effectiveness at ASE 2016, OOPSLA 2016, and PLDI 2016 by asking reviewers if they can guess author identities. We find that 74%-90% of reviews contain no correct guess and that reviewers who self-identify as experts on a paper's topic are more likely to attempt to guess, but no more likely to guess correctly. We present our findings, summarize the PC chairs' comments about administering double-blind review, discuss the advantages and disadvantages of revealing author identities part of the way through the process, and conclude by advocating for the continued use of double-blind review.
Citation:
Claire Le Goues, Yuriy Brun, Sven Apel, Emery Berger, Sarfraz Khurshid, and Yannis Smaragdakis, Effectiveness of Anonymization in Double-Blind Review, Communications of the ACM, 2017.
Bibtex:
@article{LeGoues17cacm,
  author = {Claire Le~Goues and Yuriy Brun and Sven Apel and Emery Berger and Sarfraz Khurshid and Yannis Smaragdakis},
  title = {\href{http://people.cs.umass.edu/brun/pubs/pubs/LeGoues17cacm.pdf}{Effectiveness of Anonymization in Double-Blind Review}},
  journal = {Communications of the ACM},
  venue = {CACM},
  year = {2017},
	
  abstract = {Double-blind review relies on the authors' ability and willingness to
  effectively anonymize their submissions. We explore anonymization
  effectiveness at ASE 2016, OOPSLA 2016, and PLDI 2016 by asking reviewers if
  they can guess author identities. We find that 74%-90% of reviews contain no
  correct guess and that reviewers who self-identify as experts on a paper's
  topic are more likely to attempt to guess, but no more likely to guess
  correctly. We present our findings, summarize the PC chairs' comments about
  administering double-blind review, discuss the advantages and disadvantages
  of revealing author identities part of the way through the process, and
  conclude by advocating for the continued use of double-blind review.},

  fundedBy = {NSF CCF-1319688, NSF CCF-1453474, NSF CCF-1563797, 
  NSF CCF-1564162, NSF CNS-1239498, German Research Foundation (AP 206/6)},
}