Blog An exploration of the art and
craft of software development

Blind Reviews at RailsConf 2013

Posted by Marty Haught on Friday, May 03, 2013

For RailsConf 2013, we used a different approach for reviewing all the proposals we got — all reviews were blind. This is where the reviewer can only see the proposal’s title, abstract and detailed information. We reviewers had no idea who submitted the talk, their experience level or gender/ethnicity. Though I have been organizing conferences in some fashion over the last seven years, this is the first time I’ve done a blind review process.

Before I continue, I want to point out that I strive to not let a person’s background or identity unfairly affect how I rate their talks. I think I do a good job since I truly believe that gender, ethnicity and other ways people are discriminated against play no part in the quality of a presentation they might be offering. However, the reality is I can’t be completely objective. We all carry unconscious bias with us and seeing someone’s name or bio could well color how we value their talk. Thus the first benefit of the blind review is that these things are removed.

Second, I know a lot of people in the community, many of which might be submitting talks. No doubt I could easily be swayed to give extra value to a talk from a good friend or a high quality speaker. Worse yet, I might not give an honest rating since it may hurt their feelings. The blind review eliminates these things as well.

It’s quite freeing to not worry about these things. It allows me as the reviewer to totally focus on just the quality of the abstract and talk details. It also makes talk reviews faster since I don’t need to factor in the quality of the speaker or his/her background. For RailsConf, I only did initial screen and rating of the talks and allowed others to sift through nearly 400 proposals in narrowing down to around 50 talks.

If you’re a conference organizer, I encourage you to do your first screening blind, especially if you have a review committee. I would state explicitly in your Call for Proposals that you’ll be reviewing talks blindly and ask that submissions do not reveal the speaker’s identity, gender or ethnicity. It would also be good to have someone not on the committee screen the proposals to make sure they’re sufficiently scrubbed (meaning there’s no leak of the speaker’s identity, gender or other identifying details).

I do believe that as an organizer you’ll still want to curate the final talk selection and considering the actual speaker and their experience is important. In all of my conferences I’ve seen some submissions that looked pretty good but the speaker didn’t have the experience or savvy to pull off what they were proposing. I don’t know if you can get around this with a truly blind review process. However, I see no reason not to have initial talk review and ratings be done blind.

blog comments powered by Disqus