The OpenWater Blog

7 Tips for Speeding Up the Abstract Judgement Process

Abstract judgement is frequently the most time consuming and data-intensive stage of the abstract management process.  Traditionally, this process involves:

• Piles of PDFs, Word documents, and Excel spreadsheets 
• A crossfire of internal sharing of reviewer lists, abstracts, and scores
• Hours and hours of manually collating the data into something useful

Thankfully, there are ways to help you streamline this messy, cumbersome process. To help you discover better ways of teaching your judges how to review abstracts for conferences, we’ve put together these 7 tips that speed up the abstract judgement process: 

1. Group Your Judges Into Teams, By Topic or Expertise

If you know your reviewers’ areas of expertise (or use a platform where they can indicate this), you can separate them into groups that will grade submissions in the same topic area.  Judges with extra knowledge in the abstract’s area will be able to assign grades quicker and easier. As a side benefit, this also gives you more consistency and fairness in the review process, since the abstract judgement in each topic is done by the same sets of eyes. 

2. Weed Out Ineligible Abstracts Before They’re Submitted

An intelligent abstract submission form can do a lot of your work before the abstracts get to your judges.  A system like OpenWater can set up eligibility criteria (for example, post-graduate degree required, or only PDF attachments allowed) so that no submission can be completed without matching your base level of expectations. 

3. Give Your Grading Method a Second Look

Your judges may be doing more assessment than you actually need them to do.  Are they writing personalized comments on every abstract submission? Do they need to grade seven different categories with a 1-10 score, when four categories with a simple 1-5 score might do the trick? Could you have them rank their top 3 submissions rather than their top 5?

4. Streamline Round One of the Abstract Judgement Process

Multiple rounds of review don’t have to slow down your process.  If the first round has simple standards with a straightforward yes/no assessment or single 1-5 score, you may be able to distill your pool to just the highest quality abstracts before the abstract judgement process kicks into high gear.  The second, smaller round will increase the value of time spent on things like specific text comments.

5. Use Blind Fields to Simplify (and Neutralize) Scoring

There may be fields of the abstract submission form that would only create static for your reviewers to skim through.  If you make those portions invisible to reviewers—with a capable abstract management software—there will be less to distract them from what’s most important.  This can protect presenters from unintentional biases at the same time.

6. Use Autoscoring Software

An abstract judgement software may be able to configure customizable “rules,” each with an assigned point value, and then automatically check the abstracts against your rules to assign a score.  

For example, in OpenWater, you could promote or penalize submissions based on submission date:  

These auto-scores could act as your first wave of judging, or as a supplementary score that allows human judges to focus on higher-value portions of the abstract.

7. Prompt Reviewers With Helpful Reminders

Likely, your reviewers aren’t doing abstract judgement full time, and they’ll probably have competing priorities.  You can help them manage their time and keep track of deadlines by sending out email blasts that alert them to their progress.  Some abstract management systems have built-in functions for monitoring individual reviewer progress and mass emailing reviewers who are falling behind.  Automated personalized reminders can make a difference for presenters, too.

David Lincoln

Planning a Successful Call for Abstracts

The Power of Templates: Annual Abstract Collection Made Easy