Devising a Fair and Consistent Rubric for Scholarship Application Assessments

If you have multiple scorers, and subjective criteria to assess, a fair and impartial rubric can help

This article is part of our series on developing your own scholarship program.

Drawing up a rubric that’s fair and consistent can be a challenge; much of the thought leadership in this area comes from educational assessments. A rubric’s only required if you’re measuring subjective factors, like an essay response. If you only care about objective factors, like GPA, you probably won’t need a rubric at all. (Be aware that essays take a lot more time and effort to score than more straightforward metrics like GPA — to say nothing of the extra effort required of students to produce a good piece.)

In any case, we’ve adapted the fundamentals here. (For sample rubrics, check out the AAC&U VALUE rubrics here.) There’s a few factors you want to consider when crafting a rubric:

1) Diversity of Perspective

You’ll need to decide on a few metrics (called ‘constructs’) that matter most. For an essay, this might mean the Topic Selection, Analysis, and Writing Quality. Or if you’re reviewing multiple essays from a single applicant, you may have one construct per essay for Overall Quality.

Constructs don’t have to “match” one another — that is to say, you can have a construct for age, another for location, and a third for creativity in an art piece. Here’s another interesting taxonomy you could consider: “Accuracy, Originality, Content Coverage, Effort, Errors,” with the quantity or severity graded per scoring levels (below).

The number of constructs required will vary depending on the criteria of the award; somewhere between 3 and 6 constructs is likely sufficient for most cases. Each row of your rubric will assess one specific construct — no more, and no less!

2) Deciding on Scoring Levels (Granularity)

A Likert scale is the most common scale you’ll see, and 5 steps is a good rule of thumb to use. Many educators like 5 steps as it maps relatively easily to letter grades (A, B, C, D, and F). A Likert scale often has a “neutral” or “not sure/not applicable” option in the middle — if you’ve ever “strongly agreed” on a survey, you’ve probably seen a Likert scale in action.

You can get more or less granular in scoring by adding more levels — historically we’ve used 7 in many cases, and just 3 for certain tasks. When in doubt, 5 will probably work. The finer your subdivisions, the more challenging the semantic exercise in differentiating them enough for raters to use effectively.

Pro tip: Watch out for the “halo and horn effect” — putting too much emphasis on one aspect of the applicant, to the point of ignoring a fuller view of the applicant’s skills. Of course, you can use weighting (below) to emphasize specific factors over others in the final scoring.

3) Weighting

Your various constructs may not be equal to one another in importance, so you can weight them statistically.

Even so, this is more art than science; choose the constructs that matter most to you, and assign them the greatest weight. So long as all your weights add up to 100, you should be good to go.

4) Validity Testing (Optional)

If you’ve got statistical knowhow, and time to spare, you may want to test your rubric for statistical validity. According to Jonsson & Svingby, 2007 (https://www.sciencedirect.com/science/article/abs/pii/S1747938X07000188?via%3Dihub), rubrics used for “individual-level interpretations” (like student applications, as opposed to “population level”), reliability testing is less relevant.

If you do want to test, checking for inter-rater reliability (IRR) is most important: do multiple raters come to a similar conclusion when using the rubric and the same sample data?

You’ll want to use anonymized data that’s as close as possible to the data you intend to actually score down the line. So that means student essays, grades, videos — whatever you intend to score.

You can determine the reliability coefficient for your IRR using a formula you can learn more about here. Again, this is a little “extra,” and won’t be required for most scholarship application assessments. But if you have the time, bandwidth, and knowhow, it could be worth experimenting with your rubric by testing its IRR.

Next steps

You may want to go as far as to share the rubric with applicants, so they know exactly what is expected. You can share the document itself, or a simplified, bullet-points version — this may help some students stay on-track and complete their applications.

Check out the rest of the articles in our series on developing your own scholarship program, or reach out to a SchoolFinder Group team member to learn more about building, marketing, and adjudicating your own scholarship program.

Good luck!

info@schoolfindergroup.com

Connect with us