Learning Center


Achieve consistency scoring school-wide writing prompts

April 14, 2017

Achieve consistency scoring school-wide writing prompts

Many schools/districts monitor the writing development and growth of students through the administration of common writing assessments. When hundreds of writing samples are collected, the next step is for teachers to score them with a rubric. However, most teachers have not had training on how to achieve inter-rater reliability and consistent scoring using a rubric. Both of these are essential if teachers’ scores are to be deemed reliable and comparable.

Plan time to conduct practice-scoring sessions utilizing the Blind Scoring method. This process is referred to as “blind” because each teacher commits to a score without the influence of the others. Otherwise, they always need the affirmation of their colleagues, and teachers don’t ever build self-confidence in their own scoring. A second by-product of talking when scoring is often “loudest voice wins.” Teachers tend not to argue against the dominant personality in the group, and that is not a way to work on building scoring consistency.

Here’s how it works:

Achieving inter-rater reliability with blind scoring - Rubric1. Prepare the needed materials to conduct such a practice-scoring session with a grade-level team, department, or small group. Collect the students’ writing samples, but make sure their names are not visible; use student numbers instead. Bring enough photocopies of the scoring rubric for each teacher to have his own copy. It’s also helpful to bring sticky notes or scratch paper.

2. Look over the rubric as a group and identify a single facet, category, or trait to score for (e.g., Ideas).

TIP: Before scoring a piece for everything, ease into scoring by simply assessing one characteristic or trait at a time. This is the concept of analytic scoring–analyzing the writing for one area only.

3. Read aloud each sample. Designate a teacher to read aloud a single writing sample to the small group. (Don’t let the teacher reveal the writer’s name or any background on the writing itself. All of this is potential bias and can influence the scorers.)

4. Listen for single trait. As the one teacher is reading the piece aloud, the other teachers are all listening to the piece. They might choose to make notes onto sticky notes/scratch paper while listening. They might choose to study the rubric while listening. However the time is used, they are listening to the writing for the single trait previously determined (e.g., Ideas).

5. Settle on a single score. After the teacher finishes reading, then the listening teachers silently and individually settle on a single score based on the rubric criteria for that single trait (e.g., Ideas).

6. Reveal scores. Go around the circle revealing each teacher’s score one by one.

7. Clarify criteria. It’s likely that teachers will not all award the piece the same score, thus some discussion is necessary. The purpose of this conversation is NOT to convince people they are right or wrong. But rather, it’s to clarify understanding of the criteria and characteristics at each rubric level. Encourage teachers to make notes on their personal rubric copies so they can remember for next time.

8. Repeat steps 2-7. Blind score another piece for the same trait (e.g., Ideas). Continue this process until teachers are gaining confidence and consitency with this trait.

9. Score additional traits. Within the same day, or at subsequent practice-scoring sessions, follow the same procedures for additional traits on the rubric (e.g., Organization, Voice, Word Choice, Sentence Fluency).

NOTE: The teachers can score a writing for five traits without ever seeing it. They can hear ideas, organization, voice, word choice, and sentence fluency. However, they must see the writing in order to score its level of correctness–its conventions. So, when ready to practice scoring for conventions, pass the piece around the table for each teacher to independently read the piece.

Achieving IRR with Blind Scoring

10. Update rubric. At the conclusion of each practice-scoring session, discuss if any revisions need to be made to the rubric to support the clarifications and new understandings teachers have acquired.

Research shows that it takes about 30 papers before a scoring team achieves consistency and reliability. So, the more practice time you have, the stronger you will get. Once teachers have gained inter-rater reliability, they can begin to score common writing assessments individually for all six traits. However, plan to have an abbreviated group-scoring session before independently scoring writing samples during each writing prompt cycle. This allows teachers to recalibrate and get back in sync.

0 0 votes
Article Rating
Notify of
Inline Feedbacks
View all comments
CompCON 2024
Convert rubric scores to grades


Convert rubric scores to grades

Develop school-wide/district-wide writing assessments


Develop school-wide/district-wide writing assessments