In June of 2003, we changed our scoring system for the Talking Hands Award after extensive testing to a maximum of 45 points which included a thirty point criteria and a maximum of 15 additional points, but the main objective of honoring teaching and learning remained unchanged.
The prior system was a maximum of 120 points based upon 33 questions with a maximum of 3 points each and 21 points extra which could be earned.
Hypothesis:
Due to the subjectivity of the old scoring system, the new objective scoring system should enable a 90% of prior selective honorees to achieve a minimum score to qualify, prior asking our judges for their unanimous determination.
Methodology:
Ten honorees were selected at random from the years 1999 through 2002 and our review staff was asked to use the new scoring system and arrive at a score. The initial results were plus or minus 5 points from the median score, which was unsatisfactory, so the questions were fine tuned and explanations were given in footnotes where there was a variance in the score. Footnotes and explanations were eventually added to almost every criteria question as a learning experience for everyone, including those who might not even be interested in earning an award for their teaching and learning excellence.
A second set of 10 honorees from the same time frame were submitted to our reviewers with a variance of 2 points which was acceptable.
The score sheet was then sent to the two original sets of honorees, all responded and the variance in score was 5 points, so the mean average which the reviewers had produced was revised to 23 points.
A final set of 10 honorees was sent to our reviewers and the honorees, the scores variance was plus or minus 2 points.
Conclusions:
Statistically, the reviewers and an applicant have a 92% probability of arriving at the same score of 23 based upon the forty honorees used in our sample study.
In addition, the objectivity of the scoring with a simple yes or no coupled with examples and explanations removed most doubt or inequity in the scoring system now adopted.
Summary:
Although the scoring was intended by change to remove any subjectivity and allow an applicant and our review staff to arrive at the same scores, the purpose of the award has remained unchanged, which is to honor teaching and learning excellence.
Copyright: 1986-2010