Award Program Evaluation Questionnaire
Introduction
A questionnaire was developed to determine whether specific sections and practices of the Talking Hands Award program were effective or in need or revision or enhancement.
The questionnaire was then sent to 50 award honorees on January 15, 2008 who had requested in writing the results of their website review, scoring and comments. The fifty targeted website owners were selected at random based upon those who still had a web presence and were asked to respond not later than March 31, 2008. A second questionnaire was sent to 100 using the same selection criteria in August 2008.
Technical Requirements of The Questionnaire
Before the questionnaire was compiled a planning rubric was developed which served as an outline for the questions "which should be" asked. Then depending upon the outcome, a basic course of action.1
Features:
The questionnaire should require narrative input with comment, rather than a bullet list; the questions must be unambiguous requiring the submitter to be familiar or refamiliarize themselves with the mission statement of the Talking Hands Award and a valid response poll would be greater than 80% responding to be considered valid by myself and staff.2
The Questionnaire Questions as developed
- Does the questionnaire relate to the outcome being measured?
- Does it address anything extraneous?
- Do the questions reflect current conceptions of excellence in the field?
- Can everyone understand the questionnaire?
- Is the questionnaire developmentally appropriate?
- Can the questionnaire results be applied to a variety of tasks?
- Is the rubric fair and free from bias?
- Does it reflect teachable skills or does it address variables over which we have no control such as culture, gender or home resources?
- Is the questionnaire useful, feasible, manageable and practical?
- Will it provide the kind of information you need and can use effectively?
First Questionnaire and Results
47 of the fifty questionnaires were returned within the requested time frame. Of the three not responding and after a second mailing, two responded. An independent group tabulated the results based upon the 49 respondents. The results are shown under each question. The complete questionnaire consisted of 10 questions. Percentages are expressed as: a percent of answers furnished by the 49 respondents.
Question 1: What if any of my associations, AP ratings and "about myself" or any other consideration influenced your decision to request in writing the results of your web site review?
Results: 45%-clarity of purpose including honorees supportive of the purpose; 25%-knowledge expressed in narrative, design and formal education. Mutually inclusive; 25%-the value of an in-depth website review in a formal form; 5%-All others.
Question 2: What were you doing on the Internet when you found my AP which prompted you to apply for my website award?
Results: 45%-Searching Accessibility web site issues; 30%-looking for a CSS tutorial; 20%-looking for suitable poetry (birthday, patriotic and so forth); 5%-All others.
Question 3: What do you gather as my intent to "build a better Internet" after reading the Talking Hands Award criteria?
Results: 55%-Accessable by anyone in any browser (graphic or non graphic); 30%-portable to mobile devices; 10%-assessing the desirability of advancing technical expertise in web building; 5%-All others.
Question 4: What if any comment do you wish to offer as to our continuance of a web site review after finding DQ items?
Results: 85%-Extremely useful. Changes are apt to be made with your provision which would "better the Internet"; 10%-would probably not either influence a change or reapplication; 5%-All others.
Question 5: What would you expect from me to render a professional evaluation of your website?
Results: 40%-Completness:substantiated reasons why a particular "task" would "better the Internet;" 40%-help and guidance; 15%-honesty and objectivity; 5%-All others.
Question 6: What if anything within the Award Section did you find useful to such an extent that you would make a recommendation for others to explore?
Results: 30%-tutorials; 30%-informationals which addressed specific questions we had; 20%-visuals and step by step instructions; 20%-All others.
Question 7: What if anything would you do differently in order to make the score sheet more "understandable?"
Results: 80%-Express in a tabular form based upon skill level; 20%-All others.
Question 8: Considering that recent surveys indicate that less than 2% of all web sites are W3C/WDG and WCAG/WAI level one compliant, why do you think either or both certifications should at least be attempted?
Results: 40%-Shows a willingness to progress individually or professionally; 30%-Reaching a broad audience, rather than specific targeted groups; 30%-All others.
Question 9: Why did you apply for the Talking Hand Award?
Results: 80%-wished to have an independent analysis of the website; 20%-All others.
Question 10: Assuming that you wished to start an award program of your own, what would be your considerations and how would you go about establishing the award program?
Results: 25%-Prepare an outline of what is to be accomplished; 20%-Determine what topic would be the theme; 20%-Time and skill level required; 15%-General or targeted group; 15%-Design for interoperability; 5%-All others.
Second Questionnaire and Results
In August 2008 a second questionnaire was sent to another separate group of 100. 97 responded and the results tabulated below. The questionnaire was just three questions, while 10 were in the first mailing. The results did not vary significantly.
Question 1: Considering that recent surveys indicate that less than 2% of all web sites are W3C/WDG and WCAG/WAI level one compliant, why do you think either or both certifications should at least be attempted?
Results: 60%-Shows a willingness to progress individually or professionally; 30%-Reaching a broad audience, rather than specific targeted groups; 10%-All others.
Question 2: Why did you apply for the Talking Hand Award?
Results: 85%-wished to have an independent analysis of the website; 15%-All others.
Question 3: Assuming that you wished to start an award program of your own, what would be your considerations and how would you go about establishing the award program?
Results: 40%-Prepare an outline of what is to be accomplished; 20%-Determine what topic would be the theme; 10%-Time and skill level required; 15%-General or targeted group; 10%-Design for interoperability; 5%-All others.
While the results and or the questionnaire may not apply to your award program, the time and effort required by our staff certainly has lent encouragement and some degree of satisfaction. Especially when one may consider that one person can make a difference, although slight.
All we have to do is try. That is all our Lord asks of us when we first meet.
Footnotes:
1 The questionnaire planning rubric consisted of five sections, detailed below:
Truth
- What is my current situation? (honestly and objectively look at current situation)
- What is the core issue or problem?
- What are the consequences associated with this problem?
- What are the causes associated with this problem?
Vision
- What would I like my situation to be or look like (ideally)?
- What are my goals?
Alternatives
- What options do I have?
- What are likely outcomes?
Obstacles/Barriers
- What personal obstacles or barriers hold me back?
- What are the consequences?
Action
- What can I do?
- What can I do?
- What is one thing I can do immediately?
- How is this congruent with my goals and values?
- What can I do on a ritual (daily, several times a week, weekly) basis?
- What am I willing to do?
- What are the consequences for these actions?
2 Respondents were encouraged to write in their own answers to the questions, rather than making any suggestions which might skew the results.