Practical Assessment

gathering, evaluating and using data

Nicholas Schiller & Sue F. Phelps
Washington State University Vancouver

Practical Assessment:

• Real solutions to real needs that you all can go forth and implement
  • Activity: brainstorming existing assessment needs
  • Activity: matching needs with methods
• Design Assessment
  • Assessing personal projects, creations, and instruction
  • Small scale, "Did it work?", quick and dirty assessments
• Program Assessment
  • Larger scale & more scientific assessment of programs and services
  • Surveys, focus groups, program assessment, etc.

Activity: Brainstorm assessment needs

• What do you need to assess? (needs)
• What kind of assessment needs to you have?
• What do you want to measure?
• What have you created?
• Do you need to justify a funding request?

Design Assessment:

• Objects we design:
• classes, web pages, handouts, activities, etc
• We can learn from designers outside of libraries (instruction, web, and game designers)
• Build assessment into our designs

Outcomes Oriented Design:

From Deb Gilchrist and the ACRL IL Immersion faculty (modified)

• What do we want you to be able to do? (Outcome)
• Identify practical assessment solutions for your libraries' needs
• What do you need to know in order to do this well? (Content)
• Basic library assessment principles
• Your assessment needs
• Examples of assessment methodologies
• Criteria for selecting a methodology
• What activity will facilitate the learning? (Action)
• Brainstorm list of assessment needs
• Group activity (brainstorming / matching)

Outcomes: continued

• How will you demonstrate the learning? (Assessment)
• Observation of brainstorm and group activity ("Anthropologist")
• Implementing new assessment techniques in their libraries and reporting back (Modified blog comment)
• Clearest Point / Muddiest Point
  
  
• How will we know if you all have done this well? (Criteria)
• Active engagement in discussion and activities
• Successfully match solutions to the list of needs

Many Small Assessments

• Don't let assessment infrastructure outweigh your service
  
  
• Vary techniques and measure different things
  
  
• Make the assessment part of the design rather than tacked-on as an afterthought

Many Small Methodologies:

• Muddiest point / clearest point assessment
• Peer review assessment
• Quick write / journal assessment

"Don't Listen to Users"

• To design an easy-to-use interface, pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior. (Jakob Nielson on web usability)
• Similar to a reference interview: respectful ≠ literal
• Anthropologist Assessment: be an anthropologist, forensic detective, or other keen observer of user behavior as patrons interact with the object you have designed

Assessment 2.0

• Use new communication techniques!
• "Let's make it easy for an individual to make a meaningful contribution with little effort." - Clay Shirkey
• Blog Comment Assessment
• Make it easy, natural, and expected for interested patrons to offer feedback
• This is built right in to many new communications technologies, take advantage!

Find other sources of assessment

• Snitch assessment
• Ask other assessors / evaluators
• Parents or teachers

Who else does assessment?

• Educators
• CAT Classroom Assessment Techniques
• ACRL Immersion
• People who think about the web:
• Clay Shirkey Here Comes Everybody
• Jakob Neilson's Alertbox
• Steve Krug: Don't Make me Think!
• Video Game Designers:
• Kim Swift and Valve Software's Portal team
• Watch the developers' commentary from Portal, part of the Xbox 360 and PC game package The Orange Box

Program Assessment

• Larger scale & more scientific assessment of programs and services
• Surveys, focus groups, program assessment, etc.

Focus Groups and Interviews:

• Use when:
• You are unsure of the issues
• To write survey questions
• To target a specific population
• When you want to know WHY
• Focus groups are more efficient and economical than one to one

Issues:

• Plan ahead
• Cost to the library
• Food for group members
• Other incentives
• Facilitator skills are very important
• Good timing, directive
• Good to have a note taker, too
• Transcription is tedious
• Not always necessary

A Tiny Little Survey:

Are You Cold?

Surveys:

• When other methods won't work
• Strategic planning
• Patron satisfaction
• Plan for additional services
• Gap analysis
• Large number of respondents
• Quantitative analysis

Issues:

• Time consuming
• Make it a group effort
• Good questions are key
• Cost
• Paper or electronic: email or web site
• Analysis
• Tally paper by hand or electronically
• Survey Monkey
• SPSS

Survey Monkey Results:

SPSS Results:

Space Studies:

• When you can make changes
• Moving furniture
• Remodeling
• New building
• When you have to prioritize
• When parts of your building are not being used
• To validate other studies

Issues:

• Not a lot of literature on this topic
• You get to be creative!!
• Can bring up questions for a survey or validate survey
• General questions
• Target specific area of issues

Seating Survey:

Rubrics:

• All purpose tool for assessment
• A guide for rating performance with a full range of descriptors for each level
• You can use them to assess almost anything
• Validity with several evaluators

Information Literacy

Issues:

Links to Presentation Materials

• Main Presentation Page
• Suggested Readings
• Assessment Methodologies Lists