Usability Testing

return to PlanYourPlace

---

The projects requires the application of usability testing, since the users of the platform will be "normal people" and planners without expert knowledge for computers and GIS. Below we outline literature, test methods, and a possible test plan for the PYP project. See also the mindmap that we made during the 1st usability meeting on LucidChart.

Reading

two helpful reads on the topic are:

• Rubin, J., and D. Chisnell. Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. Wiley-India, 2008 (online in the library: http://library.ucalgary.ca/books).
• Nivala, A-M., S.Brewster, and T. Sarjakoski. “Usability Evaluation of Web Mapping Sites.” Cartographic Journal, The 45, no. 2 (2008): 129-138. (can be found on zotero)

What covers usability?

We want to measure the absence of frustration (i.e. avoid frustration by good design). Components:

• Usefulness : can the software/thing do what it was build to do (achieve my goals)? (function-wise)
• Efficiency : how fast can I accomplish the goals
• Effectiveness : how many errors did I make when trying to reach my goals? : error rates
• Learnability : do I need training before I use the software?
• Satisaction: my feelings and opinions about the software
• Accessibility: can I use the software from work, from home or in what ever situation? Can disabled people use the software?

What a test plan should contain

from Rubin and Chisnell (2008), Chapter 5

• How - what type of test is applied
• When - when do we test w.r.t the development schedule
• Where - where do we test: in class, at work, at home, etc.
• Who - who do we test: age groups, computer literacy groups, planning literacy groups, etc.
• Why - why do we test: to identify problems? to decide about a design? etc
• What - what do we want to test: layout, designs, functions, accessibility, help documents, etc.,
• What - what data do we want to gather: error metrics, preferences, completion time, etc.

Test methods

types of tests/evaluations (Chapter 1.4): Please note, that the order corresponds to the possible application in the development process.

1. ethnographic research: observe users at some place to profile them and tasks they do
2. participatory design: employ future users in the design process
3. focus groups: evaluate preliminary concepts with representative users (its more about discovering feelings = qualitative)
4. surveys: allow to generalize results (larger sample size)
5. (cognitive) walk-throughs: conceptual or early prototype evaluation for given tasks by the design team
6. (open & closed) card sorting: design for "findability" of functionality
7. paper prototyping: create mockups, and ask others (e.g. users) about the design
8. expert evaluations: experts assume they are a test group and check the software
9. usability testing : SEE BELOW: collecting data while observing end-users doing real tasks (can be formal or less formal)
10. follow-up studies : e.g. to see how users use the current product (what's missing etc.)

The term usability testing considers only tests that involve a representative group of testing participants. Other types of testings are possible too - see below. The following four tests are described in Chapter 3

• exploratory (or formative) test : testing the high level concepts only (qualitative)
• assessment (or summative) test : detailed functionality testing, task-based (quantitative)
• validation (or verification) test : to test against benchmarks or to check if problems have been solved, task-based, measuring for instance error rates, completion time, preferences, etc.
• comparison test : comparing designs, implementations or with competing products: Which is easier to use? use if performance and preference data. Best is to use radically different designs in a test, not similar ones.

For an application example of these tests see Chapter 3.6

Scenario and task-based testing

• Example for a scenario: You visiting London as a tourist...
• Example for task: When you are in London you know that you want to visit on one day first Big Ben and then Madame Tussauds. Use the web mapping application to find out how you can go from Big Ben to Madam Tussauds!

Task based testing allows to measure performance, such as success rates, completion times, etc.

What needs to be tested for PYP

We should test all the functionality that is outlined in the development milestones:

• finding and signing up for PYP (1. year)
• informing people and city (1. year)
  • creating a development project
  • joining a development project (as an interested person)
  • uploading, viewing, downloading of documents
  • reporting something
• discussion with other visitors and planners (1. year)
• ranking and voting (1. year)
• evaluation (2. year)
  • understanding of evaluation concepts (e.g. metrics for sustainability)
  • presentation of evaluation results
• sketching (2. & 3. year)
  • using sketch tools
  • modification of plans
  • creation of new plans
• sharing plans (2. & 3. year)
• 3D visualization (2. & 3. year)
  • navigation
  • understanding of perspectives, what is seen, etc
• ... more detail should probably be added

Test plan drafting

.. the draft should answer: what is tested - how (what test) - with whom (participants) - where (location) - when