Example - The Blankchester City
Web Site
The requirement is originally defined as:
"Blankchester is a cathedral city in the centre
of England. It is an historical
city with tourists visiting but also has modern industry. As part of the
"Blankchester - a New Millennium" project, Blankchester City Council
have commissioned a web site to raise the profile of Blankchester. The project
will also contribute to the "Blankchester - access to all" equal
opportunities project. It is anticipated that it will be used by tourists and
by residents for information about Blankchester, and by local tourist
information officers for finding information when answering telephone
enquiries. The web site may be accessed via Internet browsers, or for local
authority employees via the proposed Intranet. In the Intranet the local
authority employees will have access to additional information and pages. The
site may be used simply to browse information and additionally to make
bookings for tourist accommodation and events. The website shall be designed
with reference to the requirements of the Disability Discrimination Act, and
shall therefore comply to WAI and RNIB guidelines."
A particular difficulty with setting requirements for a
web site is correctly identifying the tasks. The only safe way is to find out
what users actually want to do with the site (perhaps by investigating how
people use other similar sites, or by user-based evaluation of very early
paper prototypes). The IT
department visits two other Tourist Offices, at Dashmouth-on-Sea and Cipher
Hills, to see how the tourist officers used the site and to look at their site
feedback. This information is fed into the validation and review
process. The validation and review of these requirements gives rise to a
context checklist, part of which is shown below:
Figure 1
context checklist:
|
Blankchester
City web site - Context Checklist
|
|
Goals for this
system: raise the profile of Blankchester, provide access to
information ("access to all" campaign), provide booking system
for accommodation and events in the City.
|
|
Users
|
|
|
|
|
Type
|
Tourist
|
Resident
|
Tourist
Information officer
|
|
Skills and knowledge
No
/ low / good Web/IT
knowledge
No
/ low / good Local knowledge
No
/ low/high previous
experience of Hotel booking
|
ü
ü
ü
|
ü
ü
ü
|
ü
ü
ü
|
|
Attributes (people)
with
sight / hearing / language / mobility
difficulties /
with
other special needs / without special needs
|
ü
|
ü
|
ü
|
|
Smoking
/ non smoking
With
children / without children
With
pets / without pets
|
ü
ü
ü
|
ü
ü
ü
|
ü
ü
ü
|
|
List of tasks
Browsing
general information / tourist attractions / events
Hotel/accommodation/events
searches and bookings
Browse
additional intranet pages
(for
space in the example this is not expanded)
|
ü
ü
N/A
|
ü
ü
N/A
|
ü
ü
ü
|
|
Task checklist
|
Browsing
|
Searches
|
Bookings
|
|
Goal
User
Blankchester
City
|
Finds information
Profile raised
|
Finds information
More rooms let
|
Room
/ event ticket
More
rooms let
Larger
attendance at events
|
|
Output
|
List of general
information meeting personal criteria
|
List of hotels
meeting personal criteria
|
Booked and confirmed
accommodation
|
|
Frequency
|
5000
hits per day
|
5000
hits per day
|
500
per month
|
|
Duration
|
4
clicks max traverse.
|
Hotel
search 1 click from home
page
|
book
hotel less than 1 minute from selection
|
|
Environment
in which the software/system is to be used
|
Tourist
office - Tourist Information Officer's PC - IE5
Tourist
office / library public PC IE5
Other
user - could be a range of browsers - test for IE5 + IE4 + Netscape4
Design
note: ensure text only version of the site available.
(Note: this is a newly identified requirement and must be fed
back to the designers)
|
| |
|
|
|
|
|
Note: A context checklist
may also consider side effects, flexibility, physical/mental demands,
dependencies, safety, and criticality.
The requirements, context checklist and paper
layout of the screens is inspected, using heuristic evaluation checklists. The
context checklist is then used to draw up tests using equivalence partitioning
and boundary value analysis for users against environment.
Figure 2
Partitions
The user partitions are:
1.
Tourist
2.
Resident
3.
Tourist officer
4.
No / low Web/IT knowledge
5.
No / low Local knowledge
6.
No / low previous experience of Hotel booking
7.
good Web/IT knowledge
8.
good Local knowledge
9.
previous experience of Hotel booking
10.
with sight difficulties
11.
without sight difficulties
12.
with hearing difficulties
13.
without hearing difficulties
14.
with language difficulties
15.
without language difficulties
16.
with mobility difficulties
17.
without mobility difficulties
18.
with other special needs
19.
without other special needs
20.
smoker
21.
non smoker
22.
with children
23.
without children
24.
with pet
25.
without pet
The environment partitions are:
26.
Tourist office - TI's PC - IE5
27.
Library public PC IE5
28.
User's own equipment - IE 5
29.
User's own equipment - IE 4
30.
User's own equipment - Netscape 4
31.
Tourist officer uninterrupted
32.
Tourist officer also answering phone enquiries
The tasks will also be
partitioned, from the context checklist, but for ease of explanation we will
consider only one task: hotel booking. BS
7925-2 says that more than one partition can be checked in a single test case,
so we combine the partitions to make a smaller number of tests.
The table below shows 7 tests:
Figure
3 Action/task list – test cases
|
ACTION/TASK:
Book a hotel room
|
|
Test case
|
New Partitions
tested
|
user type / skill /
Attribute (people) / environment
|
Actions
|
Expected outcome
|
|
1
|
1, 4, 5, 14, 16, 20,
23, 24, 27
|
A tourist with sight
difficulties and a guide dog, little web experience, and no local
knowledge, no children, uses the library public PC. She wants a smoking
room. English is not her first language.
|
Browses for a 4 star
hotel and books it
|
Able to book a room
(smokers) at the 4 star Jupiter Hotel, which welcomes pets and guide dogs.
Site / PC has tools to offer information in audio and in languages other
than English
|
|
2
|
2, 6, 7, 8,
10, 11, 13, 15,17,19, 30
|
A local person with
good local knowledge and good IT /web skills but no experience of hotel
booking uses their own PC (Netscape 4) to book hotel rooms for a visiting
party of friends. No Special needs
|
Browses for hotel in
the £80-100 per night range and books hotel
|
Able to find and book
Mars hotel, able to find and print map to Mars hotel
|
|
3
|
3, 9, 26, 32
|
A tourist officer with
little web experience, good local knowledge, previous experience of hotel
booking, no special needs, at own PC, takes enquiry and booking from
tourist in office, while being interrupted by phone calls
|
Browses for hotel in
the £40-60 per night range and books hotel
|
Able to find and book
Saturn hotel, able to answer additional questions, provide map to Saturn
hotel. No timeouts.
|
|
4
|
12, 28
|
A local person with
good local knowledge and good IT /web skills but no experience of hotel
booking uses their own PC to book hotel rooms for a visiting party of
friends. Hearing difficulties, own PC (IE5)
|
Browses for hotel in
the £80-100 per night range and books hotel
|
Able to find and book
Mars hotel, able to find and print map to Mars hotel. Major information given visually.
|
|
5
|
16
|
A local person with
good local knowledge and good IT /web skills but no experience of hotel
booking. Person with mobility difficulties (wheelchair and arthritis in
hands and wrists) Uses library PC to book a hotel room for a friend
|
Browses for hotel in
the £80-100 per night range and books hotel
|
Able to find and book
Gaia hotel, able to find and print map to Gaia hotel. Height and angle of
PC suits needs. Keyboard and pointer device easy to handle and control
without pain.
|
|
6
|
31
|
A tourist officer with
little web experience, good local knowledge, previous experience of hotel
booking, no special needs, at own PC, takes enquiry and booking from
tourist in office, without being interrupted by phone calls
|
Browses for hotel in
the £40-60 per night range and books hotel
|
Able to find and book
Saturn hotel, able to answer additional questions, provide map to Saturn
hotel.
|
|
7
|
18, 21, 22, 25, 29
|
A tourist whose child
has an allergy to animals, no pets, little web experience, and no local
knowledge, her own PC (IE4) to book a no-smoking room
|
Browses for a 4 star
hotel and books it
|
Able to book a
no-smokers room for self and child at the 4 star Cathedral Hotel which has
a no pets policy and check for allergy aware hotels
|
Note 1 - table could
alternatively be drawn up showing all the partitions tested by each case thus
allowing the tests to be used in any order
Note 2: a similar table of
tests would be drawn up for the other tasks
Tests are also drawn up using use cases. An example use
case is shown in Figure 4 on page 4.
Figure 4 Test Scenarios
|
Scenario
name:
|
Enquiry
for hotel
|
|
System:
|
Blankchester
web site
|
V0.9
|
Date:
12-1-2000
|
|
Goal
of test
|
The
goal of this use case is to test the basic enquiry job, where a
(potential) tourist is searching for accommodation
|
|
|
Description
|
|
Context
|
This
is core functionality, so this is an important test of
a frequent usage function
|
|
Actors
|
Any
user may use this function :
Tourist
- low web user, Tourist - high web user, BCC Tourist officer (trained),
Local person - low web user, Local person - high web user
|
|
|
|
Checked
|
Comments
|
|
Preconditions
|
The
PC is on and functional
The
user is connected via their normal ISP
The
user is online and looking at the web site home page and has chosen to use
the drop down selections
|
|
|
|
Description of main
flow
|
2
star in walking distance of cathedral
1.
the user clicks on Hotels
2.
the user clicks on 2 star
3.
the user clicks on location - cathedral
4.
the user clicks on Find
|
|
|
|
Expected
results
|
the
system displays 2 hotels details - "Wardens House" and
"Deans House" and the user can print or save the information
|
|
|
|
Alternative
flows
|
The
user wants to see 3 star hotels - all in Blankchester
1.
the user clicks on Hotels
2.
the user clicks on 3 star
3.
the user clicks on find
|
|
|
|
Expected
results
|
the
system displays a drop down list with all 3 star hotels (15 on test
database)
|
|
|
|
Failure
conditions
|
No
booking conformation number displayed
Wrong
rating hotels displayed
|
|
|
|
Additional
attention points
|
Note
response times
|
|
|
With the constraints on local authority budgets, it is
decided not to use a full usability lab test, but instead to use the
"Thinking Aloud" protocol with an observer logging what the user does,
using the use cases and selecting the user group from the equivalence
partitioning tests already defined. After
the tests are run, the users will be asked to complete a SUMI questionnaire.
Representative users will be asked to complete typical tasks, and
measures taken of effectiveness, efficiency and satisfaction.
It was expected that all Tourist Office users could successfully complete
a hotel booking in an average time of less than 5 minutes. All SUMI scores should be above the industry average of 50.
Context used for the Tourist Office test: For this
part of the test, eight typical users from the Tourist Office were selected who
had the key characteristics and capabilities, but no previous experience of the
web site. The other characteristics
of the participants that might influence usability were recorded, together with
the age group and gender (see Figure 5 on page 4):
Figure 5 User group characteristics
|
|
Time in job (years)
|
Windows experience (years)
|
Internet / web site experience (years)
|
Attitude to computers/ web
(1-7)*
|
Gender
|
Age group
|
|
1
|
5.5
|
3.5
|
0
|
6
|
F
|
20-35
|
|
2
|
0.8
|
2.1
|
0.8
|
1
|
F
|
20-35
|
|
3
|
2.1
|
2.5
|
2.1
|
3
|
M
|
20-35
|
|
4
|
4.9
|
3.5
|
1.5
|
2
|
F
|
36-50
|
|
5
|
0.7
|
0.7
|
0.7
|
2
|
M
|
20-35
|
|
6
|
1.6
|
2.1
|
0
|
3
|
F
|
36-50
|
|
7
|
4.3
|
1.4
|
0
|
4
|
M
|
36-50
|
|
8
|
2.7
|
4.6
|
2.7
|
4
|
M
|
20-35
|
*1=prefer
to use a computer as much as possible, 7=prefer to use a computer as little as
possible
The intended context of use for this test was the
Tourist Office, and so the tests will be carried out there, but it is arranged
so that participants could work alone without any interruptions. They are to be
observed by an observer/tester. Rules for carrying out the tests are agreed:
participants would be shown the evaluation suite, informed that their
interaction would be recorded, they would be asked to sign a release form, asked
to confirm the information they had provided about, and to score their attitude
towards use of computers. Tasks
were to be timed using a stopwatch. Sessions
would not be videotaped but the user asked to use "Thinking aloud" so
that the observer/tester could take notes.
At the end of the sessions, participants would complete the SUMI
satisfaction questionnaire.
Instructions for user testers would be written, for
example:
Figure 6
User instructions
|
Thank you for helping us in this evaluation. The
purpose of this exercise is to find out how easily people like you can use
the city web site. To achieve this, we will ask you to perform some tasks,
and your performance will be observed.
Then, to help us understand the results, we will ask you to
complete a standard questionnaire, and to answer a few questions about
yourself and your usual workplace. The aim of this evaluation is to help
assess the product, and the results may be used to help in the design of
new versions. Please remember that we are testing the software, not
you.
When you have finished each task, or got as far as
you can, please phone us by dialling 1234.
I am afraid that we cannot give you any assistance with the tasks.
You have just been shown the Blankchester web
site. You now need to
find out whether it could meet your current needs.
You will perform the following tasks:
1.
take some time to familiarise yourself with it and specifically the
hotel "find and book" functions,
2.
book a hotel room meeting your requirements
We are interested to know how you go about
these tasks and whether you find the software helpful or not. LET US KNOW WHEN YOU ARE READY TO BEGIN
Task 1–
Familiarisation period - Spend
as long as you need to familiarise yourself with the web site. (YOU HAVE
UP TO 20 MINUTES) LET US KNOW
WHEN YOU ARE READY TO MOVE ON
Task 2 – Book
hotel room (YOU HAVE ABOUT 15 MINUTES FOR THIS TASK) Use the
software to book a hotel room: DATE: 23 November 2001, PLACE: The Blue
Flag Inn, Blankchester. LET US KNOW WHEN YOU HAVE FINISHED
|
After the tests, the mean completion rate, mean goal
achievement, mean task time, mean completion rate efficiency and mean goal
achievement efficiency would be calculated for the task "booking a hotel
room". After the last task,
participants would be asked to complete a subjective ratings scale and the SUMI
questionnaire. The evaluator would
then asked them about any difficulties they had encountered.
Metrics to be gathered were decided as:
§
Effectiveness: Completion Rate Percentage of
participants who completed each task correctly. Mean goal achievement Mean
extent to which each task was completely and correctly achieved, scored as a
percentage.
§
Efficiency: Task time: mean time taken to
complete each task (for correctly completed tasks). Completion rate
efficiency: mean completion rate/mean task time. Goal achievement
efficiency: mean goal achievement/mean task time.
§
Satisfaction: measured using the SUMI questionnaire, at
the end of the session, giving scores for each participant’s perception of:
overall satisfaction, efficiency, affect, controllability and learnability.
Note: SUMI provides a set of statistical tables and graphs
– these are not shown here but can be viewed in the web-sites in the
references.
In addition to these user tests, the following
accessibility checks are carried out:
§
The web site design is checked against the WAI guidelines
(inspections against checklist)
§
The web site is checked against the RNIB guidelines (inspections
against checklist)
§
The web site is checked against the Bobby tool (www.cast.org)
for suitability for use with a text browser
The usability tests are to be documented using the Common Industry Format described in ISO9126.
A.
Other techniques and sources of information
A1.
Techniques listed in the literature and on the web
We do not have space here to
cover more of the techniques available. Other
techniques are:
|
Type of technique
|
Examples
|
Comments
|
|
Inquiry
|
Contextual
Inquiry
Ethnographic
Study / Field Observation
Interviews
and Focus Groups
Surveys
Questionnaires
Journaled
Sessions
Self-reporting
Logs
Screen
Snapshots
|
These
are a selection of methods that gather information as a system is in use,
either by observation of the user, or by asking the user to comment.
|
|
Inspection
|
Heuristic
evaluation
Cognitive
Walkthroughs
Formal
Usability Inspections
Pluralistic
Walkthroughs
Feature
Inspection
Consistency
Inspection
Standards
Inspection
Guideline
checklists
|
These
are all variations on Review, walkthrough and inspection techniques, with
specialised checklists or specialist reviewers.
|
|
Testing
|
Thinking
Aloud protocol
Co-discovery
method
Question
asking protocol
Performance
measurement
Eye-tracking
|
These
are techniques which help the user and the usability tester/analyst to
discuss/discover how the user is using and thinking about the system
|
|
Related Techniques
|
Prototyping
:Low-fidelity / High-fidelity / Horizontal
/ Vertical
Affinity
Diagrams
Archetype
Research
Blind
voting
Card-Sorting
Education
Evaluation
|
Although
not primarily usability techniques, these are techniques that some writers
on usability have recommended. They are all ways to elicit reactions from
users.
|
A2.
Sources of information
Usability research continues and information in this draft
is based on drafts of standards ISO 9126 and ISO 14598, a workshop run by
Improve QS bv , material from the following web sites and documents:
A3.
Accessibility
We have not in this example covered the design methods and
the test methods for website accessibility.
Readers are strongly recommended to use the following resources for
information the reasons for designing with accessibility in mind, or tools,
checklists and hints for designing and testing for website accessibility:
For Usability and accessibility standards and guidelines go to the Testing
Links page.