Submission and Marking Procedure

Submission and Marking Procedure

Learning Outcomes 

This assignment is designed to provide practical experience of analysing usability requirements, and carrying out an analysis of usability requirements and priorities, performing a systematic usability evaluation using a standard method, and producing a report of your findings.

Submission and Marking Procedure

Part A:  Individual entries on the Module Group Discussion Board for Tutorials 8 to 12.  It is worth 25% of the total mark for the report.

Part B:  This is an individual assignment. It is worth 75% of the total mark for the module.

Your report should be submitted electronically as a Word document or PDF via the Turnitin link for Assignment Two under Assessment on Blackboard.

Task

Your firm of interaction design consultants is trying to build up a portfolio of impressive work, to enable it to pitch for business convincingly in the future.

Your task is to produce a set of usability requirementsand a usability evaluation of an interactive system, plus a report of your results, by applying a systematic evaluation methodology. You have a completely free choice of what interactive system you evaluate.

Carrying out the Assignment

Producing the usability evaluation assignment will involve

1.      Choosing an interactive system to study.

2.      Identifying the use cases or aspects of the functioning of the system to be considered, and briefly describing them in your documentation. (These don’t need to be a complete set of use cases; for very complicated systems focusing on one part of what they do is just fine. However you should give a clear indication of what subset of the functionality of the system you are considering, and what you are not considering. If in doubt, cover less functionality in more detail.)

3.      Define a set of usability requirements, considering what the design really needs to get right to achieve a good user experience, and defining the requirements precisely enough that it would be possible to measure the system’s performance.

4.      Choosing an evaluation methodology. You should apply a standard evaluation methodology such as user testing, cognitive walkthrough, or heuristic evaluation.

5.      Defining an evaluation procedure. This will include stating one or several user tasks to be tested or considered with exact descriptions of the scenario and the goal the user is trying to achieve, as well as what the evaluator will do to collect results and produce an evaluation. The evaluation procedure needs to be described in full, separately from the description of the results.

6.      Carrying out the evaluation. This will involve applying the procedure and documenting what happens, and what the procedure finds. (If applying your procedure looks like an excessive amount of work, or producing an excessively large volume of documentation, ask advice; we would prefer an evaluation giving detailed insight into part of the functionality to an evaluation with broad coverage but a thinner or more superficial analysis.)

7.      Deriving findings about the usability of the interactive system from the results of the usability evaluation. This should include consideration of how strong and how general the conclusions are.

Written Submission 

Your report should comprise the following elements:

·         Part ONE: The interactive system and its users. A brief statement of what the interactive system is and what it does – sufficient to make the rest of the report comprehensible; plus a description of the user populations and the assumptions it is reasonable to make about the capabilities of the users.The word count should be between 150 and 400 words – longer only if really needed.

·         Part TWO: The use cases. Brief accounts of the use cases considered, plus a statement of what you are not considering, if you are only looking at part of the system. A use case diagram is optional. The word count should be between 100 and 300 words.

·         Part THREE: The usability requirements. Brief but exact statements of five precisely focused, testable usability requirements. The word count should be between 300 and 600 words.

·         Part FOUR: The evaluation methodology. An exact description of the evaluation procedure to be followed, including what the methodology you are using, exact descriptions of user tasks being considered, instructions to be given to users in user testing, or the set of guidelines used in heuristic evaluation. In principle, you need step by step descriptions of the correct procedures for completing the use cases you are considering in a heuristic evaluation or cognitive walkthrough, or asking people to perform in a user trial. However if the tasks are long and complicated with a variety of possible paths, something briefer might be needed; consider what is cost-effective, and ask advice if in doubt.The word count should be between 300 and 800 words, plus documentation of instructions or task steps if included.

·         PartFIVE: The evaluation. The results of applying the evaluation procedure: what you saw test subjects doing, measurements of their performance, answers to questions and so on; or evidence for violation of particular design guidelines; or descriptions of how and why beginning users might go wrong in particular places, etc. The word count should be between 1000 and 2500 words.

·         Part SIX: The findings of the evaluation. The findings of your evaluation about the usability of the interactive system. Include comments on how the findings relate to the results of the evaluation procedure, and ideally about how strong the evidence is, as well as judgements of how serious you think the usability problems are. An itemized bullet point structure is likely to be easier to read than long paragraphs of text. This should also include an appraisal of the strengths and weaknesses and successes and failures of the evaluation process. The word count should be between 500 and 1000 words.

·         Appendix. Your notes made during observations of user trials, while conducting a heuristic evaluation, etc, should be included in an appendix. Handwritten notes should be scanned or photocopied. Acknowledgements of any help received, any information we should have when assessing the assignment, etc. 

Guidance 

Some advice on how to do the assignment. 

Writing

DO NOT write more than you need to. Do not bother with unnecessary introductions or generalities about usability or human computer interaction. This is just unwelcome extra work for both you and your tutor. Brief means brief. Terse is good. However you do need to be detailed and exact about your procedure and your results and findings. What Part One needs will depend on the choice of system, but please take seriously thinking about the assumptions you can make about the capabilities of the users. Include word counts for sections.

The Usability Requirements

This section of the assignment is essentially separate from the usability evaluation. You DO NOT need to test your usability requirements. It’s good if you do (and you need to think about how you could), but (1) you are welcome to use a usability evaluation method that doesn’t address them, and (2) we don’t want you to limit your imagination to requirements that you can easily test in your assignment.

Your task is to create a set offive usability requirements covering different aspects of the human computer interaction for your chosen interactive system. The set of usability requirements should include at least one with a performance measure, and at least one with a preference measure. You need to consider which subset of your user population each requirement applies to;including one or two for disabled users is also good if they are likely users of your chosen system.

A usability requirement should include

·         Statement of the usability attribute

·         Statement of how it will be measured

·         Statement of the criteria that will represent attainment of the specification

·         Statement of the subset of users to which the specification applies

·         Statement of the pre-conditions of measurement (e.g period of training)

The criteria of attainment should include

·         The worst case: The lowest acceptable level of performance

·         The planned case: The target level of performance

·         The best case: The highest level of performance that can be hoped for in favourable conditions

·         The now level: The level of performance achieved by the current system to be replaced, if applicable.

The usability attributes to be evaluated should be precisely focused on particular user tasks and be clearly and narrowly defined aspects of the usability of the system for the task. (Look at the handout on Some Usability Factors for hints.) These ought to be a wide variety, to demonstrate your ability to consider different aspects of usability. The measures should be clearly defined so the reader can see exactly will be counted or measured, and should be feasible; you will get credit for well-chosen valid measures. The subset of users to be covered by the usability requirements needs to be clearly stated and should be varied, and the preconditions for making the measurement should be clear. The criteria of attainment should ideally be sensible.

Each individual usability requirement will be judged on having

·         A usability attribute that is precise and task-focused, and important for the success of the system. Going beyond the most obvious options will be appreciated.

·         A measurement that is sufficiently precisely defined, is feasible, and is likely to be valid.

·         Clear and sensible statements of the subset of users and the preconditions of measurement, that are restrictive enough to make the measurement meaningful.

·         Statements of attainment criteria that are clear and ideally not crazy.

The set of usability requirements will be judged on having

·         A broad and well-chosen coverage of different aspects of usability.

·         Both performance and preference measures.

·         Requirements for potential test subjects who represent a sensible range of users.

The Usability Evaluation

This does NOT NEED to have anything to do with your usability requirements.

You are expected to apply a systematic evaluation method. That is, you need to do a user trial, or a heuristic evaluation, or a cognitive walkthrough. (Make sure you know what a heuristic evaluation or a cognitive walkthrough actually is, before claiming to be doing one.) If you want to use a different approach to doing a usability evaluation, ask advice.

We are looking for thorough and detailed evaluations, and especially findings about exactly where there are actual or potential usability problems.  If you think you are doing a disproportionately amount of work, or writing an enormous amount, then you should aim to be thorough and detailed, and compromise on how much of the system you cover.

If you do a user trial, you should aim to observe carefully what your subjects do and where they make mistakes or find things confusing and awkward, and report sources of problems as exactly as possible. Timings for tasks and subjective satisfaction ratings in debriefing are valuable but less interesting than actual usability problems. Remember that having carefully designed, realistic tasks is important, and that unless you want to look at exploration or browsing the task should have clear end points and success criteria. The exact wording of instructions matters. So these need to be described exactly in your documentation.Don’tbe over-directive: provide clear goals and enough information about the scenario, but don’t tell people what to do.

Heuristic evaluations are likely to be more successful when using more detailed and concrete sets of guidelines than just Nielsen’s ten broad categories of usability problems. We recommend Nielsen’s 113 design guidelines for web homepages for doing heuristic evaluations of websites.

For cognitive walkthroughs, you need to describe the procedure including the questions to be considered at each step, and show evidence that the questions have been systematically used in the evaluation.

This is an individual assignment, and we are expecting you to work alone (apart from test subjects if you do a user trial). However some activities might be done better with more than one person doing them. You may recruit assistance, but if you have help, you need to describe this clearly in your report.

Choice of Interactive System

The assignment gives you a completely free choice of what interactive system you consider; but it needs to be a real, existing interactive system that you have access to and can study.

Possibilities include software applications such as programming language development environments or case tools or games or photo editing systems; e-commerce websites or museum websites or government websites; one of DMU’s web-based systems for students or staff; electronic devices such as remote controls for televisions or DVD players, or digital cameras, or car radios; or control panels for appliances such as microwave ovens or home heating systems; or a self-service system such as an automatic ticket vending machine. You may, if you wish, choose to evaluate two very similar and directly competing products, and assess ways in which one is superior to the other.

It’s perfectly okay to decide to evaluate a part of a big or complicated system, or consider a limited set of use cases. When in doubt, go into more detail about less of the system.

The one piece of advice we can give is to choose something that is complicated or difficult to use, or is used to carry out complicated tasks, and preferably has obvious usability problems. Studying more complicated and less frequently used features of a system is likely to be more fruitful than focusing on the standard functions people use all the time. Standard features of highly optimised systems that large numbers of people use, like Amazon, don’t make for interesting evaluations.

You may choose to interpret ‘interactive system’ very broadly and present a usability evaluation of a static information display, but this would require a sophisticated and detailed analysis of how people use it for practical tasks, and these tasks would need to be complicated enough to give you something to analyse. Ask advice if you consider this.