The evaluator effect in usability studies: Problem detection and severity judgments

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Standard

The evaluator effect in usability studies: Problem detection and severity judgments. / Jacobsen, Niels Ebbe; Hertzum, Morten; John, Bonnie E.

Proceedings of the HFES 42nd Annual Meeting. Santa Monica, CA : Human Factors and Ergonomics Society, 1998. p. 1336-1340.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Harvard

Jacobsen, NE, Hertzum, M & John, BE 1998, The evaluator effect in usability studies: Problem detection and severity judgments. in Proceedings of the HFES 42nd Annual Meeting. Human Factors and Ergonomics Society, Santa Monica, CA, pp. 1336-1340.

APA

Jacobsen, N. E., Hertzum, M., & John, B. E. (1998). The evaluator effect in usability studies: Problem detection and severity judgments. In Proceedings of the HFES 42nd Annual Meeting (pp. 1336-1340). Human Factors and Ergonomics Society.

Vancouver

Jacobsen NE, Hertzum M, John BE. The evaluator effect in usability studies: Problem detection and severity judgments. In Proceedings of the HFES 42nd Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. 1998. p. 1336-1340

Author

Jacobsen, Niels Ebbe ; Hertzum, Morten ; John, Bonnie E. / The evaluator effect in usability studies: Problem detection and severity judgments. Proceedings of the HFES 42nd Annual Meeting. Santa Monica, CA : Human Factors and Ergonomics Society, 1998. pp. 1336-1340

Bibtex

@inproceedings{dcd060e520624bd59acbc7d7cca77147,
title = "The evaluator effect in usability studies: Problem detection and severity judgments",
abstract = "Usability studies are commonly used in industry and applied in research as a yardstick for other usability evaluation methods. Though usability studies have been studied extensively, one potential threat to their reliability has been left virtually untouched: the evaluator effect. In this study, four evaluators individually analyzed four videotaped usability test sessions. Only 20% of the 93 detected problems were detected by all evaluators, and 46% were detected by only a single evaluator. From the total set of 93 problems the evaluators individually selected the ten problems they considered most severe. None of the selected severe problems appeared on all four evaluators{\textquoteright} top-10 lists, and 4 of the 11 problems that were considered severe by more than one evaluator were only detected by one or two evaluators. Thus, both detection of usability problems and selection of the most severe problems are subject to considerable individual variability.",
author = "Jacobsen, {Niels Ebbe} and Morten Hertzum and John, {Bonnie E.}",
year = "1998",
language = "English",
pages = "1336--1340",
booktitle = "Proceedings of the HFES 42nd Annual Meeting",
publisher = "Human Factors and Ergonomics Society",

}

RIS

TY - GEN

T1 - The evaluator effect in usability studies: Problem detection and severity judgments

AU - Jacobsen, Niels Ebbe

AU - Hertzum, Morten

AU - John, Bonnie E.

PY - 1998

Y1 - 1998

N2 - Usability studies are commonly used in industry and applied in research as a yardstick for other usability evaluation methods. Though usability studies have been studied extensively, one potential threat to their reliability has been left virtually untouched: the evaluator effect. In this study, four evaluators individually analyzed four videotaped usability test sessions. Only 20% of the 93 detected problems were detected by all evaluators, and 46% were detected by only a single evaluator. From the total set of 93 problems the evaluators individually selected the ten problems they considered most severe. None of the selected severe problems appeared on all four evaluators’ top-10 lists, and 4 of the 11 problems that were considered severe by more than one evaluator were only detected by one or two evaluators. Thus, both detection of usability problems and selection of the most severe problems are subject to considerable individual variability.

AB - Usability studies are commonly used in industry and applied in research as a yardstick for other usability evaluation methods. Though usability studies have been studied extensively, one potential threat to their reliability has been left virtually untouched: the evaluator effect. In this study, four evaluators individually analyzed four videotaped usability test sessions. Only 20% of the 93 detected problems were detected by all evaluators, and 46% were detected by only a single evaluator. From the total set of 93 problems the evaluators individually selected the ten problems they considered most severe. None of the selected severe problems appeared on all four evaluators’ top-10 lists, and 4 of the 11 problems that were considered severe by more than one evaluator were only detected by one or two evaluators. Thus, both detection of usability problems and selection of the most severe problems are subject to considerable individual variability.

M3 - Article in proceedings

SP - 1336

EP - 1340

BT - Proceedings of the HFES 42nd Annual Meeting

PB - Human Factors and Ergonomics Society

CY - Santa Monica, CA

ER -

ID: 113158240