Design
Evaluation
User Recruitment and Characteristics
We personally approached our recruited users and asked about their interest to a test user for WebAnnotator. All three of them agreed immediately so we did not have to look for a 4th candidate.
The first user is a CS researcher at MIT. The other two are undergraduates of different age and do not major in CS. We believe that they well represent our user population (recall: students and researchers). Moreover, Our users are quite diverse as their academic backgrounds are different.
We also asked our users about their previous experience with annotating document. All of them have read annotated document before but have rarely annotated themselves. The advantage of this is that we could better evaluate the learnability of our interface because of their limited experience.
Nonetheless, our users may lack the diversity in age as they are all under 25.
Procedure
Our users were asked to perform the following tasks. A short briefing of what WebAnnotator achieves was given at the beginning. However, no information about the interface was mentioned. We also encouraged them to speak up about their thought process. No demo was performed.
We helped our users at time when they were stuck. Fortunately, this did not happen frequently.
Tasks
- Annotate a wikipedia page on your favorite topic. In WebAnnotator:
- Open the page for annotation.
- Highlight, underline and add notes to pieces of text in the page.
- Delete an annotation you have just made.
- Suppose you have annotated other webpages before using WebAnnotator.
- Find all your saved annotated pages. (Note: about 30 saved pages were created beforehand in order to test the search box)
- Reopen the one you just annotated.
- Open the one named Project 1: An ABC Music Player.
- Share your page with your best friend.
- Suppose the page is already shared with some people:
- Remove one of them
- Change one user's access rights from "can edit" to view-only.
- Add your best friend in.
- Suppose the page is already shared with some people:
Observation
User 1 (CS Researcher)
- Task 1
- After highlighting, did not notice the popup annotation buttons at first. Was looking for a tool bar at the top and searching throughout the webpage.
- Adding multiple annotations to the same piece of text seemed difficult.
- Was looking for a "save" button before noticing the message "All changes saved" at the top.
- The task was successfully completed otherwise.
- Task 2
- Did not realize that the search box implements only prefix search. Entered keywords like "abc" and "player".
- Located the entry for "Project 1: An ABC Music Player" by manually searching through all the entries.
- Had some difficulty to go back to the dashboard from an annotated page. Did not remember that the button "Saved Pages" would bring her back to the dashboard. Recommended changing the wording to "All pages".
- The task was successfully completed otherwise.
- Task 3
- In the dialog box, the "plus" button was not pressed before clicking "done".
- The task was successfully completed otherwise.
User 2 (Math Undergraduate)
- Task 1
- Task 2
- Task 3
User 3 (Mechanical Engineering Undergraduate)
- Task 1
- Task 2
- Task 3
Interestingly, all three of our users chose a Wikipedia page, which was not mentioned in any verbal or written description throughout the testing. This may indicate that Wikipedia articles are among the most popular webpages to be annotated and more effort should be geared to make our interface more usable for Wikipedia articles.