Week 9: More Usability Testing

I decided to work through spring break so that I could finish up this project sooner and have more time to work on my poster.  This week, I conducted two more usability tests, this time with internal OT employees who have a lot of experience with another content management system but have only taken the training for WEM (our team is actually in the process of setting up a new instance of WEM so that they can start using it for part of the corporate website).  These are people I have worked with a little, remotely, and only met once in December so it was still comfortable because they weren't total strangers like my next round of participants. During the first test, my WebEx account wasn't working so I got to experience troubleshooting technical difficulties on my feet. Thankfully, I had a back up plan and we used the free screen sharing tool join.me.

One of the more interesting observations from my first participant this week was his confusion about "where he was in the system" when he added a new content item and then again when I asked him to add a new folder. It was almost like the OS metaphor went too far because he thought he was somewhere within the folder structure when actually he was outside of it, like when you're saving a new Word document and you have to choose where to save it. I wonder if in the CMS the Marketing team is currently using, they can only create content while inside a folder. You certainly have that option in WEM too, but it's not required.  He also really wanted the add folder (and add channel) options to be available from contextual right-click menus. He really did not like the button placement for these options. Based on this feedback, I do think the add folder and add channel buttons should always be active (right now the buttons become enabled only if the user has highlighted an existing folder or channel, and so the new container would be created as a child). Since there is an option to choose the placement of the new container from within the creation screen, I see no reason why the add option should not always be available; this would also allow users to be able to create new containers at the root level.

Nothing terribly exciting happened this week and I don't  have any new designs or demonstrations, so instead I leave you with a song.  I first heard this three years ago during my first semester of grad school and I listened to it over and over on the Friday during spring break as I tried to write a paper for the Understanding and Serving Users course. I hope you enjoy Okkervil River, "Unless It's Kicks."

Week 8: Usability Testing Begins

It's amazing, really, how much work goes into usability testing. Each test takes about an hour but there are many more hours that go into finding participants, prepping materials, testing technologies, reviewing the test afterward and evaluating findings just to name a few things. This week, I spent some time further refining the test plan and prototype. I realized that when a participant clicks through the folder grid, the folder tree should update at the same time; it's funny the things you overlook when you have been super involved in a project. I think this is part of the reason why developers don't normally do the testing too—they are too close to the project. I think I am definitely missing things and probably biasing my test results just due to the fact that I also created the prototype.

The UX team has a very nice spreadsheet it uses when conducting usability studies that I had to spend quite a bit of time adjusting for my needs, filling it out with my scenario and tasks. It has a macro that allows the facilitator to easily record time on task, which was helpful. It was also easier to take notes in a spreadsheet while also administering a test than I thought it would be. To that end, I conducting my first two tests this week and they both went well and took the right amount of time. There was only one failure of one task and I think it was due more to issues with the prototype than of the system design; the user just didn't notice that the squares I was using to indicate checkboxes were checkboxes. Once I pointed this out, she had no problem completing the task.  So I'm not sure if that really counts as a task failure or just a prototype failure.

These are supposed to be checkboxes, but they are a little too large!

On the plus side, both testers used the folder grid instead of the folder tree to drill down and select a content item; this is one of the major changes I'm suggesting to the interface so it was great to see people default to using it.

It was nice to start my testing with iSchool students and to be able to do in-person testing to work out more kinks, practice running through the tasks, and get better at reading everything aloud then asking questions. (It made me wonder after the fact if was supposed to get IRB approval but since these tests are for a company and not really UT, I think I'll be okay.)  The rest of my tests will be administered remotely, which has all kinds of potential for problems.  I ran though a technology test to try out Webex and see what kind of lag I was getting.  I think it worked out okay. I've included a short video of how the test will appear to me while the remote user has control of my screen. The audio quality is pretty terrible so I will avoid using my cellphone for these tests.




Next week will be more testing!

Week 7: Usability Test Pilot

This week I have spent my time preparing to begin usability testing.  I created several more screens for the prototype earlier in the week and polished the testing task list after meeting with Robin and Tanya to get some feedback.  Wednesday, I conducted a pilot test in person but also shared it over Live Meeting to another laptop so that I could gauge quality and response times.  The unfortunate thing is that since it's video streaming live over the Internet, there is some lag, so even if I am allowed to record my remote testing sessions, they won't pick up everything.  I have some concerns about being both facilitator and note taker during these tests but it will be good experience.  Below is an excerpt from the pilot test demonstrating what I thought was the most flawed part of my prototype:


Overall the pilot went very, very well. It look just under 50 minutes and the participant was able to complete all tasks. I got some great feedback about changing some of the wording and breaking up one of the tasks into two tasks. I found some broken hotspots within the prototype as well. Most importantly, I discovered an entire interaction missing from the prototype, as demonstrated by the video. I went back and created a way for participants to be able to use either the folder tree or the folder grid when attempting to select a single content item. Since I proposed to allow users to navigate folders and channels using the folder grid, that needs to be functional from the beginning screen of this task—it will be valuable to see how many users choose to use the tree versus the grid upon first being presented with this picker layout. I can't believe I almost didn't include that! (Participants could hide the tree in order to use the grid as I explain in the video, but that will not give me the testing data I need in the most accurate way.)

My prototype is up to 116 screens :/ I just didn't realize the differences in doing true interaction design for a piece of software versus wireframing for a website—I'm learning that these are two unique tasks with their own separate needs so I am glad for the experience. I still might make another tweak to my overall design again based on the pilot. I haven't decided whether a single click to a folder in the tree should display the contents of that folder in the grid, or if that should remain a double-click.

I've scheduled most of my test participants over the coming weeks, starting with two iSchool students next week followed by two OpenText employees the week after. Following that initial round of testing, I will make adjustments to the test plan and prototype before conducting tests with real OpenText customers the last week of March.