Working Together: User Testing

By August last year, wireframes for The Carers’ Centre website were complete, and it looked like we had achieved the journeys we wanted for users through the site. There was only one thing for it — to do some testing! Janine and Laura reflect on what we did:

Janine: On our journey to become more user-centred, we’ve learned the value of constantly going back to carers, and other stakeholders, to check what they think. And to do this as early as possible in the process. I guess this is the ‘fail fast’ approach of agile working. If we’ve got it wrong, I’d rather know after a month of workshops and a day of designing wireframes, than 2 months of effort (and cost!) of our digital partner designing a site we then have to amend (or, worst case, scrap entirely) to deliver what carers need. This was one of the factors which was key when choosing a digital partner, and Digital Wonderlab’s approach to testing wireframes with users as early as possible was exactly what we were looking for.

The types of user testing & when to use

Laura: There are a couple of ways to approach user testing. If you have a pool of people you can reach out to, to take part, this is a great way of saving on recruitment fees. However there maybe times that you can’t access a key audience group and you will need recruiters to seek this audience for you. In the case of The Carers’ Centre we used a mix of the two. This gave us the opportunity to speak with people who had no knowledge of The Carers’ Centre and see if the proposed navigation would make sense to someone who did not know the charity and their offer.

When it comes to the testing sessions, there are a few approaches you can take. Firstly, you can do quick rounds which take about 30 minutes. You can focus on a key journey, for example in this case for the adult carers we tested the core journey of the sign-up form, explored whether they could find the form and how easily and quickly they could complete it. Secondly there are longer sessions which are usually about an hour where we explore Several journeys through the site to ensure secondary journeys also work for the different audience groups and you haven’t missed anything.

The challenge of organisation

Janine: When we were ready to test, and clear on how to test, we set about getting our testers. We were looking for a handful of people in the main user groups we were working with: adult carers; young carers; friends and family of carers and professionals working with carers. We wanted to speak with approximately 20 people in total, with a carer base of 3000 this should be a piece of cake, right? It turns out recruiting and then scheduling people in for user testing was almost as much of a skill as undertaking the testing itself!

It seems obvious in hindsight, but why would anyone want to test a prototype? What’s in it for them? We had thought a quick email to our core carer group would result in 4 to 5 people from each group stepping up, but that wasn’t the case. We needed an incentive. The offer of a small voucher to say thank you helped, and we definitely learned you must make sure there is something in it for a tester. It doesn’t need to be financial, but they do need to get value from the experience.

With enough people willing to test we were halfway there, but then came the challenge of fitting everyone in to testing slots on the days we had allocated. Which proved a logistical nightmare! We didn’t think to use a tool to enable us to do this, so it involved a lot of emailing back and forth to fit people in, change slots around and make sure everyone who wanted to test could do. It was a massive time sink — worth it, but in future we would use an online scheduler to enable people to choose a time which worked for them. Why didn’t we think of that at the start?!

Scheduling user interviews in advance, using tools that support this, is learning we’ll take forward for next time!

The skill of the interview

Laura: Planning is key for user testing. You need to clearly set out the objectives of the testing prior to the user testing sessions so you know what you need to explore. Ask the questions ‘What do we want to understand?’, ‘What are the key tasks we want to explore with this audience group?’. The sessions can go so quickly that if you have not planned, and completed a pilot test of the session with a member of the team, you will not know if you have covered everything that you need to.

Our approach at the start of the session is to always find out a bit about the person you are testing, it helps to get them settled and makes it feel less like a test. We will also try and take information they have provided about themselves and then create a scenario around this which relates to them so they can naturally use the prototype and see where this takes them. This helps to explore the language you have for the navigation and any messaging you want to explore on the prototype. Once a user has completed this task we then conduct follow up tasks to make sure we have explored and answered the questions we set out in planning. Toward the end of the session we leave just enough time to wrap up and ask any final questions. At this point you have an opportunity to cover any final queries to make sure you have answered all the questions set out in planning.

Janine: It was such a learning experience for us at The Carers’ Centre to be able to observe the user interviews. We did this both in person and online, using Zoom to watch but not interfere with what was going on. Not only did we learn a lot about the site, we also learned how to improve the way we ask for feedback both on the website moving forwards, and other services and products we develop!

Previously our approach to involvement and feedback has been to explain what we’re trying to do and ask for people’s views. Generally this is in a group setting and is a conversation between us and users. Watching Laura, it was clear how much bias this can introduce. Our introduction can guide their view as to what feedback we are looking for. A group conversation can also mean some people are not willing to share their initial thoughts and feelings.

Laura just let people use the website without telling them much at all, letting them ask questions. She listened and responded with her hallmark response ‘What do you think?’ It really was eye opening, because this is exactly what you need to know. You are not going to be in the room with every user telling them how to engage with a site, or product or service. They need to understand what they need to do, or where they need to go, to get the help they need without you.

The feedback was varied, and rich, really helping us understand whether we’d created something that people who didn’t know who we were would want to, and be able to, use. And I am so much more confident in this feedback because it felt led by the users, not led by us.

Listen for the unexpected

One key thing which came out of this new way of gaining feedback was how much easier it is to hear the unexpected, or for feedback to be given on things you hadn’t thought about. We were able to undertake interviews with people who had never heard of The Carers’ Centre before. We were hoping they’d show us if they could find certain bits of information on the site and tell us what they’d expect to find. Yet the best feedback we got was on our logo! This wasn’t something we thought we’d be testing. Yet most people looked at our logo and asked ‘So, is this part of the Council?’ Laura’s famous phrase ‘What do you think?’ came in to play and the feedback was that people weren’t sure. As we had the words ‘Bath & North East Somerset’ under the logo, which they felt was linked to the Council.

Why was this important? It’s not that there is anything wrong with being link to the Council and in fact they do provide funding for some of what we do. But we are an independent charity. We want to be seen as such, not least so we can raise vital funds through donations.

We hadn’t been asking for feedback on this aspect, but thanks to the user focussed approach of the interviews, there was space for unexpected results like this which have been vital in helping us consider how the site needs to both function and look. Can you spot the change to our logo on the new site as a result?!

Final thoughts

User testing takes time (and therefore money). Our experience is that this is money very well spent, and we hope to make user testing a key part of our ongoing work to develop the website. We would have loved to do more, but it does cost. I could have spent at least twice as long hearing from users about what they thought! It’s a balancing act, of doing enough to hear key themes against the budget or resource that’s available. Having started this process, we are keen to make sure we plan user testing at key points in the future, as well as having other ongoing feedback mechanisms, to keep our site as user-centred as it can be.

Laura: It was great that we were able to conduct so much testing with this project, I love testing as it removes the ‘I thinks’ with ‘We observed in testing that users need this’. It removes opinions and assumptions and replaces them for facts. As Janine mentioned it does take time and money so if you can only do one round of testing before building your site that’s ok, the insight you will gain just by speaking to a few of your key audience is invaluable and will save you money in the long run. It will also give you confidence that you are building a site that your users will want to use.