When technology goes bad – traumas during recent user testing.

Published on 3 March 2010 in BBC, Work, BBC, BBC iPlayer, Freesat, usability, work

One of the things I’m always keen to do on large work projects is a bit of user testing – where we get real people to come in, try using our services and see what happens. The idea is to see what works, and more importantly, what doesn’t.

We do sometimes use dedicated testing companies for this work, however the BBC does have a small team of usability gurus and there’s a testing lab on site. The main room is decked out to look a bit like a living room, although the video cameras kind of spoil the effect. Then there’s a viewing room where team members can watch proceedings.

For TV based services we normally test using prototypes built in Flash. The prototype runs off a laptop connected to a TV, and an infra-red sensor allows the testee to use a normal remote control to control the service. However the preference is to test real services running real code if we can. That takes serious development time, however sometimes its possible, and the lab has a TV signal for such times.

Recently we planned a round of testing for BBC iPlayer on Freesat. The plan was to test some changes to the user interface, however whilst looking minor they could have a big impact on the way people use the service. Because the user interface already existed, it was quicker to build some set top box versions than it would have been to build in Flash, and we’d have the benefit that we’d have full access to the live data and video.

We had three variants to test, two days booked in the lab and ten people lined up to try it out.

The previous week we’d dutifully gone into the lab to run a test session – check the satellite and broadband connections were all working correctly – that kind of thing. They were and all we needed to do was wait until Tuesday morning when the tests would kick off.

Early on Tuesday I arrived to get the set top box set up. And that’s when the trouble began.

Over the weekend there had been electrical works in the building, and as such part of the lab was in disarray. A frantic half hour was spent by the person who knows how the lab works, just trying to get the camera working properly again.

Then we had internet connection problems. The BBC network runs everything through proxy servers for internet connections, however set top boxes don’t usually offer you the ability to set a web proxy so to run services we need what we call a “dirty connection” – raw access to the internet.

The lab’s dirty connection was on a network which had its own DNS server. The power works had seen the server shutdown, and unfortunately when the server reboots, it hadn’t rebooted properly, thus requiring manual intervention.

With minutes to spare, we got everything set up and ready for the first test. Phew. We’d made it.

Halfway through the first test, the internet connection failed again. Not the DNS server this time – that was still running fine. After much struggling, we had to abandon the first test.

Much scratching of heads, I eventually stumbled on what looked like the problem – for some reason, the set top box didn’t like the particular ethernet cable or socket we were using. We changed to a different cable and a different socket and got everything working perfectly and we sighed a huge sigh of relief that test two wouldn’t need to be cancelled as well.

Then mid way through test two we got another problem – the box lost connection to our development server where the prototypes were being hosted. We could get through to other servers, however the development one had disappeared. Before randomly coming back a few minutes later. It was a problem that would plague us throughout the day although thankfully didn’t cause any major problems.

Quite what the problem was, we never did find out – the server in question is a standard BBC development server that our team were using next door without a problem. The problem could have been anywhere. Still, it was something that could be worked round.

Day two arrived and once more we had no connection to the internet, and the lovely bloke who knew how everything worked had the day off. I’m therefore sure he was very happy to spend half an hour talking me through diagnosing the problem and working out how to get the server up and running again. It looked liked the power work had disrupted everything once more.

The time taken to sort it all out meant we’d had to cancel the first session of the day. This was not good as arranging the sessions costs money and we get nothing useful out of a cancelled session! However we were up and running once more for the second of the day.

Mid way through the second one, the connection disappeared once more. The DNS server was running, however it wasn’t doing any DNS look ups. The exact problem could have been anything. We were stuck and admitted defeat. We’d need to find another room. We wouldn’t have the viewing room and cameras, however we’d at least be able to test.

The first problem was finding a room. We needed a meeting room with a satellite feed and a dirty internet connection. And preferably a sofa. Such rooms in the BBC are hard to find. We came up with three options.

There was the Blue Room – the BBC’s gadget demo space, which has all sorts of cool stuff in it, including a Freesat box. Unfortunately it was booked up all afternoon being used for an important demo.

Our next thought went to Ada Lovelace which has TVs, comfy chairs and the all important broadband link. It took was booked up all afternoon with a large meeting…

Finally the Board Room was considered. No comfy chairs, however we were desperate. Ah. No. In use all afternoon.

In the end we managed to commandeer part of an informal meeting area on the third floor, owned by our colleagues in FM&T: Journalism. It had a TV feed; it had a net connection. We knew there may be some small meetings going on in the background, however it was better than nothing. Beggars can’t be choosers.

I now had about 90 minutes to rig up one of our boxes and get it working before the third test. A doddle I thought!

I dutifully crowbarred our Humax PVR set top box in to the rig. Only to find it had no signal and wouldn’t show a picture. Wondering if it was just the box being stroppy, I ran upstairs and borrowed a different – a Humax HD box this time. I got downstairs again, saw the box had a signal but then I found the box was tuned in to our development environment and needed a retune. Okay I thought. Retune.

First part of the retune process – enter your postcode. I did. Postcode invalid I was told. I entered another. Invalid. I entered my parents postcode. Invalid.

Sighing big style, I went back upstairs and plugged the box in at my desk where it retuned perfectly and worked fine. I took it downstairs and plugged it in. No signal. Excactly the same problem as the PVR box.

I eventually deduced that, for some reason, the satellite feed going through to the third floor wasn’t carrying the “Freesat Home Transponder”. This is the data area which contains the postcode tables and EPG data, and clearly what the set top boxes used to determine if there was a signal. Without access to the Home Transponder, the Humax boxes wouldn’t work. Clearly the satellite signal distribution system wasn’t providing a full signal to the third floor. As the room hadn’t got a Freesat box, no one had ever spotted the problem. No doubt fixable, however not in the time we had available!

Clearly using our Humax boxes wasn’t an option however I wondered if another make of box would work. Groping around the office I found a Technisat set top box running a development version of their firmware. Thus it may feature an odd quirk or two, however it was worth a shot.

Thankfully the Technisat box didn’t require the presence of the Home Transponder so turned on fine. Now all we had to do was check the internet connection and we were home dry.

The connection worked.

Result! Ten minutes to spare before our third test of the day and we were up and running. The test was conducted with me lurking in the background hoping nothing else would break. At one point the box locked up (such things happen with pre-production firmware) however a swift reboot later and it was running once more.

Breathing a huge sigh of relief (once more) I left, leaving the fourth test to start.

Half way through the fourth test, the satellite signal apparently failed several times. When the satellite signal fails, the set top box shuts down everything and says “No signal”. The fourth test had to be abandoned.

That left one final test which managed to be completed thanks to some very lucky failures. We were testing three variants, and, by all accounts, the signal failed just as they’d finished testing each variant. It was amazingly lucky timing and meant that at least we’d got the final test completed. It was over. I packed up the set top box and restored the room’s equipment as I found it.

We did at least get something out of it – it wasn’t as much a disaster as originally feared. Experience has shown me that during user testing, something will go wrong. However it was a stunning example of those scenarios where if something could go wrong, it would to stunning effect!