Bespoke vs Off-the-shelf – watching the BBC blogs comment problems
Those with long memories might remember that an occasional topic I’ve hovered round, is the area of blogs and the BBC. It’s something I’ve taken a great interest in for many years – going back as far as something like 2002 when, tucked away in Bush House, a few people started getting really excited about the concept of the BBC blogging on what was then, something that was nowhere near as big as it is today.
When they finally did start happening, there I was, watching excitedly. Indeed at one point there was going to be a new job role created for rolling out blogs – I couldn’t have got my application in fast enough. If it had happened anyway. As I recall, it got stuck by budget issues and was staffed in different ways. Anyway…
Blogs for me are a great way for the BBC to communicate with the people who use, and pay for, its services and it’s great that they’ve been a success. Indeed, probably too much of a success if the continuing comment problems are anything to go by. The problems in trying to put up a single comment are, frankly, terrible. Timeouts… Server problems… You’re not even sure if your comment has even got through to the server backend half the time.
Over on the BBC Internet Blog, Executive Producer and all round nice bloke, Jem Stone, has written a post about forthcoming changes that they’re implementing to the comment system to try and make sure people can actually have their say. It’s currently being trialed on the PM Blog, and uses the BBC’s message board system, DNA.
This has lead to a question on one of the comments of the BBC Internet Blog asking this question:
Honest question – why are you reinventing the wheel to begin with?
Is there any particular reason why the BBC seems institutionally biased against using existing proven open source solutions?
I found this an interesting point – but not quite in the way most people would expect.
As you may know, for many years I worked on the BBC’s then message board software, Howerd 2 – a bespoke message board solution which replaced a previous bespoke message board system, and which was itself replaced by a bespoke message board system.
And every now and then, when things went wrong, certain members of the public would ask “why are you re-inventing the wheel? What’s wrong with existing proven solutions?”
The answer was always simple – scale. Existing solutions don’t always scale to the demands of the BBC.
For example, there are currently 32 BBC message boards, with around 200 sub-topics. Millions of pages are viewed each day, and millions of posts made every month (including an army of inevitable attempts at spamming).
There actually used to be more boards – around 80 at one point. And of course that included the infamous 606 which at one point, was accounting for about half the posts on the entire message board network, and causing severe problems for the rest of the boards due to timeouts, and server errors.
Of course what the user saw was a bespoke system failing. They were being told that off-the-shelf solutions didn’t scale but saw a bespoke system not scaling as well.
If the user-facing problems of the BBC message boards sound familiar, that’s actually because the result the user sees on both the message boards and the blogs is the same. What’s the difference?
The difference is actually in the approach.
For its blogs, the BBC didn’t use a bespoke solution. Instead it adopted an off the shelf solution in the form of Movable Type (indeed the first trial blog was, if I remember correctly, built using Typepad!) It’s an approach other large-scale sites used – notably The Guardian’s Comment is Free (which seems often to be a case of “Comment is full of raving madmen.)
By using a standard, off the shelf solution, the BBC did what many would have wanted – stop wasting time building your own stuff when others have already done it.
The trouble is, it doesn’t always scale well. And this time, the evidence is before our eyes. We the public can see an off-the-shelf solution in-situ. And it’s struggling.
Fact is that relatively few websites ever have the needs and demands put upon them, that the BBC does. Right now the BBC has 50 odd blogs running on its blog installation. I don’t know the way it’s set up behind the scenes, but when any blogging software is being designed, how often is it designed to cope with such numbers?
Now of course, publishing blogs in that kind of number is fine – publishing doesn’t take up that much in the way of resources. But when you look at the scale of comments that that number would generate on a site the size of the BBC’s, then you enter a world where problems are liable to appear. And that’s before we even approach the inevitable world of comment spam.
On my own blog, comment spam has outweighed genuine comment dramatically. Before I recently installed Spam Firewall, I’d estimate that that for every genuine comment, at least 100 spam comments were getting through to me. And that’s not even counting the attempts that didn’t succeed. And this site is frankly small fry. Scale that up to the BBC’s size and you inevitably enter a different world of battle.
Some of the BBC’s problems are inevitably the result of running rather elderly copies of Movable Type due to managing to block the upgrade path due to customisations made behind the scenes. It’s a harsh lesson but one to learn from.
Now in all this, I’m not trying to say that bespoke good, off-the-shelf bad. Both have their benefits, and both have their weaknesses and one thing that needs to be done is to evaluate them both.
What I think is most interesting about the current BBC Blog solution, is actually it straddles both worlds. The PM blog itself is still posted via Movable Type (and indeed the old comments posted via MT still appear). If it works and works well, why build something new to do that? And if they can get back on the upgrade path, they could be on the open source path too. The bespoke part of the upgrade comes only with the comments – a bespoke path followed before and that is known to work, and should hold the site in good stead for the inevitable growth in usage as the BBC blogs network grows.
Andrew is now awaiting the inevitable comment to be made that actually, they’re replacing Movable Type as the publishing backend! And he’s also most impressed with his computer’s sympathy for those people struggling to make comments on the BBC’s blogs – whilst writing this post, his PC decided to freeze dramatically!