Bespoke vs Off-the-shelf – watching the BBC blogs comment problems

Published on 26 February 2008 in , , , , , ,

Those with long memories might remember that an occasional topic I’ve hovered round, is the area of blogs and the BBC. It’s something I’ve taken a great interest in for many years – going back as far as something like 2002 when, tucked away in Bush House, a few people started getting really excited about the concept of the BBC blogging on what was then, something that was nowhere near as big as it is today.

When they finally did start happening, there I was, watching excitedly. Indeed at one point there was going to be a new job role created for rolling out blogs – I couldn’t have got my application in fast enough. If it had happened anyway. As I recall, it got stuck by budget issues and was staffed in different ways. Anyway…

Blogs for me are a great way for the BBC to communicate with the people who use, and pay for, its services and it’s great that they’ve been a success. Indeed, probably too much of a success if the continuing comment problems are anything to go by. The problems in trying to put up a single comment are, frankly, terrible. Timeouts… Server problems… You’re not even sure if your comment has even got through to the server backend half the time.

Over on the BBC Internet Blog, Executive Producer and all round nice bloke, Jem Stone, has written a post about forthcoming changes that they’re implementing to the comment system to try and make sure people can actually have their say. It’s currently being trialed on the PM Blog, and uses the BBC’s message board system, DNA.

This has lead to a question on one of the comments of the BBC Internet Blog asking this question:

Honest question – why are you reinventing the wheel to begin with?

Is there any particular reason why the BBC seems institutionally biased against using existing proven open source solutions?

An 800lb Gorilla In The Room’s comment on the BBC Internet Blog

I found this an interesting point – but not quite in the way most people would expect.

As you may know, for many years I worked on the BBC’s then message board software, Howerd 2 – a bespoke message board solution which replaced a previous bespoke message board system, and which was itself replaced by a bespoke message board system.

And every now and then, when things went wrong, certain members of the public would ask “why are you re-inventing the wheel? What’s wrong with existing proven solutions?”

The answer was always simple – scale. Existing solutions don’t always scale to the demands of the BBC.

For example, there are currently 32 BBC message boards, with around 200 sub-topics. Millions of pages are viewed each day, and millions of posts made every month (including an army of inevitable attempts at spamming).

There actually used to be more boards – around 80 at one point. And of course that included the infamous 606 which at one point, was accounting for about half the posts on the entire message board network, and causing severe problems for the rest of the boards due to timeouts, and server errors.

Of course what the user saw was a bespoke system failing. They were being told that off-the-shelf solutions didn’t scale but saw a bespoke system not scaling as well.

If the user-facing problems of the BBC message boards sound familiar, that’s actually because the result the user sees on both the message boards and the blogs is the same. What’s the difference?

The difference is actually in the approach.

For its blogs, the BBC didn’t use a bespoke solution. Instead it adopted an off the shelf solution in the form of Movable Type (indeed the first trial blog was, if I remember correctly, built using Typepad!) It’s an approach other large-scale sites used – notably The Guardian’s Comment is Free (which seems often to be a case of “Comment is full of raving madmen.)

By using a standard, off the shelf solution, the BBC did what many would have wanted – stop wasting time building your own stuff when others have already done it.

The trouble is, it doesn’t always scale well. And this time, the evidence is before our eyes. We the public can see an off-the-shelf solution in-situ. And it’s struggling.

Fact is that relatively few websites ever have the needs and demands put upon them, that the BBC does. Right now the BBC has 50 odd blogs running on its blog installation. I don’t know the way it’s set up behind the scenes, but when any blogging software is being designed, how often is it designed to cope with such numbers?

Now of course, publishing blogs in that kind of number is fine – publishing doesn’t take up that much in the way of resources. But when you look at the scale of comments that that number would generate on a site the size of the BBC’s, then you enter a world where problems are liable to appear. And that’s before we even approach the inevitable world of comment spam.

On my own blog, comment spam has outweighed genuine comment dramatically. Before I recently installed Spam Firewall, I’d estimate that that for every genuine comment, at least 100 spam comments were getting through to me. And that’s not even counting the attempts that didn’t succeed. And this site is frankly small fry. Scale that up to the BBC’s size and you inevitably enter a different world of battle.

Some of the BBC’s problems are inevitably the result of running rather elderly copies of Movable Type due to managing to block the upgrade path due to customisations made behind the scenes. It’s a harsh lesson but one to learn from.

Now in all this, I’m not trying to say that bespoke good, off-the-shelf bad. Both have their benefits, and both have their weaknesses and one thing that needs to be done is to evaluate them both.

What I think is most interesting about the current BBC Blog solution, is actually it straddles both worlds. The PM blog itself is still posted via Movable Type (and indeed the old comments posted via MT still appear). If it works and works well, why build something new to do that? And if they can get back on the upgrade path, they could be on the open source path too. The bespoke part of the upgrade comes only with the comments – a bespoke path followed before and that is known to work, and should hold the site in good stead for the inevitable growth in usage as the BBC blogs network grows.

Andrew is now awaiting the inevitable comment to be made that actually, they’re replacing Movable Type as the publishing backend! And he’s also most impressed with his computer’s sympathy for those people struggling to make comments on the BBC’s blogs – whilst writing this post, his PC decided to freeze dramatically!


  • Hi Andrew
    Thanks for that. You’re right. As well as replacing the comments bit of MT with the in house platform; DNA (for scale/spam reasons) we’re upgrading MT3.2 to MT4.1 (and so on) and getting back on the upgrade path as you suggest. The licence we’re using is MT’s rather confusing mix of MTOS and a commercial licence so whether you can call it open source or not i’m unsure.
    Coming soon to rest of BBC blogs.
    Thanks for calling me a nice bloke 🙂

  • Matt says:

    “The answer was always simple – scale. Existing solutions don’t always scale to the demands of the BBC. ”
    “I don’t know the way it’s set up behind the scenes, but when any blogging software is being designed, how often is it designed to cope with such numbers? ” last month handled 4.2 million new blog posts and pages, 4.7 million legit comments, and Akismet blocked 378,341,497 spams. If your needs are smaller than that, perhaps we should talk. We’re doing blogs for the NY Times, CNN, Time, Wall Street Journal, Fox News, People Magazine, and more. These are solved problems, and you can use a 100% Open Source software stack to do it (as we do).

  • Dan says:

    Must agree with Matt. The BBC’s blogging requirements are pretty big, but not as big as what WordPress already demonstratably copes easily with, without constantly grinding to a halt and requiring regular “firefighting”.
    It seems to come down to that, when the BBC wanted a blogging platform, it wanted to pay for one rather than use a free one. “If it’s free it must be bad, right? Or they’d be charging for it? And if something costs a lot of money, it must be good…”
    Today I’m sure everyone at the BBC realises that that was wrong, but for some reason don’t want to bite the bullet and move over to WordPress.

  • Andrew Bowden says:

    Well it’s not my call what blogging software the BBC uses, but I should say one reason why WordPress would be a problem, and that’s because the BBC’s live webservers don’t currently allow PHP (although that’s due to change, or so I am told).
    Whilst outwardly things like this can look simple (i.e. why use X when you can use Y), there’s often a lot that goes on behind the scenes which make “simple” decisions a lot more complicated.
    Also do remember, that whilst the BBC’s current blogging requirements are big now, what will they be in the future. Can WordPress scale to whatever plans the BBC will no doubt have down the line? I’m sure the BBC Blogs team have a wishlist a mile long of potential blogs that people want to launch – I know this from my own past experience! Could WordPress cope on the BBC’s servers with double, triple, quadruple what the BBC currently demands? Ten times as much? Twenty? What if just quarter of all BBC radio shows had Blogs? (That would be an enormous amount of blogs in itself!)
    I don’t know – I haven’t done the research! But important questions that people inside the BBC (and on any website with very large scale ambitions) have to ask.

  • John Wright says:

    But Andrew that’s the point: there are externalities that factor into the decision, but it would still be better to start with the best platform and the one which is most demonstrably capable of performing well expanded. Unless you’re saying that the BBC will have the largest blogs on the internet then WordPress has already proven all you need it to prove.
    That said, I appreciate the nightmare you must have of being part of turning this around; I’m in media myself and know what it’s like to hear complaints.

  • Andrew Bowden says:

    If nothing else, it’s worth re-iterating the point I mentioned earlier – the BBC’s website doesn’t currently run PHP. So WordPress was frankly not an option at the time. Even when PHP does come online, I suspect WordPress’s dynamic publishing model will give it black marks because the BBC prefers static where possible because it’s easier on the servers.
    However I will be absolutely frank. If every web producer in the BBC had the blogs they probably want, then frankly I can see the blogs on the BBC easily giving a serious run for its money!
    It’s impossible to imagine that from the outside. But it’s true.

  • Dan says:

    Well then, get php made available. The BBC does control its own servers… doesn’t it?
    But WordPress — especially the MU versions, with the SuperCache plugin, and with the php cache accelerators available — is by far the fastest-performing blog platform out there.
    Clearly MT isn’t, as the BBC blogs’ current performance shows!

  • Andrew Bowden says:

    There’s a whole raft of stuff out in the Blogosphere about the BBC’s web platform, but to put it briefly – you simply don’t threaten the stability and structure of a frankly HUGE website by changing its server configuration, just for one application. The BBC’s web servers are not your average single server, LAMP setup – they’re a bit more complicated than that!
    The BBC is renewing its web infrastructure and has taken a wider view of the matter – PHP is being added, but it’s part of a huge renewal project. I could have sworn there was something about this on the BBC Internet Blog but unfortunately I can’t find it…
    Incidentally how much spam does handle? It’s all very well talking about legit comments, but spam takes up far more of the comment handling, so I’d be fascinated to know how many comments WordPress actually handles. (Obviously I know what Askimet handles before anyone says anything)

  • Andrew Bowden says:

    Incidentally, whilst I know this might not go down well with the WordPress fanboys, there’s one very nice user benefit to the way the BBC is changing its blog comments system – and ultimately when it comes down to it, technology is absolutely irrelevant, and user experience is everything.
    The user benefit to merging the BBC’s message board software to handle the blog comments, is that now someone who posts on message boards, blogs and a few other places on the BBC website, will be able to keep track of all their comments in one place. Here’s Peta’s page – Peta is one of the people who works on community software for the BBC. One page shows her message board comments, another page shows comments she’s posted elsewhere (and of course, shows the world too)
    Whether it be WordPress or Movable Type handling the comments, that kind of feature wouldn’t automatically be in place. There would be different systems for handling blog comments, board posts etc.
    Personally, I think this way is far nicer for people using this stuff on a day to day basis. Stuff the technology choice because that doesn’t mean diddly squat. Cos user experience is KING! As a user I don’t care whether it’s DNA, WordPress or Movable Type handling this stuff – it doesn’t matter one bit. What I want as a user is the absolute BEST user experience possible. Not good user experience – but the BEST!

  • Dan says:

    Akismet is a hosted service running on the same servers as, so it’s hard to seperate the two.
    Of course the BBC has a large server environment — nobody disputes that. But so does Wikimedia and, who both run distributed LAMP stacks with PHP caches and Squid html caches for acceleration. Solved problems, once again.
    Your “Peta’s page” example is already possible though WordPress and the intergration possible with bbpress, the bulletin-board brother of wordpress (same codebase). They can share the same database, so users are logged in to both the WordPress blogs and bbpress forums at the same time via single-sign-on, and yes, you can list both blog and and forum comments together. Again, solved problems.
    (I do hope we won’t be forced to “sign up” to post to BBC blogs in the future?)
    And of course everyone wants the best UX, but as the BBC blog comments have been failing for a year now, those in glass houses shouldn’t throw stones…

  • Andrew Bowden says:

    I wasn’t really thinking of Askimet as a server – just more about how many comments handles as part of its day to day basis. What’s its capacity – when does it begin to struggle. Does it begin the struggle?
    As for the servers, well yes, lots of people have made LAMP scale – its perfectly possible and doable. Loads of people do it. That’s not the point. Fact is the BBC server configuration has been around since the 1990s and it hasn’t changed much in that time. It’s stable and dependable.
    When you change things, there is the potential to introduce problems. So people running enterprise server farms – no matter what technology they use – have a tendency not to change things. And when you do change them, you consider carefully the changes you’re going to make, and you implement carefully. The last thing you do is rush in and implement one big server change, just for one application.
    As for message boards, I have to say, I think the chances of the BBC moving away from its existing software are VERY slight. It’s had message board capacity problems in the past and ultimately the software now works reliably most of the time. Few hicups here and there, but everything has downtime when it comes down to it. It ain’t going to be dropped, just so a WordPress powered blog could integrate with a WordPress powered bulletin-board to give the best UI.
    I know you have your views, and you can say them as much as it wants. Everyone knows the current BBC comments problems suck. Especially my colleagues who work on it. But they’re the ones who know the full picture – which to be absolutely honest and fair, neither you or I do.

  • Dan says:

    You’re quite right, nobody messes with enterprise software if it can be avoided. Wikimedia, to use my example again, still runs mysql 4 and memcache, despite both being years old (but still supported with security patches) and superceded by nominally “better” software.
    It does strike me though, with the multiple CMSs inside the Beeb, the validation problems the CPS in particular seems to be creating, and the restrictions of being pretty much Perl-only (though I hear php is coming, and perl on rails is a clever solution to a tight situation), do suggest that perhaps the road Siemens have gone down is a dead-end.
    Not every path can be successful…

  • Andrew Bowden says:

    Multiple CMS’s aren’t necessarily a bad thing in my opinion – as long as they’re well written and flexible. There’s little point in duplicating the same thing between 20 systems (or whatever) in my mind, but sometimes a different tool might be better for the same job. Can/should one system handles film reviews, picture galleries, news stories, recipes, football tables etc? Can? Yes. Should? I couldn’t say.
    Can’t blame Siemens for the server infrastructure situation though – they ultimately are paid to do what the BBC tell them to do. If the BBC management had decided to do all this years ago, it could and would have been done. But then, was it really needed three, four years ago? In many ways, the answer is probably not…