Jump to content

The Upgrade - Revisited. #OperationPrune


Recommended Posts

Last week was not great obviously so I wanted to explain a bit about what happened and what is going to happen next!

 

What Happened Last Week

The forum upgrade itself was a big one. I've let the software fall behind over the past couple of years and so we're now having to make quite a big jump from where we are to the latest version of the forum that we use.

 

I tested this for a few months on a separate server and ran through the process probably about a dozen times in test before deciding to take the plunge and do it for real.

 

The upgrade itself was fine, it went without a hitch. The problem we had was that after the upgrade is completed it starts to run a series of "maintenance tasks" one of which is to rebuild all the posts in a new format for the new software.

 

These tasks run with low priority, they take a long time to complete, but the aim of this is that we were supposed to be able to open the forum whilst they were running. In test I was able to do that without a problem, but on live with a couple of thousand people hitting the forum, it just ground to a halt and everything more or less failed.

 

I took all the users off but the jobs were still running slowly. I knew from testing that they would take around a week to complete which is why I then decided to run the cut down "wasteland" version of the forum for a few days whilst the upgrade ran on a different server.

 

Towards the end of the week we got to the very last maintenance job and I had high hopes that we were through but the last job kept failing and falling over before eventually I just gave up and rolled back to the original server and the original forum and that's where we are today.

 

The cause of the slowness is really all down to the posts table. We have 8.6 million posts in the database, and whilst in commercial database terms that is not a lot, for this software it appears to be too much for it to cope with when running through the upgrade. I've tred throwing shedloads of processing power and memory at it but it just doesn't seem to be able to handle it.

 

I've looked at other forum applications but the big enterprise ones are too expensive to be feasible and the other ones, well they look shit to be honest.

 

Having spent months on this, literally, and having spent a load of money too, I'm not for giving up! There are functional, security, and usability reasons for wanting to upgrade. The new software performs much better on mobile especially and with 60% of our traffic coming via mobile it's important to me that we keep pushing on.

 

What Happens Next

Really the only way forward is to prune the forum. We're carrying millions of posts we don't need. By reducing the amount of data in the posts table I think we can improve the speed of the upgrade to the point where it becomes viable. I reckon we can get down from 8 million posts to nearer 2 million without really affecting anything noticeable.

 

To do that I'm going to start a number of prune jobs over the next week or so.

 

Apart from P&B gold which won't be touched the task will be told to delete any topic that has had no posts in it for the last 2 years. Any topic with active posts in the last 2 years will be untouched, and of course Gold will be untouched.

 

At the end of that we'll have a smaller post table, and I will do some more tests, and then we'll schedule the upgrade again.

 

So there you go!

Link to comment
Share on other sites

  • Replies 289
  • Created
  • Last Reply

TLDR.....

 

All I'm seeing are excuses.....

 

We're all rooting for you

 

If I were you I'd blitz those word and photo association threads, plus the random images/gifs threads in the Nightshift. That would free up some space?

Link to comment
Share on other sites

Sounds like a plan. I would also suggest that you just bin the politics forum too - it deserves to be purged. 

 

I think keeping stuff like that in one place is the right thing to do. Means if you aren't interested in politics or you think that the section is shit, then it's easy to avoid.

Link to comment
Share on other sites

TLDR.....

 

All I'm seeing are excuses.....

 

We're all rooting for you

 

If I were you I'd blitz those word and photo association threads, plus the random images/gifs threads in the Nightshift. That would free up some space?

 

Space isn't an issue, it's just volume of individual posts that is the issue.

 

I honestly think I could have got away with pruning 5 million posts and nobody would have noticed, but I like to be up front with what I'm doing so everyone can bollock me when it goes wrong.

Link to comment
Share on other sites

Div, it would hardly be the end of the World but I've previously bumped what I believe is the oldest remaining post on the forum and went to do it again just then but it's listed as having been archived and I'm unable to bump it.

 

I've not seen anything listed as archived on P&B before!?

 

Link

Link to comment
Share on other sites

Div, it would hardly be the end of the World but I've previously bumped what I believe is the oldest remaining post on the forum and went to do it again just then but it's listed as having been archived and I'm unable to bump it.

 

I've not seen anything listed as archived on P&B before!?

 

Link

Damnit...there goes my plan to congratulate Div on coming up with a brilliant solution, before proceeding to bump every thread that's >2 years old...

Link to comment
Share on other sites

Div, it would hardly be the end of the World but I've previously bumped what I believe is the oldest remaining post on the forum and went to do it again just then but it's listed as having been archived and I'm unable to bump it.

 

I've not seen anything listed as archived on P&B before!?

 

Link

 

Yes mate, archiving the old posts was my original plan after last week but it doesn't really resolve the problem so I've binned archving in favour of pruning.

 

ps; I have queued that particular topic for "unarchiving" so if you want to preserve it you can post in it so it escapes the cull!

Link to comment
Share on other sites

Yes mate, archiving the old posts was my original plan after last week but it doesn't really resolve the problem so I've binned archving in favour of pruning.

 

ps; I have queued that particular topic for "unarchiving" so if you want to preserve it you can post in it so it escapes the cull!

 

^_^

Link to comment
Share on other sites

Last week was not great obviously so I wanted to explain a bit about what happened and what is going to happen next!

 

What Happened Last Week

The forum upgrade itself was a big one. I've let the software fall behind over the past couple of years and so we're now having to make quite a big jump from where we are to the latest version of the forum that we use.

 

Became a media darling, and forgot where he came from. Fucking sickening.

 

Everything else seemed reasonable though.

Link to comment
Share on other sites

Just get rid of the Word Association thread. L23 and Booitsme must've went round in circles hunners of times already anyway. The BRALT as well. ;)

Came in here to post that. You could clear the 'Things you want to share with P&B' thread too. I use it a fair bit, but I doubt anyone wants to flick back a few pages in it, so it would be no great loss.

 

Look how helpful everyone's being with their suggestions of what threads to clear. I'll need to bring this back down a notch.

 

You could get rid of all the old Rangers' threads because they're dead so won't read them.

Link to comment
Share on other sites

Came in here to post that. You could clear the 'Things you want to share with P&B' thread too. I use it a fair bit, but I doubt anyone wants to flick back a few pages in it, so it would be no great loss.

 

Look how helpful everyone's being with their suggestions of what threads to clear. I'll need to bring this back down a notch.

 

You could get rid of all the old Rangers' threads because they're dead so won't read them.

 

That's a fair shout, same goes for RTBC, PTTGOYN and pish folk write on Facebook.

Link to comment
Share on other sites

Huge swathes of P&B history will be gone.

 

Cock teasing bitch, GLOVES, Akabusi sex tales, badgers in the drain.  All gone.

 

I'm in favour though.  Destroy history.  Only look forward.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...