FabSwingers.com > Forums > Fabswingers.com site feedback > Time offline: Full explanation, supporter passes extended

Time offline: Full explanation, supporter passes extended

By *j_mark OP Couple over a year ago
Forum Mod

Totteridge/Whetstone

Hi,

In summary:

We had on-off server problems since Friday which we now believe are fixed. You probably saw "couldn't connect" messages or slow pages during this time. We're really sorry about this

We've added a FULL DAY to all site supporter passes to apologise and make good.

Long story:

We wanted to provide an explanation of the recent glitches on the site.

We're in the process of upgrading all our servers. The main fab webserver is being replaced by new ones which gives us plenty of scope to grow and add cool stuff (and we're continuing to grow fast; the site serves up over 7 million dynamic pages each day, i.e. not including static files and image content).

We did the upgrade last Friday (18th March) after a lot of testing. However, we started to get the odd complaint that some users were seeing occasional glitches. When you run a busy site with over 50,000 people logging in each day it's sometimes hard to understand when there's a real problem because we get a steady stream of reported issues, some of which are down to local connection issues, some aren't. Eventually we started to see some of these issues ourselves.

While we could have rolled back to the old server at any time we took the decision (possibly the wrong one in retrospect) that it would have been difficult to analyse and fix any potential problems on the new server, and our own stats showed that most (99%+) of page requests were being successfully delivered. So we tried to analyse and resolve the issue on the new servers while they were live.

Today we made the decision to flip back to the old server, while we continued to test the problem on the new ones. With the help of our network experts we finally recreated the problems using a series of test, which then allowed us to trace the issue down to a network driver which for some reason was dropping packets on our new servers.

This was fixed by updating the driver software this Wednesday afternoon (23rd March) and we've been re-testing it continuously, and subsequently gone live with it again at approx 19.15 this-evening. We had to flip the site twice today, once back to the old server (lunchtime) and again to the new server (19:15-ish). During these times the site went offline for 5-10 minutes. And while the site was on the old server we experienced the old capacity problems! Aargh!!!!

We're now back up and running on the new servers and are quietly confident the glitches are resolved. We will continue to monitor.

We've improved various procedures about how we comission and test new servers through this.

Sometimes the site is so busy that we're pushing the boundaries of our web servers, software, databases, etc.

That said, we're commited to having the highest standards of site availability and site performance and we'll always make good if we don't achieve that which is why we've added a full day to all site supporter passes.

Admin x

Reply privately, Reply in forum +quote or View forums list

By *j_mark OP Couple over a year ago
Forum Mod

Totteridge/Whetstone

Just to confirm, all site supporter passes are now extended.

Admin x

Reply privately, Reply in forum +quote or View forums list

By *oantrimcpl2010Couple over a year ago

Lisburn

thxs

Reply privately, Reply in forum +quote or View forums list

By *uckoldandWifeCouple over a year ago

Manchester

all that work and the site is still free to the majority of users, just how do you do it, thanks

Reply privately, Reply in forum +quote or View forums list