Catholica Spiritual Marketplace - Just in case anybody can help...

Just in case anybody can help... (reply)

Tue, 08 Dec 2009 17:05:02 +1100

Thanks, Brian. Donatus is a wizz on this sort of stuff and we're corresponding by email. The whole thing is exceedingly strange as all the usual indicators (uniques, visits, pagereads, hits and bandwidth) are basically pretty stable and normal. It's only the cpu usage that has gone haywire. There was a problem with the forum software when I upgraded to the new version about 6 months ago but it only occurred when members tried to access the user area and it tried to count up how many posts each member had made. The guy who developed the software found a fix for that (basically by disabling the count of posts for each individual member). Curiously enough the admin section of the website still does that count but it doesn't cause the server to fall over and display a 505 error. The changes I made yesterday flattened the rate of overload to only 2% and I'm now waiting to see what happens tonight. The whole thing is weird. People on the blog of the software developer's site have told me that in some places this forum software is operating with databases of over half a million entries with no problems. We are a relative minow compared to them at just under 38,000 entries.

Just in case anybody can help... (reply)

Tue, 08 Dec 2009 16:52:23 +1100

Donatus' questions re separate servers and service-provider configuration for the two forums is a good one. Also does your server provider give you a dedicated server for Catholica pages, or is it shared with other clients? Has any of the software on the server been upgraded recently - new version of Apache (or the Microsoft equivalent)? New or upgraded security software? Has anyone been more ambitious with their programming? Was it on the 28th that the expanding and contracting bikini ladies appeared? Are they or other advertisements now being incorporated into your server pages before being despatched out to us? Any other new server-side functions being performed? Database lookups for example.
At least you aren't at 100% yet.
Best wishes for this.
brian3144

Just in case anybody can help... (reply)

Tue, 08 Dec 2009 12:46:51 +1100

Thanks, D. I've just woken up and have to complete the layout for the brilliant concluding part of Andrew Kania's address at Eton last week. I'll be back in touch after I've done that. You might be just the person I need to get to the bottom of this.

Cheers,

Just in case anybody can help... (reply)

Tue, 08 Dec 2009 07:47:17 +1100

I don't understand why the load would be effecting the main forum but not the marketplace forum?

Is the site multi-hosted? Do you use a load balancer? Might be that one of the VMs in the cluster has fallen over perhaps?

Do you have a diagram of how the server/s are arranged?

-D

Just in case anybody can help... (reply)

Tue, 08 Dec 2009 02:27:59 +1100

Here is a graph of the cpu load on our servers from 7 September up until 7 December. You will notice the usage started to ramp up on 28 November. Can anybody think of anything that has occured from that date that might explain the increase? For example has anyone suddenly discovered some new function on the forums and only begun using it from around that date such as searching your own posts. No names/no packdrill/no accusations — I am simply trying to identify possible causes or it might be that some new facility someone has discovered needs to be modified so that it works properly and doesn't overload the server.

Technical Problems further update (0116Hrs Tuesday)...

Tue, 08 Dec 2009 01:21:01 +1100

Extremely frustrating...

Dear all,

I am sorry to report our technical problem is still without solution. The load on the cpus has increased to 89% in the latest update despite my attempts yesterday to strip down quite a number of things that I thought may have been causing the server to overload and to make the main forum read only.

I am considering radical surgery to remove a large section of the archive from the database and store it off-site temporarily to see if that brings about an improvement in the next 24 hours.

Reading the technical blogs, other people have experienced similar frustrations with other types of forum software and it is a pain in the butt when it erupts. There is nothing obvious showing up in the log files of who is accessing our site that gives any immediate indication of a particular activity that has suddenly caused the server to overload. We also don't appear to be the subject of any malicious or denial of service attacks. What is happening appears to still have a mysterious cause. Despite our problems with the forum the number of visits to Catholica was maintained yesterday (Monday) with 1502 — that's about bang on average for weekday visits these days — and ironically the bandwidth (the total of information downloaded to users computers) was slightly lower than normal. At present then the overloading is not causing the site to actually crash or slow.

Again my apologies for this disruption to our normal operations and please feel free to continue the conversation on this forum.