Multiple FSP-related forums down (DoS'ed?) at the same time!?

[AL the Inconspicuous]

I've been getting nothing but them "504 Gateway Time-out" errors from the main FTL site and the BBS for a while this morning, so I started my usual "panic threads" on NHLiberty and on the FSP Inc forum.  I see a gap in posts on this BBS between 1:55:34 AM and 10:22:17 AM today (EST), so this clearly isn't just me.

I got panicky because NH Underground forum has been throwing "sorry, because of high stress on the server, this feature is currently unavailable" errors as well, and it's still downright unusable.  From what I understand, those sites are hosted by different companies and have nothing in common except that they're the more radical of the FSP-related forums out there. 

Dreepa on FSP Inc forum says:  "There was a porc 411 about free keene the other day."

BrianS on NH Liberty forum says:  "nhliberty.org was down for a period last night.   I don't remember the exact timing.   Somewhere around 10PM maybe?"

That makes it very suspicious...  I've been smelling an increase in government snooping all over the Internets.  Could this be a start of some sort of an anti-liberty DoS (Denial of Service) campaign?

[rookie]

This site was down for me for a bit too a day or so ago...can't remember exactly when...


I just went and did something else and came back later...can't remember exactly what...

[Bill Brasky]

Maybe the locality took a shit.  It happens.  Since most of that stuff is in the same neighborhood, and all. 

[AL the Inconspicuous]

That's a problem.  We need fault tolerance and decentralization - cluster them Web & DB servers to hosts in Nigeria 'n stuff. 

[error]

Dan did an emergency maintenance on FTL this morning to correct the problem, so FTL, FK, etc., would have been down. This consisted of installing php-fpm.

I've been looking for signs of denial of service attacks and find nothing. The problem right now seems to be that we need yet more performance tuning, and perhaps bigger hardware, to support this Drupal monstrosity.

[sillyperson]

this Drupal monstrosity.

Don't get me started.

Anyway, Step #1 is enable caching, but Dan knows that

[blackie]

The problem right now seems to be that we need yet more performance tuning, and perhaps bigger hardware, to support this Drupal monstrosity.

Is the whole implementation hosted on a single tier?

[error]

this Drupal monstrosity.

Don't get me started.

Anyway, Step #1 is enable caching, but Dan knows that

Boost doesn't do too much for you when most of your traffic is from logged-in users. Got any other ideas?

[AL the Inconspicuous]

This is why the last few proprietary Web-based systems I've designed from scratch only did minimal dynamic work, with all the data being in pregenerated static files and put together on the client side via AJAX.  In one case even user input was perfectly passive, with a separate simultaneous process to read it from the frequently-rotated binary Web-server logs (modified publicfile) and update the static .json.gz data-files, usually with no perceivable delay.  Now there's a system that can easily scale forever!

As for making this inherently-inefficient LAMP monstrosity shine brighter - I'm no MySQL guru, so your guess is as good as mine.  I'd experiment with different storage engines, indexing, table partitioning, more RAM, etc.  Just don't even think about archiving old threads as static files and removing them from the live forum, or I will track you down and use you for head transplantation experiments (which would probably work in Johnson's favor though).

[error]

Well, let's see. We got rid of the A in LAMP. That helped a LOT. I'm still tuning MySQL but I think it's mostly set now. PHP is using FPM now, which seems to have improved matters.

The big problem with this particular situation is, well, Drupal.

[AL the Inconspicuous]

It's weird that Ian mentioned the "nhfree.com forum" at the end of the Elena Katz interview (first segment of Feb 12th FTL), while that forum has been completely unusable with that "high stress on the server" issue (see above) for at least a month...