Ngix "504 Gateway Timeout" - Techies, can you help?

[FTL_Ian]

Well, I think this deserves its own thread.

My admins appear to be stumped by this problem that has been plaguing the server for a long time.  It has gotten FAR worse recently since the FTL 2.0 launch, resulting in the server being down for hours at a time.

They have tried many fixes, but the problem has not gone away.

According to Dan, "Some code, or  sequence of code, is causing a PHP process to hang."

Trying to figure out what code that is, is apparently not an easy task.  Would love to hear some ideas on how to address this.

[FTL_Ian]

Latest fix:  We will be forcing PHP to restart every 15 minutes, but that still doesn't address the root of the problem.

[blackie]

According to Dan, "Some code, or  sequence of code, is causing a PHP process to hang."

I blame Linux.

[ForumTroll]

Bad RAM?

[theCelestrian]

Generally hung processes center around the DB connections - I'm assuming Dan has eliminated the possibility that connection threads are not closing properly, thus causing a "traffic jam" on your servers, correct?

Edit: I'm also assuming you found php was hanging due to running top in your terminal as well, correct?

Only other potential place I could see you having hanging issues is if the new 2.0 site is heavily using AJAX, and some of your XMLHttpRequests (or whatever framework - prototype.ajax, jquery, etc...) maybe be hitting some of your php files (that aggregate the data) so often that it could be potentially causing the problem I mentioned above.

[Robin]

yeah i pretty much felt like i was detoxing from FTL all day

 

[blackie]

It sounds like a common issue.

http://blog.taragana.com/index.php/archive/how-to-stop-crashing-hanging-of-php-cgi-spawn-fcgi-with-nginx-lighttpd/

[theCelestrian]

It sounds like a common issue.

http://blog.taragana.com/index.php/archive/how-to-stop-crashing-hanging-of-php-cgi-spawn-fcgi-with-nginx-lighttpd/

Good find.

[error]

It sounds like a common issue.

http://blog.taragana.com/index.php/archive/how-to-stop-crashing-hanging-of-php-cgi-spawn-fcgi-with-nginx-lighttpd/

We've already done that. It had no real effect. Now we have php-fpm. Still no real effect. So we're dealing with something else.

[theCelestrian]

It sounds like a common issue.

http://blog.taragana.com/index.php/archive/how-to-stop-crashing-hanging-of-php-cgi-spawn-fcgi-with-nginx-lighttpd/

We've already done that. It had no real effect. Now we have php-fpm. Still no real effect. So we're dealing with something else.

When this happens, has anyone tried SSH'ing into (I'm assuming) mysql to see how it's responding?

[ForumTroll]

Why not just use Apache?

[theCelestrian]

Why not just use Apache?

A good question - generally you might be compelled to use a lighter build if your boxes gets über high amounts of traffic, but you don't have the cash to either buy more boxes and load balance, or cache your content with a service like Akamai.

Edit: When you do start geo-location caching, then how your website works and how you develop for it completely changes, because your site no longer simply "refreshes" as soon as you upload the content to your server.

[error]

Because Apache is a huge hog which was causing even worse problems than this.

[blackie]

Because Apache is a huge hog which was causing even worse problems than this.

Sounds like it's time for a hardware upgrade.

[error]

Because Apache is a huge hog which was causing even worse problems than this.

Sounds like it's time for a hardware upgrade.

Already ordered.