Using the top tool - my server overloading - GoFuckYourself.com

camperjohn64 · 02-03-2010, 09:31 AM

My server is overloading with very few users. When running #top, what do you guys see on your system? Can you post an example of what you see for comparison?

Code:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5291 nobody    33  18 41236  15m 5796 S  3.3  0.8   0:21.91 httpd
27520 named     24   0 38108 3224 2192 S  0.3  0.2  22:06.92 named
 7670 nobody    33  18 41268  15m 5632 S  0.3  0.8   0:12.91 httpd
 8787 nobody    33  18 41112  15m 5592 S  0.3  0.8   0:05.54 httpd
 9518 nobody    33  18 41112  15m 5508 S  0.3  0.8   0:04.47 httpd
 9604 nobody    33  18 41112  15m 5444 S  0.3  0.8   0:05.17 httpd
 9808 nobody    33  18 41236  15m 5512 S  0.3  0.8   0:03.39 httpd
10328 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:03.12 httpd
10336 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:01.94 httpd

Do you see similar values on your servers?
Do you see the similar values for the VIRT column as I have above (for both NAMED and HTTPD)?
Does a typical httpd request take 0.2% memory on your system as well?

I'm hosting single static pages. No PHP.

Yes yes I know this is dependant on server config I am just looking for examples of what is within normal ranges.

SmellyNose · 02-03-2010, 09:36 AM

Quote:

Originally Posted by camperjohn64

My server is overloading with very few users. When running #top, what do you guys see on your system? Can you post an example of what you see for comparison?

Code:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5291 nobody    33  18 41236  15m 5796 S  3.3  0.8   0:21.91 httpd
27520 named     24   0 38108 3224 2192 S  0.3  0.2  22:06.92 named
 7670 nobody    33  18 41268  15m 5632 S  0.3  0.8   0:12.91 httpd
 8787 nobody    33  18 41112  15m 5592 S  0.3  0.8   0:05.54 httpd
 9518 nobody    33  18 41112  15m 5508 S  0.3  0.8   0:04.47 httpd
 9604 nobody    33  18 41112  15m 5444 S  0.3  0.8   0:05.17 httpd
 9808 nobody    33  18 41236  15m 5512 S  0.3  0.8   0:03.39 httpd
10328 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:03.12 httpd
10336 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:01.94 httpd

Do you see similar values on your servers?
Do you see the similar values for the VIRT column as I have above (for both NAMED and HTTPD)?
Does a typical httpd request take 0.2% memory on your system as well?

I'm hosting single static pages. No PHP.

Yes yes I know this is dependant on server config I am just looking for examples of what is within normal ranges.

That looks fine.

What makes you say your server is overloaded?
What does "free -m" show?
What is your load average? (uptime or top)
Are you using swap? - This is shown in top aswell.

Also, I always run "top -c" rather than "top". It's much more helpful.

Lace · 02-03-2010, 10:05 AM

On a side note, the site in your sig has been suspended.

liquidmoe · 02-03-2010, 10:08 AM

Sometimes when you have high CPU processing on httpd processes and nothing else it could be indicative of poorly structured rewrite rules in one of your .htaccess files.

You may want to take a look at that.

camperjohn64 · 02-03-2010, 10:48 AM

About every 12 hours, or 18 or whenever, my server gets "stuck". uptime goes to load average: 150.26, 150.37, 151.32 and everything stops. I do a httpd stop, wait, then httpd start and it runs fine:load average goes back to 0.01 etc.....but only for another X hours then it sticks again. Completely random. Could be in 20 minutes, or 18 hours.

I have been looking for it for about a month, and can't find what the deal is. I have shut down Mysql, suspended (almost) all accounts except for the account that gets most visitors, and it still gets stuck. I have a SINGLE static page, and bam, still crashes. No php, no nothing.

I say "stuck" because it is still running, and I can shell to it, just that httpd is no longer processing requests, and doesnt flush out the old ones. It's stuck.

I have hired 2 admins to help out, but both look at the server and are stumped after about an hour...

camperjohn64 · 02-03-2010, 10:49 AM

Quote:

Originally Posted by Lace

On a side note, the site in your sig has been suspended.

Yes I did that, to rule out that website being the cause...most other sites are also suspended until I find this.

Klen · 02-03-2010, 10:50 AM

Is that dedicated or shared hosting?

StaceyJo · 02-03-2010, 10:59 AM

So technical for me here. hehh

CYF · 02-03-2010, 12:43 PM

that top output looks fine

Every 12 or 18 hours? You have anything setup in cron that is processor intensive?

Have you checked logs for bots? Sometimes they hammer a site and bring it to a crawl.

Are you running apache? What version? What OS?

camperjohn64 · 02-03-2010, 01:25 PM

Quote:

Originally Posted by liquidmoe

Sometimes when you have high CPU processing on httpd processes and nothing else it could be indicative of poorly structured rewrite rules in one of your .htaccess files.

You may want to take a look at that.

Good idea. The main site does use a 404 rewrite, and it's been working for 5 years now with up to 100,000 visitors a day at it's peak, but maybe I changed something in one of the other smaller sites.

Quote:

Originally Posted by CYF

that top output looks fine

Every 12 or 18 hours? You have anything setup in cron that is processor intensive?

Have you checked logs for bots? Sometimes they hammer a site and bring it to a crawl.

Are you running apache? What version? What OS?

Running CentOS, apache, basic LAMP server...CentOS release 4.8 (Final)

I will shut down all my cron jobs to make sure. I forgot about that! But shutting down MySQL should shut down any cron jobs that use MySQL data. But good idea!

I don't think it's bots or traffic, because if it was just too much traffic, then after restarting apache, it would get stuck straight away again. It happens at odd times.

Here is some interesting notes when I managed to "catch it in the act" of getting stuck. When it is starting to get stuck, I notice uptime will start to report higher and higher values. During this time I can get the apache status. Once its stuck, then I can't. But I did manage to catch it in the act, and here is what uptime showed me:

0.12 0.08 0.09
5.12 0.12 0.08
30.2 5.12 0.12
90.0 30.2 5.12
150.1 150.2 149.0
(server is now toast)

In a very short time, it's as if it gets "stuck". Once it's stuck, I shutdown apache, restart it, then everthing runs fine for 13 more hours, then it happens again.

lynx --dump localhost/whm-server-status

So, tell me what you make of these apache status reports:

http://www.photofight.com/xx.txt (started to get stuck)
http://www.photofight.com/xy.txt (a little more stuck)
http://www.photofight.com/xz.txt (now it's really stuck)

Notice the jump halfway down, where the milliseconds for a response becomes 1921913212 ?!?!? Search for 9068 and you will see q2growth.com GET /Printing-Supplies/Printing-Supplies.html HTTP/1.1

Shutdown apache and restarted:
http://www.photofight.com/xq.txt (yay its not stuck)

If it was simply too much traffic, then restarting apache would cause it to overload immediatly. So it can't be just a matter of too much traffic. There is something fishy and I can't figure it out. I can't work on ANY more websites until I figure this out too, since I don't have any money coming in while its 'stuck' and it is top priority.

I love the word stuck. Its the best way to describe it.

Ideas? Notice how requests build up, but dont get out! They just stack on top of each other!

borked · 02-03-2010, 02:16 PM

why/how is your apache instance serving up both HTTP/1.0 and HTTP/1.1 ???

Everything on srv1.ryd0.com is 1.0 whereas all the rest are 1.1

--edit
nm, it's client based

rowan · 02-03-2010, 02:23 PM

Hmm... just thinking about possible issues, and this one jumped out... why do you have mysql on the server if all pages are static? Do you run a script that rebuilds pages?

borked · 02-03-2010, 02:36 PM

What's weird is all your apache requests are pretty much "sending reply"
Normally, you have a big mix of "Reading Request",. Closing Connection, Waiting for Connection, and just a few sending replies.

Yours are all sending reply.

Here's an output from one of my servers that is handling ~60 requests/sec to your 7 requests/sec.

Code:

Parent Server Generation: 0
Server uptime: 1 day 21 hours 51 minutes 46 seconds
Total accesses: 9393716 - Total Traffic: 5.1 GB
CPU Usage: u1865.79 s183.61 cu3.53 cs0 - 1.24% CPU load
56.9 requests/sec - 32.2 kB/second - 579 B/request
16 requests currently being processed, 25 idle workers

R__C__CC______C_W__WR____CC...CC....CC.C_.....C__.__.._._.......
......_.........................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
....

You can see the # requests currently being processed on yours is gradually increasing over time, which means something is blocking apache, and like mentioned it could be a fucked up rewrite rule that is looping or something.

In any case all the sending replies is not normal

camperjohn64 · 02-03-2010, 04:14 PM

Quote:

Originally Posted by rowan

Hmm... just thinking about possible issues, and this one jumped out... why do you have mysql on the server if all pages are static? Do you run a script that rebuilds pages?

I normally run programs that use Mysql PHP. I just shut it down to make sure it wasn't a MySQL hang up or something.

I put up the static page to rule out the PHP too. Basically I removed everything to get it stable, then I would add items and find out what is causing it. Still no luck as even with a single static page, it is still fucking up.

I am a good PHP programmer, but a lousy admin. :-(

beta-tester · 02-03-2010, 04:27 PM

It's interesting, but I do have the same issue as you described. The server load normally is totally ok, around .01, or so. But at some point of time during the day it also gets soo high, and the apache got stuck. Until I restart it, or let it do it by itself, the HTTP pretty much doesn't work at all.

I tried to look into it but couldn't find anything that would cause this. It's really annoying ;(

camperjohn64 · 02-03-2010, 04:31 PM

Quote:

Originally Posted by beta-tester

It's interesting, but I do have the same issue as you described. The server load normally is totally ok, around .01, or so. But at some point of time during the day it also gets soo high, and the apache got stuck. Until I restart it, or let it do it by itself, the HTTP pretty much doesn't work at all.

I tried to look into it but couldn't find anything that would cause this. It's really annoying ;(

Are you using Cpanel? I am using Cpanel at the moment.

camperjohn64 · 02-03-2010, 04:33 PM

Holy shit I see a pattern in my hosts down!!!

4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world!

That means a backup, a cron job. Backup or something.

I predict a crash at 8:30 tonight!

beta-tester · 02-03-2010, 04:36 PM

Quote:

Originally Posted by camperjohn64

Are you using Cpanel? I am using Cpanel at the moment.

yes Cpanel with WHM.

Could be backup, but the thing is that Apache gets overloaded, not other process, that's the problem.

Domain Broker · 02-03-2010, 04:37 PM

Do you have a shitty host? Why aren't they looking into this?

rowan · 02-03-2010, 05:08 PM

Quote:

Originally Posted by camperjohn64

Holy shit I see a pattern in my hosts down!!!

4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world!

That means a backup, a cron job. Backup or something.

I predict a crash at 8:30 tonight!

So get checkin' your crontab then...

Under FreeBSD when you make a snapshot of a drive that has a zillion files it can sometimes freeze the OS for a minute or two, you can type at the cmdline but as soon as it needs to access a HD it just waits... I know you're on a completely different OS, just saying that backups are not always 100% seamless.

CYF · 02-03-2010, 05:22 PM

Quote:

Originally Posted by camperjohn64

Holy shit I see a pattern in my hosts down!!!

4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world!

That means a backup, a cron job. Backup or something.

I predict a crash at 8:30 tonight!

crontab -l

If you need help decoding when stuff is scheduled to run I can help.

camperjohn64 · 02-03-2010, 06:14 PM

Quote:

Originally Posted by Domain Broker

Do you have a shitty host? Why aren't they looking into this?

Its a dedicated server so they are hands free. Are there hosts that take care of dedicated admin problems? I will switch in a heartbeat...

Ca$h-collector · 02-03-2010, 08:05 PM

interesting infos

borked · 02-03-2010, 11:53 PM

for every username in
ls /var/spool/cron

run (as root)

crontab -u <username> -l

to see which crontab is the culprit.

camperjohn64 · 02-04-2010, 01:58 AM

I tell you one thing though, I've learned ALOT about my server while trying to track this down. I have focused my skills on PHP and development, not so much admin, but I am actually getting confident in handing some admin functions now.

ilbb · 02-04-2010, 02:23 AM

I recommend set cron every minute and log apache server status so you can see which pages/scripts are requested in the peaks. This will help you determine which script is causing server overload

camperjohn64 · 02-04-2010, 06:05 AM

Quote:

Originally Posted by ilbb

I recommend set cron every minute and log apache server status so you can see which pages/scripts are requested in the peaks. This will help you determine which script is causing server overload

I did this 100 times already. It is not a specific script that is overloading it.

ps -aux gives me this

Code:

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
nobody    6852  0.4  0.9 41988 18228 ?       RN   04:38   0:45 /usr/local/apache/bin/httpd -k start -DSSL
nobody    8893  0.4  0.8 41884 17608 ?       RN   05:17   0:37 /usr/local/apache/bin/httpd -k start -DSSL
nobody    9950  0.4  0.8 41756 17884 ?       RN   05:38   0:32 /usr/local/apache/bin/httpd -k start -DSSL
nobody   10153  0.4  0.9 44352 19716 ?       RN   05:42   0:30 /usr/local/apache/bin/httpd -k start -DSSL
nobody   10682  0.3  0.8 41756 17032 ?       RN   05:52   0:27 /usr/local/apache/bin/httpd -k start -DSSL
nobody   10900  0.4  0.8 41780 17336 ?       RN   05:57   0:27 /usr/local/apache/bin/httpd -k start -DSSL
nobody   11720  0.4  0.8 41640 16820 ?       RN   06:11   0:24 /usr/local/apache/bin/httpd -k start -DSSL
root     11891  1.2  1.7 38516 34880 ?       S    06:14   1:11 spamd child
nobody   12856  0.4  0.9 44232 19332 ?       RN   06:37   0:19 /usr/local/apache/bin/httpd -k start -DSSL
nobody   12973  0.4  0.8 41652 16812 ?       RN   06:40   0:17 /usr/local/apache/bin/httpd -k start -DSSL
nobody   12975  0.4  0.8 41640 16952 ?       RN   06:40   0:18 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13359  0.4  0.9 44220 19472 ?       RN   06:48   0:15 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13507  0.4  0.8 41640 16736 ?       RN   06:52   0:15 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13770  0.4  0.8 41580 17052 ?       RN   06:59   0:11 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13903  0.4  0.8 41528 16676 ?       RN   07:01   0:12 /usr/local/apache/bin/httpd -k start -DSSL
nobody   14287  0.5  0.8 41528 17040 ?       RN   07:11   0:10 /usr/local/apache/bin/httpd -k start -DSSL
nobody   14638  0.5  0.8 41524 16712 ?       RN   07:22   0:08 /usr/local/apache/bin/httpd -k start -DSSL
nobody   14773  0.7  0.8 41516 16644 ?       RN   07:27   0:08 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15054  0.7  0.8 41528 16956 ?       RN   07:33   0:06 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15333  0.9  0.8 41568 16412 ?       RN   07:38   0:04 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15649  1.3  0.8 41528 16208 ?       RN   07:44   0:02 /usr/local/apache/bin/httpd -k start -DSSL
mailnull 15667  0.0  0.2  9596 4076 ?        S    07:44   0:00 /usr/sbin/exim -bd -q60m
mailnull 15669  0.0  0.0     0    0 ?        Z    07:44   0:00 [exim] <defunct>
nobody   15719  1.4  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15722  1.4  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15723  1.1  0.7 41364 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15724  1.0  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15725  1.0  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15726  1.0  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15728  1.1  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15730  1.1  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15731  1.3  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15732  1.1  0.7 41364 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15733  1.2  0.7 41364 15344 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15734  1.3  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15735  1.3  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15736  1.2  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15737  1.2  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15740  1.1  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15741  1.1  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15742  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15744  1.1  0.7 41364 15332 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15746  1.0  0.7 41364 15388 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15747  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15751  1.1  0.7 41364 15340 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15752  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15753  0.9  0.7 41372 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15754  1.1  0.7 41364 15340 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15755  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15756  1.1  0.7 41364 15344 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15757  0.9  0.7 41364 15384 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15758  1.1  0.7 41364 15332 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15759  0.9  0.7 41364 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15760  1.1  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL

Is it normal that an httpd process will take 0.7% memory, and 1.1% CPU time?

Server got stuck again this morning at exactly 4:41am. Still didn't find what cron job it is...

Code:

# crontab -l
* * * * * /root/loadcheck.sh >/dev/null 2>&1
11,26,41,56 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap
55 1 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache
45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel/bin/optimizefs
51 0 * * * /scripts/upcp
0 1 * * * /scripts/cpbackup
0 2 * * * /scripts/mailman_chown_archives
35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check
0 6 * * * /scripts/exim_tidydb > /dev/null 2>&1

At least now I can predict when it will crash..4am, 12, 8pm. Now I just gotta find out why.

02-04-2010, 06:13 AM

Quote:

Originally Posted by camperjohn64

Its a dedicated server so they are hands free. Are there hosts that take care of dedicated admin problems? I will switch in a heartbeat...

NationalNet, they look after anything and everything with your managed, dedicated server(s). And, the service turnaround time is UNBELIEVABLE.

SmellyNose · 02-04-2010, 07:59 AM

30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache

Comment out this one. It runs at 30 minutes past the hour, every 4 hours

Or, run it yourself and see how long it takes to run and if it causes your server to halt.

PastorSinAlot · 02-04-2010, 08:02 AM

choopa would knock this out for u

camperjohn64 · 02-04-2010, 08:10 AM

Quote:

Originally Posted by AdultStoriesNow

30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache

Comment out this one. It runs at 30 minutes past the hour, every 4 hours

Or, run it yourself and see how long it takes to run and if it causes your server to halt.

But the server is down every 8 hours...

I commented out both lines:
30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache
45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel

And I will see what happens...

camperjohn64 · 02-04-2010, 08:14 AM

Or it is possible that the first task, takes more than 4 hours to run, and therefore the second time it is run, halts the server until the first is done. Causing the halt every 8 hours (2 tasks while second is waiting for first to finish).

Hmmm...

SmellyNose · 02-04-2010, 09:53 AM

It's unlikely the first task took that long, but it is possible. I didn't check your times properly (I skimmed the thread, so sue me :P ), so it is more than likely the second task that runs every 8 hours which is causing issues.

Let us know how it goes

camperjohn64 · 02-04-2010, 12:18 PM

Well that wasn't it. Server uptime is 150.0 150.0 150.0 at 11:17am...Both cron jobs were removed..

Back to square one. Having a server that doesn't work, with so few visitors sucks.

beta-tester · 02-07-2010, 12:11 PM

try checking MaxClients in Apache. I just figured out in some of the error logs that it says maxclients are riched, try increasing this value. I increased it, and now I'm monitoring to see what happens.

drogers · 02-07-2010, 03:09 PM

Quote:

Originally Posted by camperjohn64

Well that wasn't it. Server uptime is 150.0 150.0 150.0 at 11:17am...Both cron jobs were removed..

Back to square one. Having a server that doesn't work, with so few visitors sucks.

It could be robots maxing Apache childs and causing load. You should check the logs for this.

camperjohn64 · 02-08-2010, 08:04 AM

The problem seems to be that apache threads are not exiting. They stay in the top of the output buffer.

Here check it out. Here are some outputs as the server is slowly crashing...

http://www.photofight.com/xx.txt
http://www.photofight.com/xy.txt
http://www.photofight.com/xz.txt

See how the top thread is never leaving?

SmokeyTheBear · 02-09-2010, 09:22 AM

check if your getting botted

netstat -anp |grep ‘tcp\|udp’ | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

borked · 02-09-2010, 02:25 PM

Quote:

Originally Posted by camperjohn64

See how the top thread is never leaving?

that is what I said - the server is forever sending replies and taking forever to finish sending each reply.

Are you sure you don't have a rogue js somewhere? I know I can visit a site, and my browser will be forever "receiving item 19 of 20" or something similar. Means some js or remote DNS call is taking forever to complete. But it can bring down a server if that server is having enough hits at that time...

HomerSimpson · 02-09-2010, 03:17 PM

you need to tweak your apache...

Varius · 02-09-2010, 04:44 PM

What do you have set for the following values and which version of Apache?

MaxRequestsPerChild
KeepAlive
KeepAliveTimeout
MaxKeepAliveRequests

camperjohn64 · 02-09-2010, 08:32 PM

MaxClients 250
MaxRequestsPerChild 100
KeepAlive On
KeepAliveTimeout 3
MaxKeepAliveRequests (not set)

fatfoo · 02-09-2010, 09:10 PM

Server overloading - that's not good. Good luck.

Varius · 02-10-2010, 08:33 AM

Quote:

Originally Posted by camperjohn64

MaxClients 250
MaxRequestsPerChild 100
KeepAlive On
KeepAliveTimeout 3
MaxKeepAliveRequests (not set)

Looks ok but without being on the server, hard to say. Try turning KeepAlive Off and see if your problem gets solved...

02-03-2010, 09:31 AM	#1
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	Using the top tool - my server overloading My server is overloading with very few users. When running #top, what do you guys see on your system? Can you post an example of what you see for comparison? Code: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5291 nobody 33 18 41236 15m 5796 S 3.3 0.8 0:21.91 httpd 27520 named 24 0 38108 3224 2192 S 0.3 0.2 22:06.92 named 7670 nobody 33 18 41268 15m 5632 S 0.3 0.8 0:12.91 httpd 8787 nobody 33 18 41112 15m 5592 S 0.3 0.8 0:05.54 httpd 9518 nobody 33 18 41112 15m 5508 S 0.3 0.8 0:04.47 httpd 9604 nobody 33 18 41112 15m 5444 S 0.3 0.8 0:05.17 httpd 9808 nobody 33 18 41236 15m 5512 S 0.3 0.8 0:03.39 httpd 10328 nobody 33 18 41112 15m 5432 S 0.3 0.8 0:03.12 httpd 10336 nobody 33 18 41112 15m 5432 S 0.3 0.8 0:01.94 httpd Do you see similar values on your servers? Do you see the similar values for the VIRT column as I have above (for both NAMED and HTTPD)? Does a typical httpd request take 0.2% memory on your system as well? I'm hosting single static pages. No PHP. Yes yes I know this is dependant on server config I am just looking for examples of what is within normal ranges. __________________ www.gimmiegirlproductions.com

02-03-2010, 10:05 AM	#3
Lace Too lazy to set a custom title Industry Role: Join Date: Mar 2004 Posts: 16,116	On a side note, the site in your sig has been suspended. __________________ Your Paysite Partner Strength In Numbers! StickyDollars \| RadicalCash \| KennysPennies \| HomegrownCash

02-03-2010, 10:08 AM	#4
liquidmoe Confirmed User Join Date: Mar 2002 Location: NY Posts: 4,994	Sometimes when you have high CPU processing on httpd processes and nothing else it could be indicative of poorly structured rewrite rules in one of your .htaccess files. You may want to take a look at that. __________________ Take Luck!

02-03-2010, 10:48 AM	#5
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	About every 12 hours, or 18 or whenever, my server gets "stuck". uptime goes to load average: 150.26, 150.37, 151.32 and everything stops. I do a httpd stop, wait, then httpd start and it runs fine:load average goes back to 0.01 etc.....but only for another X hours then it sticks again. Completely random. Could be in 20 minutes, or 18 hours. I have been looking for it for about a month, and can't find what the deal is. I have shut down Mysql, suspended (almost) all accounts except for the account that gets most visitors, and it still gets stuck. I have a SINGLE static page, and bam, still crashes. No php, no nothing. I say "stuck" because it is still running, and I can shell to it, just that httpd is no longer processing requests, and doesnt flush out the old ones. It's stuck. I have hired 2 admins to help out, but both look at the server and are stumped after about an hour... __________________ www.gimmiegirlproductions.com

02-03-2010, 10:59 AM	#8
StaceyJo Confirmed User Join Date: Mar 2008 Posts: 8,960	So technical for me here. hehh __________________ /_______ WebCashMaker ______\ \| _TeenageDecadence - Young Board Naked Teens. \| \| ____ NonNudeGirls - Female Puberty Photos. ____ \| \| _ HerSelfPics - The ORIGINAL exGF SelfPic site. __ \| \.______ xPosing - Wife Photosharing site. _______./

02-03-2010, 10:50 AM	#7
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	Is that dedicated or shared hosting?

02-03-2010, 12:43 PM	#9
CYF Coupon Guru Industry Role: Join Date: Mar 2009 Location: Minneapolis Posts: 10,973	that top output looks fine Every 12 or 18 hours? You have anything setup in cron that is processor intensive? Have you checked logs for bots? Sometimes they hammer a site and bring it to a crawl. Are you running apache? What version? What OS? __________________ Webmaster Coupons Coupons and discounts for hosting, domains, SSL Certs, and more! AmeriNOC Coupons \| Certified Hosting Coupons \| Hosting Coupons \| Domain Name Coupons

02-03-2010, 02:16 PM	#11
borked Totally Borked Industry Role: Join Date: Feb 2005 Posts: 6,284	why/how is your apache instance serving up both HTTP/1.0 and HTTP/1.1 ??? Everything on srv1.ryd0.com is 1.0 whereas all the rest are 1.1 --edit nm, it's client based __________________ For coding work - hit me up on andy // borkedcoder // com (consider figuring out the email as test #1) All models are wrong, but some are useful. George E.P. Box. p202 Last edited by borked; 02-03-2010 at 02:20 PM..

02-03-2010, 02:23 PM	#12
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Hmm... just thinking about possible issues, and this one jumped out... why do you have mysql on the server if all pages are static? Do you run a script that rebuilds pages?

02-03-2010, 04:27 PM	#15
beta-tester Rock 'n Roll Baby! Join Date: Sep 2004 Location: USA, temporarly Posts: 22,562	It's interesting, but I do have the same issue as you described. The server load normally is totally ok, around .01, or so. But at some point of time during the day it also gets soo high, and the apache got stuck. Until I restart it, or let it do it by itself, the HTTP pretty much doesn't work at all. I tried to look into it but couldn't find anything that would cause this. It's really annoying ;( __________________ Sig for sale. Affordable prices. Contact me and get a great deal ;) My contact: ICQ: 944-320-46 e-mail: manca {AT} HotFreeSex4All.com

02-03-2010, 04:33 PM	#17
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	Holy shit I see a pattern in my hosts down!!! 4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world! That means a backup, a cron job. Backup or something. I predict a crash at 8:30 tonight! __________________ www.gimmiegirlproductions.com

02-03-2010, 04:37 PM	#19
Domain Broker So Fucking Banned Industry Role: Join Date: Oct 2004 Posts: 2,427	Do you have a shitty host? Why aren't they looking into this?

02-03-2010, 08:05 PM	#23
Ca$h-collector Confirmed User Join Date: Apr 2006 Location: ...heaven.. Posts: 134	interesting infos

02-03-2010, 11:53 PM	#24
borked Totally Borked Industry Role: Join Date: Feb 2005 Posts: 6,284	for every username in ls /var/spool/cron run (as root) crontab -u <username> -l to see which crontab is the culprit. __________________ For coding work - hit me up on andy // borkedcoder // com (consider figuring out the email as test #1) All models are wrong, but some are useful. George E.P. Box. p202

02-04-2010, 01:58 AM	#25
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	I tell you one thing though, I've learned ALOT about my server while trying to track this down. I have focused my skills on PHP and development, not so much admin, but I am actually getting confident in handing some admin functions now. __________________ www.gimmiegirlproductions.com

02-04-2010, 02:23 AM	#26
ilbb Confirmed User Industry Role: Join Date: May 2005 Location: EU - Czech republic Posts: 3,025	I recommend set cron every minute and log apache server status so you can see which pages/scripts are requested in the peaks. This will help you determine which script is causing server overload

02-04-2010, 07:59 AM	#29
SmellyNose Confirmed User Industry Role: Join Date: Aug 2009 Location: me at smellynose.com Posts: 206	30 /4 * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache Comment out this one. It runs at 30 minutes past the hour, every 4 hours Or, run it yourself and see how long it takes to run and if it causes your server to halt.

02-04-2010, 08:02 AM	#30
PastorSinAlot So Fucking Banned Industry Role: Join Date: Jun 2007 Location: Compton Posts: 2,837	choopa would knock this out for u

02-04-2010, 08:14 AM	#32
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	Or it is possible that the first task, takes more than 4 hours to run, and therefore the second time it is run, halts the server until the first is done. Causing the halt every 8 hours (2 tasks while second is waiting for first to finish). Hmmm... __________________ www.gimmiegirlproductions.com

02-04-2010, 09:53 AM	#33
SmellyNose Confirmed User Industry Role: Join Date: Aug 2009 Location: me at smellynose.com Posts: 206	It's unlikely the first task took that long, but it is possible. I didn't check your times properly (I skimmed the thread, so sue me :P ), so it is more than likely the second task that runs every 8 hours which is causing issues. Let us know how it goes

02-04-2010, 12:18 PM	#34
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	Well that wasn't it. Server uptime is 150.0 150.0 150.0 at 11:17am...Both cron jobs were removed.. Back to square one. Having a server that doesn't work, with so few visitors sucks. __________________ www.gimmiegirlproductions.com

02-07-2010, 12:11 PM	#35
beta-tester Rock 'n Roll Baby! Join Date: Sep 2004 Location: USA, temporarly Posts: 22,562	try checking MaxClients in Apache. I just figured out in some of the error logs that it says maxclients are riched, try increasing this value. I increased it, and now I'm monitoring to see what happens. __________________ Sig for sale. Affordable prices. Contact me and get a great deal ;) My contact: ICQ: 944-320-46 e-mail: manca {AT} HotFreeSex4All.com

02-08-2010, 08:04 AM	#37
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	The problem seems to be that apache threads are not exiting. They stay in the top of the output buffer. Here check it out. Here are some outputs as the server is slowly crashing... http://www.photofight.com/xx.txt http://www.photofight.com/xy.txt http://www.photofight.com/xz.txt See how the top thread is never leaving? __________________ www.gimmiegirlproductions.com Last edited by camperjohn64; 02-08-2010 at 08:09 AM..

02-09-2010, 09:22 AM	#38
SmokeyTheBear ►SouthOfHeaven Join Date: Jun 2004 Location: PlanetEarth MyBoardRank: GerbilMaster My-Penis-Size: extralarge MyWeapon: Computer Posts: 28,609	check if your getting botted netstat -anp \|grep ‘tcp\\|udp’ \| awk ‘{print $5}’ \| cut -d: -f1 \| sort \| uniq -c \| sort -n __________________ hatisblack at yahoo.com

02-09-2010, 03:17 PM	#40
HomerSimpson Too lazy to set a custom title Industry Role: Join Date: Sep 2005 Location: Springfield Posts: 13,826	you need to tweak your apache... __________________ Make a bank with Chaturbate - the best selling webcam program Ads that can't be block with AdBlockers !!! /// Best paying popup program (Bitcoin payouts) !!! PHP, MySql, Smarty, CodeIgniter, Laravel, WordPress, NATS... fixing stuff, server migrations & optimizations... My ICQ: 27429884 \| Email:

02-09-2010, 04:44 PM	#41
Varius Confirmed User Industry Role: Join Date: Jun 2004 Location: New York, NY Posts: 6,890	What do you have set for the following values and which version of Apache? MaxRequestsPerChild KeepAlive KeepAliveTimeout MaxKeepAliveRequests __________________ Skype variuscr - Email varius AT gmail

02-09-2010, 08:32 PM	#42
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	MaxClients 250 MaxRequestsPerChild 100 KeepAlive On KeepAliveTimeout 3 MaxKeepAliveRequests (not set) __________________ www.gimmiegirlproductions.com

02-09-2010, 09:10 PM	#43
fatfoo ICQ:649699063 Industry Role: Join Date: Mar 2003 Posts: 27,763	Server overloading - that's not good. Good luck. __________________ Send me an email: [email protected]