Stop throwing hardware at your problems. Tune your server!
 

I help a lot of people with server issues, script problems, server moves etc. One thing I keep seeing over and over again is people complaining about excessive server load from a script getting just a little bit of traffic.

Example: Last night I was going look into server load on a dual core server with 4GB of ram. The customer was ready to get a beefier server to handle the "serious traffic" (his words, not mine). He was getting less than 30k a day across all sites but server load during the morning was causing his server to start swapping.

Most of his traffic was on a wordpress site with a bunch of video embeds. Some traffic was from an old drupal site. His host--who just so happens to post here--was recommending a new server to him. After getting a lecture from the host, they finally gave up the root login.

Apache was configured so poorly that I am surprised it worked as well as it did. Mysql settings weren't that bad but really more appropriate for a server with less ram. After reconfiguring apache and mysql I made a couple more tweaks and the load was a fraction of what it was before. Then after adding w3 total cache to his wordpress there was barely any load at all. Page load times were infinitely better too.

Since we have root the host no longer wants to support the server which I guess is ok. They didn't do their job configuring it properly in the first place or making adjustments as necessary. I'm surprised how much it happens really. I guess hosts are more interested in selling the customer a more expensive server rather than get their current server working correctly.

Hosts, take care of your customers or someone else will.

What host?

name the host, if what you say is true then it's valuable information for other webmasters to know.

Couldn't agree more. I seem to spend half my time these days optimising servers for friends etc and I'm not suprised to hear the host recommended a new server to him, that seems to happen all too often these days. While I can understand why the host would do this for business reasons, I do wish more "webmasters" would understand that unless your in the Top 100 sites a quad-core-hyper-ninja box probably isn't required. After all, you could setup a P4 with 2GB box to handle 30k a day if you really wanted :winkwink:

I see alot of this too. Server's crashing until 30 seconds are spent changing apache timeouts, keepalives, max clients, etc.

Hummm, Like to know the host also!

I'm not really interested in naming the host since what they are doing isn't really all that unusual today. I also do some server admin work for another host and I shouldn't be badmouthing the competition.

I think a good warning sign is when the sales staff are more responsive to you than support is. Always be wary when your host suggests new hardware. Sometimes it really is necessary but a lot of the time it isn't.

and that's about all I did. I enabled query caching, did a few tweaks to my.cnf, installed a couple of apache mods and edited fstab. Literally an hour worth of work (would have been less but couldn't get a mod to compile). There was no rocket science involved. I did what any support geek at a host should have been able to do upon request.

Hmmm,

did the host set this all up for him? does he have a control panel?

I have spoken to many people on this bored and many others online and I do agree with you on the fact that most people do not require a beefy server. However I do not know how other hosts work but if the client does not have a control panel and has a load issue a lot of times we will fix it for them.

The host did everything. No control panel. He even had to open a ticket just to get a new DB or email added. That's what bothers me, this was 100% the host's responsibility.

Did you added noatime ?

So true. Don't throw hardware at a problem that hasn't even been diagnosed . It's not that difficult to track down the source of the load, often down to a specific query in a script.

Most servers don't even use noatime, which in many cases can cut load in half and takes three minutes to do.

For people wondering which host, ebmasters actually demand incompetence and demand that hosts sell you hardware you don't need. How many threads on GFY are posted looking for cheap hosting? Hire many threads have you seen asking which host had the most knowledgeable staff?

Witness the success if hostgator. Many people buy from them them. They have never seen the servers and don't have access to the datacenter. That's what people buy, hosting from a company that doesn't even know what the server looks like.

The market, the decisions webmasters make, forces a host who wants to have a customers to cut corners. They have to hire cheap techs and have those techs spend as little time as possible, because webmasters are looking for cheap hosting. When you ask for cheap, you often get it. Cheap often costs a lot more in the long fun. Right now we are moving a low traffic site from a $800 / month mammoth server they don't need but got "a good deal on" to a properly tuned $150 server designed to fit their needs.

Webmasters often choose to buy the most impressive sounding specs without ever asking what they need. An independent evaluation can tell you what you need.

Here's a hint - you almost surely don't need a faster CPU. After tuning, another or faster drive might be recommended, but the cpu is already a million times faster than the drive.

The tendency for webmasters to look for "cheap" (crap) hosting is why you don't see us advertising our hosting anywhere. After we get know someone and find they are good to do business with, we might invite them to come on board if they ask us about hosting. Normally we end up saving them money. That's not done by offering phoney unlimited bandwidth on a server with a single 5400 RPM drive and we have no interest in getting into those kinds if antics. That's the kind of stuff most webmasters buy, though. It's like the pop under, prechecked cross sale, one free with $50 recurring crap, but on the hosting side, and that's what sells.

Well said, Raymor.

Cyberwurx

I use these guys. My last server was a joke. Same types of problems. Quad core running a 3.x load all the time and sometimes spiking to 10.x!

Switched companies and combined two servers. With Cyberwurx my load is .04 to .1 at most. Same hardware. All of the changes were software related. Page loads went from 1.x/2.8x down to .26 seconds.

This is one of those threads that have little nuggets of wisdom and some solid infomation and advice a webmaster can use. Thanks for the tip on W3 Total Cache. Here's a bump for those that missed it.

interesting

Hmm I have a server with the same issues, but I don't know what changes I should make to the configuration. Where do you go to find this information?

You should talk to your host about it. If it's a self-managed server you should hire a server admin to make adjustments for you.

Babaganoosh or my team can do some really geeky stuff to see exactly what the issue is. If your host can tell you IN DETAIL exactly what the problem is, great.

Babaganoosh please send me your contact information at [email protected]. I value competency so I'd like to have your information on file.

Sure. Send contact details or hit me up.

All you, raymor. I've got too much on my plate right now.:winkwink:

I'll get in touch with you soon.

A lot of it is just using the right tool for the job.

I can get most of my server work done using this one simple tool:

http://www.finart.com/page4/files/hand-tools.jpg

Being he won't name them and he said they post here I'm going with Webair.

They also hold the record for most threads started complaining about their host.

Raymor, you guys offer unix admin services?

can I get your contact info.

or alternately, recommendations to other admin groups?
i'm just debating on another natnet server, or an unmanaged with a good outside admin...

+1 for Cyberwurx. I've been with them for almost 10 years now. Great support. They always look for software solutions before throwing more hardware at an issue.

I've used about 5+ hosting companies for dedicated, managed solutions and none of them had great unix admin support even though it was "managed".

I'm trying out a few outside services that I've found good reviews on from another mainstream forum
www.rack911.com
www.admingeekz.com

but looking to test others out.

I can vouch for admingeekz.com. He is really good. Expensive but good.

So, what is the going rate to tune a server?

I have been doing server admin and related tech work for thirteen years. I literally know Apache inside and out, being an active member of the Apache modules dev team. Elias recently joined our staff and is also a qualified admin. When needed, we have relationships work the principle developers if many major components of the server system. For example, for storage related issues we are active with LVM devel and so we can draw on the best minds in Linux storage technology.


[email protected] 979-530-1300

Writing this reply reminded me how lopsided we are. We REALLY need a marketing person because the people on international standards commmitees like IETF might know what we do better than our potential customers do.

id say natnet, since the topic starter said he asked for root, and once given the host didnt want to support the box anymore. natnet got the same policy in place ;)

It depends on several factors. Mostly it depends on how thorough of a job you want us to do. We could quickly go in and fix the most important and most common issues for $50. $150 would be more typical, evaluating the major bottlenecks on your specific server and addressing them, either by fixing them or making recommendations. For one customer pushing over 400 mbps, we roughly doubled his capacity in stage one, them improved it further as needed. At, the same time, we addressed other issues like backup. He probably saved $10,000 compared to throwing hardware at it. He paid a bit more than $150 for the multiple stages of tuning, recommendations, and other improvements, course.

I think Amerinoc has the same policy, there's a few other host who post here that probably have that same policy.

wow an informative thread on GFY! Even if you have a great host i often think its worthwhile getting an outsider in to check the server just to ensure its running its best. Ray, do your or your team work with nginx setup etcs?

I would be shocked if it was Natnet as they have always had a good reputation not to mention a bit pricey.

Wow, I didn't know you were still around. I haven't talked to you in years. You helped me set up a pornstar site YEARS ago. I need to talk to you about the site in your sig. I have a new project it would be great for.

Nope, definitely wasn't natnet. Never heard anything bad about them.

Best way to boost performance on most Lamp based apps is by lessening the load on the db and web server....
Do this by caching as much as possible.
Cache front end ie forward proxy such as Varnish or Nginx, php opcode caching..ie apc, application caching.ie w3cache or similar, object caching.. ie memcached, using query caching on the database... etc etc.

I've been researching and benchmarking different setups lately (mostly with wordpress and drupal) and it's amazing how much of a difference it makes. I've seen #/reqs per second increase 20x with the right tweaking....

Good Point, in fact Gartner reports that 70% of all infrastructure resources aren't being utilized in large part, because of this.

We go to great lengths to tune customer servers based on what the server is doing. In fact, as part of our continuing education program, our Director of Programming Services just gave a great class last week to all the System Administrators on identifying database bottlenecks and the importance of keys and indexes, as well as how to determine what tables need them and how to add them. There's tons of poor programming out there with things like bad "join" queries, etc. He showed us one example where a poorly formatted query took over 31 seconds to complete. After adding some keys, it dropped to 0.01 seconds - Nice!

Other examples include moving websites to nginx or lighttpd if we think that will work better, upgrading Mysql versions to take advantage of newer functionality such as partitioning in the newer versions of Mysql, converting older Myisam tables to Innodb (where applicable) as well as working with the many caching and accelerator programs available.

We never recommend upgrades or additional hardware without exhausting every tuning effort and in fact, I routinely talk customers out of adding additional servers or upgrades :thumbsup

-- Bill

Just don't cache static files. If it looks like caching your thumbnails is working, something is broken.

That "something" that's broken is often lighttpd's atime bug. Instead, turn on noatime and ditch lighttpd for static files.

Yes. Apache + nginx setups are often good candidates for tuning.

More than once, we've improved an Apache + nginx setup by running some monitoring to get baseline performance and load numbers. This is the "before" picture which tells what the server was like before tuning. Then we tune the OS and Apache settings. Finally, we ditch the unecesary extra layer (nginx) and log the "after" numbers. While nginx may sometimes be appropriate, most often it's the wrong solution to a poorly understood problem and a better Apache configuration performs better with far less hassle and wasted resources from running two web servers instead of one.

Occasionally, nginx actually turns out to be useful in the role of proxy, though squid, being designed as a proxy, tends to be better suited to that role.

If you happen to be religious about nginx, we can tune based on keeping it. Some people just "like" nginx, regardless of performance and bugs.

My icq is in the sig, feel free to say hi :)

QUOTED FOR TRUTH ... and the reason why we do not sell hosting with an affiliate program, and we do not usually advertise. A1 Web has survived in a cut throat market by word of mouth of happy clients - since 1998. :thumbsup

I think you should charge more, lol.

i agree you guys go great lengths, like i said in my reply the give out root thing was the only bells that rang in my head and the only company i could remember that had that policy in the past, dunno if it still is like that, was natnet ;) thats all, been with you guys in the past for over a year so i know how the support guys go out of their way to get things resolved.

I love those guys who insist on always getting a "dedicated" server.

If your host is competent - you can be on virtual and never notice the difference, unless you need absolute full control over the server and require root. Once again, like has been said before, it's all in the configuration.

True story. That's about what I would charge and I am pretty sure raymor knows more than I do.

Raymor great advice! I'm a firm believer that you get what you pay for in life, hosting is no exception.