Is Mem Caching crucial to sites with big traffic?

[lakerslive]

how important is it. Do u guys have it?

[rock-reed]

On a good dedi server, it is not important.


If you are with shared....you should use caching.

[MoreMagic]

I suppose you don't have any real big traffic sites....

Originally Posted by rock-reed

On a good dedi server, it is not important.


If you are with shared....you should use caching.

[darksoul]

depends on the apps used on the sites

[Matt 26z]

Shared hosting doesnt give you access to enough RAM to do it. If you are dedicated and hit the database a lot then it can speed things up since serving commonly accessed things from RAM is faster.

[doorag]

i would go with a vps before jumping into a dedicated server.

[PeR930]

If you are on a single server use APC. If your site is on a network of servers then use memcached.

[RSD]

Better use tmpfs for caching!

[NewNick]

Originally Posted by lakerslive

how important is it. Do u guys have it?

It is very important.

Anyone who says otherwise does not know what a high traffic site is.

[sandman!]

depends on the scripts your using.

[borked]

Originally Posted by NewNick

It is very important.

Anyone who says otherwise does not know what a high traffic site is.

Orly?
I have a site pushing 200mbs at peak daily and not a single memcached in site.
The site feeds from a central db server thatnserves other sites and pushes 20mbs over the internal net and not a single memcached in site.

As has been mentioned, it depends on the script...

[PeR930]

Bandwidth does not matter lol.

[C.Carnato]

How big is big traffic?

[PeR930]

At what point memcached shows its usefullness depends on your script and server. Can you get away without it with a million pageviews per day? Depends but either way you can probably still benefit from memcached so why not use it when it's free? Another way to look at it is a slightly slower server with memcached may perform just as good as a more expensive server without.

Like I said before if you are on a single server APC is typically much better.

[borked]

Originally Posted by ViPeR_930

Bandwidth does not matter lol.

You're right, which is why I'll shut you up with traffic in the form of apache accesses... can your server not running any caching average 200 accesses per second pushing 200mbs over an entire month without any problems?

--edit here just to clarify I run no caching daemon whatsoever ---

So STFU



Note, that was a snapshot I just made (time now for me 21:51) as you can tell from the relevant axes, so no photoshop or anything to hide

[borked]

and the bandwidth to follow on the same server...



Still need memcached?

How about the server load to show we don't need it?

[borked]

my bad, the bandwidth is only an average of 90mbs over a month. ooops.

[RSD]

It all depends!

[RSD]

First you should start optimizing your database keys with indexes. Then optize your database server config. For certain heavy queries you can create different tables or use some kind of caching. Memory tables in mysql can also be used as aome sort of caching.

[borked]

So let's recap what I was saying with my example...

over an entire month, an avg of 210 accesses/sec equals ~17 million daily hits, pushing ~90mbs or ~8.5Tb daily traffic with minimal server load without any memcached or anything like that ON A SINGLE SERVER.

Like RSD says:

Originally Posted by RSD

It all depends!

I run a tight ship with very slick code and well-prepared db queries (on highly optimised db indices). You may not.

So when NewNick says:

Originally Posted by NewNick

It is very important. Anyone who says otherwise does not know what a high traffic site is.

I say, you don't know what you are talking about.

[NewNick]

Originally Posted by borked

So let's recap what I was saying with my example...

over an entire month, an avg of 210 accesses/sec equals ~17 million daily hits, pushing ~90mbs or ~8.5Tb daily traffic with minimal server load without any memcached or anything like that ON A SINGLE SERVER.

Like RSD says:


I run a tight ship with very slick code and well-prepared db queries (on highly optimised db indices). You may not.

So when NewNick says:



I say, you don't know what you are talking about.

Well arn't you a big cheese.

I suppose I don't know what I am talking about.

I do know that if we were down to 90mbits I would be hitting the panic button.

[borked]

I am saying that it all depends on what script you are running. I gave raw data on a single server running no memcached and running the scripts quite happily.

If you are pushing more than a monthly 90mbs average on a single server with memcached, care to turn off memcached and post the munin logs here?

I'm not getting into any forum fight, which is why I talk with actions (ie graphs) so as to show I am not hiding or bullshitting. But those munin graphs I posted show quite clearly I could (if I could!!) push much much more than 90mbs on that site. But truth be told, I probably couldn't because the max is 200mbs, and as it only has a gbit card, I'd be stupid to let it run anything over than 6-700mbs max (~3x what it is now). But I'm happy knowing that the scripts and db can handle all that, as can the server, without any mem caching.

And no, I'm no big cheese - I simply make my money without mouthing off about it. If you know me, you know I only ever mouth off about technical points, this particular one being one of them.

[iwiiiiiiiiii]

Memcache is useful when for each visitor you have to do many db calls and load many items in that page. If you just have few pics and videos, but not stats or anything else, it's not really necessary. In my opinion it's really to avoid overloading your db server.

[blazin]

How much bandwidth your site is pushing isn't that relevant....

If your site is doing lots of CPU/DB intensive processing on each request, and those requests don't change a lot from visitor to visitor... then yes memcached can help.

And for those that mention APC as an alternative... it's not.. well not really... APC primary purpose is an Opcode Cache... in simple terms, it makes PHP run faster by doing away with the need to do a full compile of your code on every request. Some programmers will use it as general purpose cache also.

[PeR930]

Originally Posted by borked

You're right, which is why I'll shut you up with traffic in the form of apache accesses... can your server not running any caching average 200 accesses per second pushing 200mbs over an entire month without any problems?

--edit here just to clarify I run no caching daemon whatsoever ---

So STFU



Note, that was a snapshot I just made (time now for me 21:51) as you can tell from the relevant axes, so no photoshop or anything to hide

Butthurt much? Maybe you got me confused the other guy. I simply said data bandwidth is irrelevant to memcached, which you agreed with. I even agreed with you that it depends on the script. What more do you want? Obviously it also depends on the server hardware and COOL it looks like you have a beastly server. You can calm down now. Is there a reason why you seem so against memcached? Maybe it bullied you in junior high.

[livexxx]

Insall new relic, half the problem is with shit devs thinking they are better than dba's

[livexxx]

Originally Posted by livexxx

Insall new relic, half the problem is with shit devs thinking they are better than dba's

the other half of the problem are dev's who don't even know what a dba is, lol

[blazin]

Originally Posted by livexxx

Insall new relic, half the problem is with shit devs thinking they are better than dba's

New Relic is awesome... but damn it's get's expensive.

[EddyTheDog]

Originally Posted by borked

I am saying that it all depends on what script you are running. I gave raw data on a single server running no memcached and running the scripts quite happily.

If you are pushing more than a monthly 90mbs average on a single server with memcached, care to turn off memcached and post the munin logs here?

I'm not getting into any forum fight, which is why I talk with actions (ie graphs) so as to show I am not hiding or bullshitting. But those munin graphs I posted show quite clearly I could (if I could!!) push much much more than 90mbs on that site. But truth be told, I probably couldn't because the max is 200mbs, and as it only has a gbit card, I'd be stupid to let it run anything over than 6-700mbs max (~3x what it is now). But I'm happy knowing that the scripts and db can handle all that, as can the server, without any mem caching.

And no, I'm no big cheese - I simply make my money without mouthing off about it. If you know me, you know I only ever mouth off about technical points, this particular one being one of them.

That's all great, however a lot of people use scripts that are not as good and memory management - There are times when MemCache or APC is a good idea.

Another idea I have been playing with is pushing generated content to an NGinX Proxy from an APache server and caching it - Amazing results with WP.

[livexxx]

Originally Posted by blazin

New Relic is awesome... but damn it's get's expensive.

damn fucking aggresive sales team, but we beat them down to $50 a server with some big main chunk payment. well worth it IMHO and all the servers get it and from now on in as well

not sure what they have as sales incentives, but just stand your ground and they cave instantly and we got 5 t-shirts and 3 helicopters too, LOL

[RSD]

Optimising your own code will suffice most of the time! Memcache can come in handy for static parts of your website. Memory reads are cheap, memory writes are troublesome.

Wordpress and magento are overkill for most of the websites, just build your own custom weblog or webshop.

[borked]

Originally Posted by ViPeR_930

You can calm down now. Is there a reason why you seem so against memcached? Maybe it bullied you in junior high.

Yes, I mistook your post as it came right after mine mentioning 200mbs etc -
I actually was making a point to counter Mr-Know-It-All-NewNick who seems to know that anyone with serious traffic knows that memcache is de rigeur, which my posts clearly demonstrate otherwise.

I've nothing against memcached, but I just think that it's kind of like painting over cracks... not really getting to the source of the problem.
As mentioned numerous times here, a well designed db and well designed queries are much more important. If your queries are using the indices, you are pulling from ram and not disk, which is what memcached does anyway. People may not control their code, but they do control their db... log slow queries to find rogue queries is the least one should be doing.

Anyway, at least my posts served to silence NewNick with his throw away "blah blah" statements, that do more harm than good to those that don't know better.

[ajrocks]

If you are doing something like dating where you need to provide geo targeting on a creative mem cache will be your best friend. Really you are wasting a lot of load if you aren't using it. Anyone with a decent amount of traffic will get increases in performance and decreased in costs.

[Varius]

In almost all cases, bad site performance stems from bad DB design and indexing.

In such cases, caching can be a great band-aid, but solving the core problems is always better in the long-run.

Usually, DB problems fall into a few areas:

1. Lack of indexing
2. Over-indexing
3. Incorrect use of Subqueries (esp. MySQL struggles with these)
4. Incorrect column types for the data's purpose

1) Many people just aren't experienced with indexing, or even if they are, don't use things like composite indexes when needed.

2) Conversations I've had with people regarding over-indexing are hilarious. They are truly clueless. Some people, simply index almost everything they can, to "cover their bases". This is NOT how indexes are meant to work. Another common error, is having overlapping indexes when not needed.

For example, say I have an index on (col1, col2, col3) and my application NEVER searches/orders by etc... on col2 or col3 by themselves - having individual indexes on actually hurts me more than it helps.

3) Almost every case of sub-queries is better off with the proper JOINs. Especially in MySQL. The cases you want sub-queries are if you are simply doing something light/quick/one-time, OR very advanced cases which probably deserve their own article

4) This one is pretty basic and a lot of people get it right, where if you know the expected data values for a column, use the "tightest" column type. For example, if you know you store country codes as 2 characters, use char(2) don't use default varchar(255).

Enum doesn't get used as often as it should. It's a great type many people shy away from as they don't like having to alter a DB table that's in production (understandable). However, as long as you are adding values and not changing the order of existing ones, the ALTER is near-instant (I've done it on multi-million record large tables, being hit intensively, in under 0.05 seconds). For me, the benefits of enum outweight the cons when used properly.

Lastly, a tip on people who index and search varchar fields, such as email_address.

Have an additional hash column and index and seek on that.

For a basic example, let's say your Admin area let's you seek customers out by email. You have email_address varchar(150). You could add email_address_md5 and make your query do:

WHERE unhex(md5(lower($email_being_searched)))

and the speed gains will be *quite* noticeable. You can also use more advanced hashes and multi-column hashes/algos, and flavor to taste