Wowza kicks ass - 500mbs and the server isn't even sweating
 

A while ago, I had a public argument with someone who dissed WowzaMediaServer saying it was shite and crapped out continuously. I disagree saying he had badly configured it. He went nuts saying he was an expert sys admin and it was because wowza was written in java, it couldn't take the load blah blah

Well, here's the proof that Wowza kicks ass. Just peaked bandwidth at 500mbs, which is pretty much pure Wowza streaming - apache is dissing up only ~10mbs.
Wowza is consuming only 1.2GB of ram to do this, and the server is barely breaking sweat.

some pics for the geeks...

http://www.borkedcoder.com/images/gfy/load/bw.gif

http://www.borkedcoder.com/images/gf...che_volume.gif

http://www.borkedcoder.com/images/gfy/load/ram.gif

http://www.borkedcoder.com/images/gfy/load/cpu.gif

http://www.borkedcoder.com/images/gfy/load/load.gif

Right.

Cool story bro.

yeah Wowza kicks ass, only don't play too much with the JAVA heap settings.

nor garbage collection... :thumbsup

Yeah that gave me some sleepless nights, but all running pretty stable now :winkwink:

we have been seeing really good results with wowza too
pretty solid software

Good to know. Java is generally known to be slow and and a memory hog but it seems to be working out for you. The Wowza FAQ says to start you need at least a quad core system with 4 GB of RAM as the baseline minimum, going up from there based on traffic, and preferably RAID 10, their FAQ says. What kind of hardware are you using?

I want to hire you i just don't know what for.

I love you, borked ...

Well, I've configured it to use 12GB if it wants, but it's quite happy using 1.2GB when the throughput was max, so they did something right with wowza3 and memory management. Wowza2 wasn't so nice...

the server is a bit of a beast:

Intel Server motherboard S5000PSL INTEL BX80574E5420A E5420 2.5Ghz quadcore x2
6x 4gb Kingston DDR2 667 fully buffered
2x SSD disk Intel 40GB for system
24 x Western Digital Raid Edition 4 2TB ADAPTEC 3805 up to 256 disk Raid 1 hardware

Of course, wowza is running on SSDs, streaming content from 24TB array, so the SSDs surely help here.

--edit here it is:

:1orglaugh:1orglaugh :thumbsup

Wowza works wonders... if you feed it the ram.

Wowza by nature will destroy disk IO if it cannot buffer the content, if the content is popular and not being sucked off the drive and cached then it will utterly destroy a server.

Not sure exactly but Wowza normally decreases bandwidth by about 30-40%.. not increase bandwidth. Alteast thats the case when people switch from nginx/lighttpd.

Unless they changed something in wowza, it feeds the content off the folder you tell it.. it doesn't care its on SSD it cares where the content is, i don't recall any sort of buffering/caching system to place this on ssd from other content.

Not trying to bash things but if your using 24tb servers for streaming content, your doing it wrong.

We have wowza servers running almost double what you are managing, 100% live streaming. (up to the capacity of the switch)

Wowza is absolutely stable. You just have to know how to use it. Once you get your settings fine tuned you can run thousands of similtaneous connections on one reasonable spec server.

:thumbsup

:thumbsup

No I'm not. :thumbsup

That's good to know! Things have been stable around 200mbs for a while so this new flood was great to see. It's even better to know that she won't flake if we max out the nic!

btw, what happens when you do that? Does the fact that traffic can't get through as the nic is full have an impact on server load? It must do, because those packets have to go somewhere... just I've never been in the position to see NIC meltdown before :thumbsup

Ok, enjoy scaling that and enjoy the cost of the hardware to scale.

When you exceed a port you start dropping packets, TCP by nature will try to retransimit, eventually the packets will stop re-transmitting and will frankly drop, causing downloads to simply slow down, buffer and just stop working. If your using RTMFP, it uses UDP, and at that point your just screwed :)

500mbps? are you running a tube?

There are 4 other servers doing other things and are also under-loaded that I can load balance to if I want (2 are even configured as edge servers, but not in use). But there is no need atm.
The eth1 nic connected to the lan is only pushing 85mbs content to them. I just don't want to put wowza on them yet for my own reasons.

You don't know what this setup is, nor how it is configured, so you are in no place to tell me I'm doing things wrong....

We also use Wowza, and agree with your wholeheartedly fine sire.

:thumbsup

Zak does that often.

I'll find something sooner or later :winkwink:

Kinky setup :D

Still need to switch to Wowza3.

I run Wowza2 the following config
Supermicro 6026T-3RF XEON i5520 2U Barebone w/ 720W Redundant PSU (Black)
Dual Intel 5600 / 5500 series Xeon Quad/Dual-Core
4x Seagate Barracuda 7200.11 (RAID 5)
6 GB memory

Server uses a 1 TB NAS RAID5 storage device

Your right, clearly we have no clue since we only run about, some retarded amount of large tube sites and get referrals for such hosting every day.

My point is simply there are most cost effective ways to scale, than having such large/expensive machines doing all the work. We like saving people money, no point in paying for things or buying things that are not needed. Unlike a lot of other hosts we care about how machines perform and don't like grinding people into upgrading because the original hardware/design was not sufficient for what their goals were.

You really should upgrade - it costs nothing (even perpetual wms2 licenses upgrade for free) and takes all of 2-3 minutes, with the only downtime being the time it takes to stop wowza, change the symlink to wms3 then restart.

Yeah I know, will do it somewhere next week :thumbsup

look, I've already argued with you once on ovh. Now you stick your neck in on wowza.

This server isn't expensive and is a shit loss expensive than a cluster to reach 24TB.

I fully know how to run a server cluster to be able to scale size. This isn't my server setup, it is my client's and I know his needs and a 24TB system is plenty for his needs.

You run a different operation, so your needs are different too.

As I said, Zak does this a lot.

He is a smart guy, but has a major chip or ego when it comes to the master of the universe thing. You will find yourself locking horns more often then not. I am not saying he is right or wrong. There are many different ways to skin a cat. However, you will find that arguing with him is like pissing in the wind.

No offense.

:2 cents:

Then you should of been aware that that Adaptec 3805 card is a piece of junk with only 128mb cache and 3G/s. Your running 6G/s drives and that card is going to become a huge problem down the road for you. Something like the 3ware or lsi 9750/9260 would of been a much better fit for this, since your drives are 6G/s drives and your bottleneck is 3G/s on that card.

Look I'm just trying to help. But its pretty apparent that no one appreciates feedback that goes against their own thinking/mentality.

Also running with OVH !!!


:thumbsup

yeah, you're right. Everyone's needs are very different and like you say, there are more ways than one to fuck^H^H^H^H skin a cat...

I like managed hosting, but like I say to my clients, you can't beat a personal sysadmin. It's like having your own PA - they know your needs and system inside out better than any managed hosting company can ever do. Simply because, one does many in a general way, the other does few in a more personalised way.

I do not think anyone minds constructive feedback. However, the way you go about it is condescending or comes off like master of the universe. If you want people to be more receptive to your feedback, you need to work on the presentation.

:2 cents:

I will admit, I am rather blunt and to the point. Some people like it some people hate it. And yes this is why I've hired a few people to do much of the things I use to do and have removed myself from a lot of this process in our operations :)

You provide an excellent service, no one is saying otherwise.

But as you well know in the hosting industry.... it seems few do it the same exact way. You can take 10 different random companies, whether piss ants or well known, and find that most of them manage their shit in ways that baffle the imagination. It has been a real eye opening experience in that regard to be sure. You would think there is more uniformity.

But I digress...

I feel your pain fine sire...

:2 cents:

That isn't true - I do appreciate constructive feedback. That adaptec card is no longer in there (I copy/pasted that from the orig invoice when there were 12x2TB drives in there).
To grow to 24, the card was replaced along with something else I forget. I'd have to dig through the emails to find what exactly it was, but I cannot be bothered. Suffice it to say, to scale from 12 to 24TB cost the price of buying 12x2TB server-grade disks. This server can scale to another 48 without haste, so I believe, with the possibility to grow that to 2x96 if required. I'll cross that bridge when I come to it, but for now, we're a long way off filling 24TB, which is a shitload of content.

The hw side of things I leave to others who are better qualified and in the know than I - I know my limits.

But thanks for your constructive feedback. :thumbsup

Bang on :thumbsup
I'm currently migrating someone away from a managed solution to their own private setup so they aren't tied down and, while not naming the company that is really well known here, I am astounded at how they setup the servers....

Anyway, I really didn't want this to become a cock-battling thread of hosting companies! I just want more people to move over to wowza... cos I have an ulterior motive :winkwink:

That makes sense because wowza is a real streaming server while the other flavour is pseudo streaming. With real streaming, you are simply delivering the content (bandwidth) "on demand" and no more, no less, whereas with pseudo, you stream from a end-user defined point and just diss up the content, regardless of where the user is up to in watching, so you're always "wasting" bandwidth unless a user watches the entire video from beginning to end non-stop.

Hello 1999.

Java isn't that bad these days.

For the moment whatever card is in there (can't find it with dmidecode - is there another tool to check with?), the IO wait is very respectable for peak times i think: await is 2.9 and %util is 35

http://www.borkedcoder.com/images/gfy/load/iowait.gif

lspci should show you what cards are currently installed.

Are you using lvm to concatenate the sdd and sdc together? or are they separate? It looks like it is, once you cross the barrier of the disk usage on the sdd and cross into disk space on the sdc partition/set you will see a breath of fresh air, currently your supporting the disk IO on one raid set. I could be wrong but thats what it looks like.

Disk contention is overcame by usually creating smaller raid sets on large spindle sizes, i.e 8x groups of 6 drives and if you concatenate them via lvm as you add content it will spread to different spindles. Once data is contained and "cached" via the system it wont seek them since it knows where they are on that raidset. + you can utilize the platters more efficiently because you have now 8 sets of disks doing various IO tasks vs 1-2 sets doing the same task. MUCH better disk IO/throughput that way.

yup, it's still on the adaptec
for the moment, c&d are separate yes. I'll look into lvm

also depending on the size of your files, they could be a few mb or afew gb in size.. adjsut the drive readahead size.

i.e hdparm -a /dev/sdd will show a result.. its probalby 256. Jack that up to 16384 i.e hdparm -a 16384 /dev/sdd so its doing by far less reads when its grabbing data for distribution. this value can be tune/adjusted for whatever your traffic/source files are like. Makes a world of a difference on large files.

Also blockdev command works great too, shorter versionof using hadparm. blockdev --report will give some information on blocksize, make the readahead a multiplier of the blocksize.

OK, now I'm learning stuff :thumbsup

set the readahead to 16384. I'll give it a bit of time to see how that affects the IO reads.

Cheers!

Spudstr do you have any server specials?

contact me on icq or email me [email protected] and I'll see what we have :)

Quite often.. I see him get into pissing matches about things he has no idea about, yet he's still arguing the master of the universe theory.

I love you all - GFY is just a big disfunctional family growing stronger than the others around them :thumbsup

Nice Borked! We love Wowza too and found that we can do nearly 1Gbps with wowza with our cloud servers. where else can you find an application that can operate at nearly line speed performance? :pimp