Wowza kicks ass - 500mbs and the server isn't even sweating

[borked]

A while ago, I had a public argument with someone who dissed WowzaMediaServer saying it was shite and crapped out continuously. I disagree saying he had badly configured it. He went nuts saying he was an expert sys admin and it was because wowza was written in java, it couldn't take the load blah blah

Well, here's the proof that Wowza kicks ass. Just peaked bandwidth at 500mbs, which is pretty much pure Wowza streaming - apache is dissing up only ~10mbs.
Wowza is consuming only 1.2GB of ram to do this, and the server is barely breaking sweat.

some pics for the geeks...

[Dirty F]

Right.

Cool story bro.

[SZNY]

yeah Wowza kicks ass, only don't play too much with the JAVA heap settings.

[borked]

Originally Posted by SZNY

yeah Wowza kicks ass, only don't play too much with the JAVA heap settings.

nor garbage collection...

[SZNY]

Originally Posted by borked

nor garbage collection...

Yeah that gave me some sleepless nights, but all running pretty stable now

[AdultEUhost]

we have been seeing really good results with wowza too
pretty solid software

[raymor]

Good to know. Java is generally known to be slow and and a memory hog but it seems to be working out for you. The Wowza FAQ says to start you need at least a quad core system with 4 GB of RAM as the baseline minimum, going up from there based on traffic, and preferably RAID 10, their FAQ says. What kind of hardware are you using?

[Dirty F]

Originally Posted by borked

nor garbage collection...

I want to hire you i just don't know what for.

[CaptainHowdy]

I love you, borked ...

[borked]

Originally Posted by raymor

Good to know. Java is generally known to be slow and and a memory hog but it seems to be working out for you. The Wowza FAQ says to start you need at least a quad core system with 4 GB of RAM as the baseline minimum, going up from there based on traffic, and preferably RAID 10, their FAQ says. What kind of hardware are you using?

Well, I've configured it to use 12GB if it wants, but it's quite happy using 1.2GB when the throughput was max, so they did something right with wowza3 and memory management. Wowza2 wasn't so nice...

the server is a bit of a beast:

Intel Server motherboard S5000PSL INTEL BX80574E5420A E5420 2.5Ghz quadcore x2
6x 4gb Kingston DDR2 667 fully buffered
2x SSD disk Intel 40GB for system
24 x Western Digital Raid Edition 4 2TB ADAPTEC 3805 up to 256 disk Raid 1 hardware

Of course, wowza is running on SSDs, streaming content from 24TB array, so the SSDs surely help here.

--edit here it is:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                        
 2035 wowza      21   0 12.6g 721m  10m S 46.1  3.0 743:32.47 java

[borked]

Originally Posted by Dirty F

I want to hire you i just don't know what for.

[Spudstr]

Wowza works wonders... if you feed it the ram.

Wowza by nature will destroy disk IO if it cannot buffer the content, if the content is popular and not being sucked off the drive and cached then it will utterly destroy a server.

Not sure exactly but Wowza normally decreases bandwidth by about 30-40%.. not increase bandwidth. Alteast thats the case when people switch from nginx/lighttpd.

[Spudstr]

Originally Posted by borked

Well, I've configured it to use 12GB if it wants, but it's quite happy using 1.2GB when the throughput was max, so they did something right with wowza3 and memory management. Wowza2 wasn't so nice...

the server is a bit of a beast:

Intel Server motherboard S5000PSL INTEL BX80574E5420A E5420 2.5Ghz quadcore x2
6x 4gb Kingston DDR2 667 fully buffered
2x SSD disk Intel 40GB for system
24 x Western Digital Raid Edition 4 2TB ADAPTEC 3805 up to 256 disk Raid 1 hardware

Of course, wowza is running on SSDs, streaming content from 24TB array, so the SSDs surely help here.

--edit here it is:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                        
 2035 wowza      21   0 12.6g 721m  10m S 46.1  3.0 743:32.47 java

Unless they changed something in wowza, it feeds the content off the folder you tell it.. it doesn't care its on SSD it cares where the content is, i don't recall any sort of buffering/caching system to place this on ssd from other content.

Not trying to bash things but if your using 24tb servers for streaming content, your doing it wrong.

[NewNick]

Originally Posted by borked

A while ago, I had a public argument with someone who dissed WowzaMediaServer saying it was shite and crapped out continuously. I disagree saying he had badly configured it. He went nuts saying he was an expert sys admin and it was because wowza was written in java, it couldn't take the load blah blah

Well, here's the proof that Wowza kicks ass. Just peaked bandwidth at 500mbs, which is pretty much pure Wowza streaming - apache is dissing up only ~10mbs.
Wowza is consuming only 1.2GB of ram to do this, and the server is barely breaking sweat.

some pics for the geeks...

We have wowza servers running almost double what you are managing, 100% live streaming. (up to the capacity of the switch)

Wowza is absolutely stable. You just have to know how to use it. Once you get your settings fine tuned you can run thousands of similtaneous connections on one reasonable spec server.

[borked]

Originally Posted by Spudstr

Not trying to bash things but if your using 24tb servers for streaming content, your doing it wrong.

No I'm not.

[borked]

Originally Posted by NewNick

We have wowza servers running almost double what you are managing, 100% live streaming. (up to the capacity of the switch)

Wowza is absolutely stable. You just have to know how to use it. Once you get your settings fine tuned you can run thousands of similtaneous connections on one reasonable spec server.

That's good to know! Things have been stable around 200mbs for a while so this new flood was great to see. It's even better to know that she won't flake if we max out the nic!

btw, what happens when you do that? Does the fact that traffic can't get through as the nic is full have an impact on server load? It must do, because those packets have to go somewhere... just I've never been in the position to see NIC meltdown before

[Spudstr]

Originally Posted by borked

No I'm not.

Ok, enjoy scaling that and enjoy the cost of the hardware to scale.

[Spudstr]

Originally Posted by borked

That's good to know! Things have been stable around 200mbs for a while so this new flood was great to see. It's even better to know that she won't flake if we max out the nic!

btw, what happens when you do that? Does the fact that traffic can't get through as the nic is full have an impact on server load? It must do, because those packets have to go somewhere... just I've never been in the position to see NIC meltdown before

When you exceed a port you start dropping packets, TCP by nature will try to retransimit, eventually the packets will stop re-transmitting and will frankly drop, causing downloads to simply slow down, buffer and just stop working. If your using RTMFP, it uses UDP, and at that point your just screwed

[DarkJedi]

500mbps? are you running a tube?

[borked]

Originally Posted by Spudstr

Ok, enjoy scaling that and enjoy the cost of the hardware to scale.

There are 4 other servers doing other things and are also under-loaded that I can load balance to if I want (2 are even configured as edge servers, but not in use). But there is no need atm.
The eth1 nic connected to the lan is only pushing 85mbs content to them. I just don't want to put wowza on them yet for my own reasons.

You don't know what this setup is, nor how it is configured, so you are in no place to tell me I'm doing things wrong....

[Barefootsies]

We also use Wowza, and agree with your wholeheartedly fine sire.

Originally Posted by borked

You don't know what this setup is, nor how it is configured, so you are in no place to tell me I'm doing things wrong....

Zak does that often.

[Dirty F]

Originally Posted by borked

I'll find something sooner or later

[SZNY]

Originally Posted by borked

Well, I've configured it to use 12GB if it wants, but it's quite happy using 1.2GB when the throughput was max, so they did something right with wowza3 and memory management. Wowza2 wasn't so nice...

the server is a bit of a beast:

Intel Server motherboard S5000PSL INTEL BX80574E5420A E5420 2.5Ghz quadcore x2
6x 4gb Kingston DDR2 667 fully buffered
2x SSD disk Intel 40GB for system
24 x Western Digital Raid Edition 4 2TB ADAPTEC 3805 up to 256 disk Raid 1 hardware

Of course, wowza is running on SSDs, streaming content from 24TB array, so the SSDs surely help here.

--edit here it is:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                        
 2035 wowza      21   0 12.6g 721m  10m S 46.1  3.0 743:32.47 java

Kinky setup :D

Still need to switch to Wowza3.

I run Wowza2 the following config
Supermicro 6026T-3RF XEON i5520 2U Barebone w/ 720W Redundant PSU (Black)
Dual Intel 5600 / 5500 series Xeon Quad/Dual-Core
4x Seagate Barracuda 7200.11 (RAID 5)
6 GB memory

Server uses a 1 TB NAS RAID5 storage device

[Spudstr]

Originally Posted by borked

There are 4 other servers doing other things and are also under-loaded that I can load balance to if I want (2 are even configured as edge servers, but not in use). But there is no need atm.
The eth1 nic connected to the lan is only pushing 85mbs content to them. I just don't want to put wowza on them yet for my own reasons.

You don't know what this setup is, nor how it is configured, so you are in no place to tell me I'm doing things wrong....

Your right, clearly we have no clue since we only run about, some retarded amount of large tube sites and get referrals for such hosting every day.

My point is simply there are most cost effective ways to scale, than having such large/expensive machines doing all the work. We like saving people money, no point in paying for things or buying things that are not needed. Unlike a lot of other hosts we care about how machines perform and don't like grinding people into upgrading because the original hardware/design was not sufficient for what their goals were.

[borked]

Originally Posted by SZNY

Kinky setup :D

Still need to switch to Wowza3.

I run Wowza2 the following config
Supermicro 6026T-3RF XEON i5520 2U Barebone w/ 720W Redundant PSU (Black)
Dual Intel 5600 / 5500 series Xeon Quad/Dual-Core
4x Seagate Barracuda 7200.11 (RAID 5)
6 GB memory

Server uses a 1 TB NAS RAID5 storage device

You really should upgrade - it costs nothing (even perpetual wms2 licenses upgrade for free) and takes all of 2-3 minutes, with the only downtime being the time it takes to stop wowza, change the symlink to wms3 then restart.

[SZNY]

Originally Posted by borked

You really should upgrade - it costs nothing (even perpetual wms2 licenses upgrade for free) and takes all of 2-3 minutes, with the only downtime being the time it takes to stop wowza, change the symlink to wms3 then restart.

Yeah I know, will do it somewhere next week

[borked]

Originally Posted by Spudstr

Your right, clearly we have no clue since we only run about, some retarded amount of large tube sites and get referrals for such hosting every day.

My point is simply there are most cost effective ways to scale, than having such large/expensive machines doing all the work. We like saving people money, no point in paying for things or buying things that are not needed. Unlike a lot of other hosts we care about how machines perform and don't like grinding people into upgrading because the original hardware/design was not sufficient for what their goals were.

look, I've already argued with you once on ovh. Now you stick your neck in on wowza.

This server isn't expensive and is a shit loss expensive than a cluster to reach 24TB.

I fully know how to run a server cluster to be able to scale size. This isn't my server setup, it is my client's and I know his needs and a 24TB system is plenty for his needs.

You run a different operation, so your needs are different too.

[Barefootsies]

Originally Posted by borked

look, I've already argued with you once on ovh. Now you stick your neck in on wowza.

As I said, Zak does this a lot.

He is a smart guy, but has a major chip or ego when it comes to the master of the universe thing. You will find yourself locking horns more often then not. I am not saying he is right or wrong. There are many different ways to skin a cat. However, you will find that arguing with him is like pissing in the wind.

No offense.

[Spudstr]

Originally Posted by borked

look, I've already argued with you once on ovh. Now you stick your neck in on wowza.

This server isn't expensive and is a shit loss expensive than a cluster to reach 24TB.

I fully know how to run a server cluster to be able to scale size. This isn't my server setup, it is my client's and I know his needs and a 24TB system is plenty for his needs.

You run a different operation, so your needs are different too.

Then you should of been aware that that Adaptec 3805 card is a piece of junk with only 128mb cache and 3G/s. Your running 6G/s drives and that card is going to become a huge problem down the road for you. Something like the 3ware or lsi 9750/9260 would of been a much better fit for this, since your drives are 6G/s drives and your bottleneck is 3G/s on that card.

Look I'm just trying to help. But its pretty apparent that no one appreciates feedback that goes against their own thinking/mentality.

[NewNick]

Originally Posted by borked

look, I've already argued with you once on ovh. Now you stick your neck in on wowza.

This server isn't expensive and is a shit loss expensive than a cluster to reach 24TB.

I fully know how to run a server cluster to be able to scale size. This isn't my server setup, it is my client's and I know his needs and a 24TB system is plenty for his needs.

You run a different operation, so your needs are different too.

Also running with OVH !!!

[borked]

Originally Posted by Barefootsies

As I said, Zak does this a lot.

He is a smart guy, but has a major chip or ego when it comes to the master of the universe thing. You will find yourself locking horns more often then not. I am not saying he is right or wrong. There are many different ways to skin a cat. However, you will find that arguing with him is like pissing in the wind.

No offense.

yeah, you're right. Everyone's needs are very different and like you say, there are more ways than one to fuck^H^H^H^H skin a cat...

I like managed hosting, but like I say to my clients, you can't beat a personal sysadmin. It's like having your own PA - they know your needs and system inside out better than any managed hosting company can ever do. Simply because, one does many in a general way, the other does few in a more personalised way.

[Barefootsies]

Originally Posted by Spudstr

Look I'm just trying to help. But its pretty apparent that no one appreciates feedback that goes against their own thinking/mentality.

I do not think anyone minds constructive feedback. However, the way you go about it is condescending or comes off like master of the universe. If you want people to be more receptive to your feedback, you need to work on the presentation.

[Spudstr]

Originally Posted by Barefootsies

I do not think anyone minds constructive feedback. However, the way you go about it is condescending or comes off like master of the universe. If you want people to be more receptive to your feedback, you need to work on the presentation.

I will admit, I am rather blunt and to the point. Some people like it some people hate it. And yes this is why I've hired a few people to do much of the things I use to do and have removed myself from a lot of this process in our operations

[Barefootsies]

Originally Posted by borked

I like managed hosting, but like I say to my clients, you can't beat a personal sysadmin. It's like having your own PA - they know your needs and system inside out better than any managed hosting company can ever do. Simply because, one does many in a general way, the other does few in a more personalised way.

You provide an excellent service, no one is saying otherwise.

But as you well know in the hosting industry.... it seems few do it the same exact way. You can take 10 different random companies, whether piss ants or well known, and find that most of them manage their shit in ways that baffle the imagination. It has been a real eye opening experience in that regard to be sure. You would think there is more uniformity.

But I digress...

[Barefootsies]

Originally Posted by Spudstr

I will admit, I am rather blunt and to the point. Some people like it some people hate it. And yes this is why I've hired a few people to do much of the things I use to do and have removed myself from a lot of this process in our operations

I feel your pain fine sire...

[borked]

Originally Posted by Spudstr

Look I'm just trying to help. But its pretty apparent that no one appreciates feedback that goes against their own thinking/mentality.

That isn't true - I do appreciate constructive feedback. That adaptec card is no longer in there (I copy/pasted that from the orig invoice when there were 12x2TB drives in there).
To grow to 24, the card was replaced along with something else I forget. I'd have to dig through the emails to find what exactly it was, but I cannot be bothered. Suffice it to say, to scale from 12 to 24TB cost the price of buying 12x2TB server-grade disks. This server can scale to another 48 without haste, so I believe, with the possibility to grow that to 2x96 if required. I'll cross that bridge when I come to it, but for now, we're a long way off filling 24TB, which is a shitload of content.

The hw side of things I leave to others who are better qualified and in the know than I - I know my limits.

But thanks for your constructive feedback.

[borked]

Originally Posted by Barefootsies

You can take 10 different random companies and find that most of them manage their shit in ways that baffle the imagination.

Bang on
I'm currently migrating someone away from a managed solution to their own private setup so they aren't tied down and, while not naming the company that is really well known here, I am astounded at how they setup the servers....

Anyway, I really didn't want this to become a cock-battling thread of hosting companies! I just want more people to move over to wowza... cos I have an ulterior motive

[borked]

Originally Posted by Spudstr

Not sure exactly but Wowza normally decreases bandwidth by about 30-40%.. not increase bandwidth. Alteast thats the case when people switch from nginx/lighttpd.

That makes sense because wowza is a real streaming server while the other flavour is pseudo streaming. With real streaming, you are simply delivering the content (bandwidth) "on demand" and no more, no less, whereas with pseudo, you stream from a end-user defined point and just diss up the content, regardless of where the user is up to in watching, so you're always "wasting" bandwidth unless a user watches the entire video from beginning to end non-stop.

[mafia_man]

Originally Posted by raymor

Good to know. Java is generally known to be slow and and a memory hog

Hello 1999.

Java isn't that bad these days.

[borked]

Originally Posted by Spudstr

Then you should of been aware that that Adaptec 3805 card is a piece of junk with only 128mb cache and 3G/s. Your running 6G/s drives and that card is going to become a huge problem down the road for you. Something like the 3ware or lsi 9750/9260 would of been a much better fit for this, since your drives are 6G/s drives and your bottleneck is 3G/s on that card.

For the moment whatever card is in there (can't find it with dmidecode - is there another tool to check with?), the IO wait is very respectable for peak times i think: await is 2.9 and %util is 35

[Spudstr]

Originally Posted by borked

For the moment whatever card is in there (can't find it with dmidecode - is there another tool to check with?), the IO wait is very respectable for peak times.

lspci should show you what cards are currently installed.

Are you using lvm to concatenate the sdd and sdc together? or are they separate? It looks like it is, once you cross the barrier of the disk usage on the sdd and cross into disk space on the sdc partition/set you will see a breath of fresh air, currently your supporting the disk IO on one raid set. I could be wrong but thats what it looks like.

Disk contention is overcame by usually creating smaller raid sets on large spindle sizes, i.e 8x groups of 6 drives and if you concatenate them via lvm as you add content it will spread to different spindles. Once data is contained and "cached" via the system it wont seek them since it knows where they are on that raidset. + you can utilize the platters more efficiently because you have now 8 sets of disks doing various IO tasks vs 1-2 sets doing the same task. MUCH better disk IO/throughput that way.

[borked]

yup, it's still on the adaptec

09:00.0 RAID bus controller: Adaptec AAC-RAID (rev 09)

for the moment, c&d are separate yes. I'll look into lvm

[Spudstr]

Originally Posted by borked

yup, it's still on the adaptec

09:00.0 RAID bus controller: Adaptec AAC-RAID (rev 09)

for the moment, c&d are separate yes. I'll look into lvm

also depending on the size of your files, they could be a few mb or afew gb in size.. adjsut the drive readahead size.

i.e hdparm -a /dev/sdd will show a result.. its probalby 256. Jack that up to 16384 i.e hdparm -a 16384 /dev/sdd so its doing by far less reads when its grabbing data for distribution. this value can be tune/adjusted for whatever your traffic/source files are like. Makes a world of a difference on large files.

Also blockdev command works great too, shorter versionof using hadparm. blockdev --report will give some information on blocksize, make the readahead a multiplier of the blocksize.

[borked]

OK, now I'm learning stuff

set the readahead to 16384. I'll give it a bit of time to see how that affects the IO reads.

Cheers!

[mikeworks]

Spudstr do you have any server specials?

[Spudstr]

Originally Posted by mikeworks

Spudstr do you have any server specials?

contact me on icq or email me [email protected] and I'll see what we have

[ladida]

Originally Posted by Barefootsies

Zak does that often.

Quite often.. I see him get into pissing matches about things he has no idea about, yet he's still arguing the master of the universe theory.

[borked]

Originally Posted by CaptainHowdy

I love you, borked ...

I love you all - GFY is just a big disfunctional family growing stronger than the others around them

[WebairGerard]

Nice Borked! We love Wowza too and found that we can do nearly 1Gbps with wowza with our cloud servers. where else can you find an application that can operate at nearly line speed performance?