Here is what you need installed on your server.
APC
http://pecl.php.net/package/APC
Memcache
http://pecl.php.net/package/memcache
W3 Total Cache
http://wordpress.org/extend/plugins/w3-total-cache/
Set your "Page Caching Method" to Memcached.
Set your "Minify Caching Method" to Memcached.
Set your "Database Caching Method" to APC
Your server load will be high while its caching and making pages. Once this is over the server load will dramatically decrease to about 40 to 50%.
Your WordPress site will load instantly.
Most of my pages are loading at 0.021
The database caching is key to performance.