upgrading to new server.. scared!
 

I know google shouldn't punish me for this move.. but i'm still worried.

Should I lose sleep over this?

yes.. you should stay up for days over this

no. try to keep the downtime as minumum as possible and you should be ok. the worst that happened to me is I lost a few positions but quickly got them back. :)

one thing you can do if expected downtime is to lock in your current stuff
using

<meta name="revisit-after" content="5 days">

and also
an expiration tag, which google will follow

basically expire it in 14 days... any changes will not be affected, page is sorta locked in as was until expiration date

Why would you lose any up time switching servers?

My sites changed by PR -1 when I moved servers a few months ago (6->5, 5->4). They were back a month later.