It really depends on your setup and what you have on it. Each server transfer is unique and needs to be evaluated first.
This gives a general idea of the steps required:
How to migrate your website FROM another host TO InMotion Hosting - InMotion Hosting
It lists main steps, but lacks an important one if you switch between a VPS or a dedicated server that is to adjust your server accordingly:
- Install tools you need (FFmpeg, MySQL, Apache, ImageMagick, etc.)
- Adjust your Apache configuration
- Adjust your MySQL configuration
- etc.
I can't recommend you a program or a tool to do it as I don't know what you plan to transfer. You'll need to determine what you'll need or get someone to do it for you.