It's not due to your bitrate, it's due to the video information being put on the end of the file.
Videos are not encoded for web streaming; A/V times are not put at the front of the media file, which they should if you want to play a certain part of the file.
I have 5GB videos which start instantenously, so that explains it.
Just properly encode them
