If you have 500kb of code its obviously going to be slower than 100kb of code. The overall size of your Javascript code will depend on the quality of your coder and the library (or libraries) they chose to use.
Google Closure has a compiler that will compile a stand alone Javascript file that contains only the necessary code for your application. (This removes the need to load an entire library)
There are also tools such as minify to reduce file size and cache code.
http://code.google.com/p/minify/