Bumping this useful thread
Core Tweaks WordPress Setup - All of the "little things", from changing permalink structure to deleting the Hello World post, and more, are all accessible from one admin page.
WP Install Profiles - Bulk download of plugins automatically from the WordPress plugin directory.
WordPress Sentinel - Very useful for testing themes and plugins. Acts as a sentinel that watches over your WordPress files and tells you when changes happen.