And no you wouldn't want to remove them. There are many applications that are still only available in 32-bit but also work just fine under a 64-bit OS. Some applications such as IE8 (Internet Explorer 8) and WMP (Windows Media Player) are available in both the 32 and 64 bit versions and will exist in both directories.
Also, you don't need to specify which program files directory the installer should use when you do an install, the installer is smart enough to know.