Another possibility is your Anti-virus; try uninstalling it and see if it has an effect. Your A/V is likely 32-bit and may not function quite as well in 64-bit. This is just a theory, I don't know for certain, but if you uninstall it is easy enough to see if that is an issue.
I think it is a good theory. But I caution not to simply uninstall and install. But rather make sure you download and install the 64-bit version of your anti-malware program.
This also assumes you used a legitimate copy of the 64-bit version to upgrade. This is important, not just for licensing, but because unauthorized copies may be infected with malware.
Note too, all your hardware now needs 64-bit drivers for optimal performance too.
Are you seeing the full 4Gb? You might also look in Task Manager to see which program is eating up that RAM.
In the end, your computer should run better (or at least as well) with 64-bit as it did with 32-bit. And 64-bit Windows 7 has some security features not found in 32-bit Windows 7 so ultimately, it is a good move to go to 64.