I know that when I bought my Kingston HyperX PC8500 1066 RAM it's SPD settings from the factory were PC6400 or 800MHz clock speed.
In order to achieve the higher clock speed, other than setting the multiplier different, I had to boost the DRAM voltage from the standard of 1.8 VDC to 2.1 VDC before it would run stable. Those were the settings recommended by Kingston.
So maybe checking the voltage setting would be a good idea too. Go to the RAM makers website and see what the recommended voltage settings are for the type of RAM you're using.