Problems with overclocking
Heat: It's an overclockers worst enemy. The excess heat that overclocking your video card produces can not only affect your visuals with artifacts, or no picture at all, but also CPU overclocking. The reverse is also true when overclocking your CPU. Artifacts are strange graphical glitches that occur when you overclock your video card too far. These can range from missing pixels and textures to weird lines running across the screen. If you can see these when you have overclocked your video card, you have a problem! Fortunately, there is an easy fix. Simply move back your memory/core speeds until the artifacts go away. If you ignore the artifacts, you run the risk of permanently damaging your video card.
As I stated, the main reason behind this is heat. Apart from installing heatsinks and coolers to your card, which is detailed above, try to increase the airflow around your graphics card. A system exhaust fan over a PCI slot which vents hot air out of the case that would otherwise linger around the graphics card is an excellent way to reduce case temperatures and thus graphics card temperatures. You can also try to set up a fan that blows cool air into the graphics card area which should also give a reasonable increase in cooling.