Yes, an MSI Radeon HD 7770 will definitely work on your ASUS M5A97 motherboard. The ASUS M5A97 motherboard is actually CrossFire-compatible. This means you could put another HD 7770 into the motherboard in the future, and enjoy close to double the performance of the single HD 7770.
The ASUS M5A97 motherboard has PCI-Express x16 slots, which means you can put virtually any video card on the market into the motherboard. But being CrossFire-compatible, it gives you a great upgrade path going into the future.