You can't buy a mobile graphics card and install it. They are build into and soldered onto the motherboard.
Parts of that laptop are upgradable, but it sounds like the graphics chip is not. Old laptops had enough space for a discrete cards, but modern ones do not.
TBH, if you want to upgrade your computer at a later date, you need a desktop.
If you still don't get what I'm trying to say, here's a more clear restatement:
This is a GTX 760m:
http://videocardz.com/images/2013/05/GeForce_GTX_760m.jpg
Those little balls are solder, and keep it stuck to the motherboard in a way that you cannot remove. The GTX 660 is the same way.
This is the smallest GTX 760 that can be replaced in a computer (and is much more powerful than the GTX 760m, besides):
https://www.asus.com/Graphics_Cards/GTX760DCMOC2GD5/
It should be obvious why putting a new graphics card in won't work for your laptop.
Now, another option is, if your Alienware was made in some GTX 760m variant, you *might* be able to buy a motherboard with the GTX 760m already on it, but installing it will be very difficult, and very expensive, as you will likely need a professional to replace the motherboards lest you break some ribbon cables and make your laptop unusable.
I'm actually quite certain that it would be cheaper to buy a new laptop than try to buy an aftermarket motherboard (one that doesn't fit your laptop) and try to install it.
You can buy gaming laptops with a GTX 765m for <$1000:
http://www.newegg.com/Product/Product.aspx?Item=N82E16834198039