It used to be that credit card issuers could raise your rate, even on existing balances, at any time and for any reason. Thanks to the Credit CARD Act, a federal law, they can no longer do this. They can, however, raise your rate on your outstanding balance if you are more than 60 days late with a payment and they can increase the interest rate on new purchases, but only if they give you 45 days advance notice so you can cancel your account.
As for what you can do, the best thing is to try to negotiate a lower rate. Call your card issuer and suggest you will take your business elsewhere if you can't get a better deal. This works best if you have other credit cards with available credit lines, since the issuer will no doubt review your credit report to decide whether or not to work with you. It's always worth a try, though. As author Marc Eisenson says, "Not asking is an automatic no!"
If you can't negotiate a better rate, try transferring your balance to another card -- either one you already have, or a new one.