kathCAT,

I do not know if you can o cannot transfer to the college other than the one in your F-1 visa without even attending the first semester. But what I know is, when you fill out the application form for the F-1 visa you have agreed to study at least the first semester at the college whose I-20 you're using. That is a consent you've agreed to. If you are able to transfer, like I have seen some people do in extreme situations, it is advised by many International Student Advisors strongly that you do not do so, because later while you'll be persuing legal documents in the future including while obtaining a new F-1 or stamping an H1, this thing might come up against you as you've not followed a simple rule of I-20 and F-1 visa.

So, Good Luck to you