this is a (relatively) free country. I think you should not worry about their empty threats, they have more to lose. No one can force you to work where you don't want to work.