Yes you can join just join the client or join vendor without letting your consultancy know.. if you don't tell anybody you have joined the client or vendor then there is no way your consultancy will find you where you are working (I am assuming you are the only one working on that client from that consultancy). This is just my 2cents.