If you know it is wrong, it is better to try and get away from it. Granted it will be difficult as lust is something that mighty kings and strong warriors have fallen for it, however, if you love your wife as you say you do, then you need to look beyond what is momentary pleasure.