Should i pursue this?

You either need to give up on this guy or leave your husband. You can't have your cake and eat it too.