Never continue in a job you don't enjoy. If you're happy in what you're doing, you'll like yourself, you'll have inner peace. And if you have that, along with physical health, you will have had more success than you could possibly have imagined.
I couldn't care less what anybody says about me. I live my life, especially my personal life, strictly for myself... Whatever you do, you're going to be criticized. I feel the one sensible thing you can do is try to live in a way that pleases you.