Should I just walk away?

You should do what your heart tells you. If you feel you should leave religion, so be it. It will still be there should you choose to come back to it. You should do whatever makes you a more successful and productive person. ^_^