I'm trying to figure out why when I downvoted an answer, my score went down as well. I don't see anywhere why that should affect my score.
That's how the reputation system works, as you can read on our FAQ; specifically:
[reputation effect when] answer is voted down -2 (-1 to voter)
This is the way it works across the entire Stack Exchange network. As for why that is how reputation works across the network, you should check out this meta-discussion on the network's main meta site.