Almost certainly what happened is the serial vote reversal script saw that you (down)voted the same person a bunch of times and reversed the votes. It matches up with the information you've provided: the votes were reversed and you got a notification for rep being returned (which happens when the votes were downvotes on answers). Plus, the timeframe seems to be about right: the script runs at 03:00 UTC. For more information about serial voting see here. In any case, moderators are not able to touch your votes, although a CM can, should it come to their attention and there be a good reason to.
The serial vote reversal script doesn't punish anyone, but if you recast all the votes it will reverse them again. Although it may not seem fair when there are posts that genuinely need to be downvoted, you should avoid downvoting a single user a lot in a small timeframe. If the situation is urgent, you can flag for a moderator (you can flag once for mod attention and then use standard flags on all the other posts, where applicable). Otherwise, I'm sure there is a lot of other low-quality content on this site that you can downvote instead.