This is a (in my opinion) good way to prevent trolling and mindless downvoting on answers. If downvoting on answers was free you could expect a lot more downvotes on answers that don't really belong there.
Also keep in mind that you gain back your 1 reputation if the answer you've downvoted gets deleted. This is especially important since in most cases the bad answers you've described above are deleted within a day.
Keep in mind that there aren't a lot of reasons where you should downvote an answer:
Low quality answers are usually deleted or edited within a day which solves the problem and gives you back your rep.
Not an answer flags are handled extremely quickly by the community. You'll get your rep back within an hour.
Answers with misleading information take a little longer to handle since it requires 3 delete votes from high-rep users but after a week or so the post will usually be gone aswell.
Spam is handled extremely fas... Oh it's gone already. (same with offensive content)
Personal problems with the answer are also an ok reason to downvote but sometimes you'll have to sacrifice 1 rep to show your dislike of the post to the public. If enough users share the same opinion as you the post might even be deleted some day.
So basically you're only getting "punished" if you have to downvote answers because you disike every answer (and probably also rainbows). If you use downvotes to keep the site's quality high you'll get your reputation back in no time.