I seem to have lost 5 reputation over it, but I'm not sure what it means or why it negatively affects me. Can anyone shed some light on this?
When a user is deleted for any reason (usually either a spammer, or someone requests themselves to have their account deleted) all of their votes are deleted with them. This means, that if the user had upvoted you previously, you'll lose that reputation again when he is deleted.
I don't think this behaviour makes sense and have asked for it to be changed on Meta.SO, but it seems to be not easily fixable.