Yea … I said it above, but I think separating the up and down votes so that they don’t contribute to the same “score” might help. Make the downvote a separate process of basically softly and quickly reporting a post/comment for being out of line.