I understand where you're coming from. However, I think that ultimately, it will cause more problems than it will solve.
Either people who have been downvoted will start calling out those who downvoted them or they will simply create a grudge list and start downvoting everyone who downvoted them.
While there might be some positive aspects, I think they will be overshadowed by the negative.
Thank you for pointing that out.
I think it's a bad idea to make that information public, as it is almost certainly going to be the cause of arguments, bitterness, drama, and tit-for-tat downvote wars.