What other less-toxic system could work instead of karma?

I think a reputation system is important, though reddit's current karma implementation is bad, there needs to be a method of identifying bad actors and forum shifters.

One refinement over karma could be that the score is kept only by community and should reflect that users contribution to the community.

Simple upvotes and downvotes also don't allow for nuance, replace them with a Buzzfeed like tag system (yes I know we all hate the site for its content but its tag system if used properly could be pretty powerful.

So instead of 'up' and 'down', you have a clickable emoji-menu like list of tags like 'interesting', 'boring', 'funny', 'WTF!?', 'Quality', 'Trash', 'Educational', 'CAT', etc...

So the reputation score for the community isn't just a flat number, rather it will tell you the kind of content a person posts over time, and doesn't carry just flat positive or negative connotation.

I mean the king of Catposting may have massive reputation in meme subs with high ranks in tags for 'Funny', 'Cute', and 'CAT' though that might not be the case if they participate in say a chemistry QnA community.

As these scores are created over time based on each users contributions (post AND comment reputation is the same thing) to the sub as scored by other people's tag selections for that users posts. The more it aligns with the community, the greater their contribution score.

Does this mean that toxic communities can form that exclude people based on reputation tags that the toxic community detests?

Unfortunately yes, that is one of the flaws of the system.

THOUGH

The fact it is contained by community means that a high rep person in an anti-trans community will not have any carryover reputation when joining a community they wish to brigade or degrade the quality of content, and their tag history will make it easy to determine their genuine engagement.