Needing charisma to have slaves makes no sense. They're slaves! You have captured then and are holding them against their will!
How handsome and silvertongued you are will not change anything!
Here is what I propose.
SLAVE REVOLT!
You can capture up to 5 slaves no matter your charisma, but the worse your charisma/slave ratio is, the higher the chances of them revolting!
Things that also could affect the chances of revolt could be:
your current HP %
What teir weps and armor you are using, compared to what the slaves are using.
If you are unconscious or not
what types of slaves
amount of slaves/ amount of CHA
Etc.
This would mean that the traditional high CHA slaver builds are still pretty safe unless they have no weapon with 20hp or something and their slaves are standing around armed with sniper rifles. Which would be tempting no matter how charismatic your slave master is
But when a low CHA build takes slaves, they would have to be constantly on guard with a weapon equiped and armor on, and be careful to never get too weak.
The player equipment/slave equipment ratio would be the most important for preventing rebellion.
A 1 charisma powerbuild fully armed and max HP leading 5 peasant slaves armed only with hammers to slave mine - virtually no chance of rebellion.
A 1 charisma powerbuild leading a team of armed combat slaves into battle, then getting his weapon shot out of his hands or losing most of his HP, or both? SLAVE REVOLT! All slaves turn hostile.
This change would make peasant slaves used for labour to be available to all characters, and make combat slaves very dangerous to low CHA people and still mildly dangerous to high CHA people. Also, ++realism.
If you want trustworthy combat minions, buy mercs.