Alts are a part of the game. Surely you would've come to peace with this by now?
It's fine as is, because it fucks your char up beyond playability, without mercs.
but if you want working character, you must give something up, which is more power mercs.
besides, it is obvious the concept behind it.
Your strength, charisma, speech, inspires, and strengthens your followers, the better leader you are, the better your followers can be, through basic training, and pure leadership.
I just wish, that such effects would take place when you enslaved npcs, would make more sense, but of course, they're just slave, not too much boost.
Think of this characters, as stumpy, bald, diabetic, no-energy, blind, but willfull people, who with their example at surviving in waste, inspire others into their best, or something similar
.