Charisma isn't about how one is looking. It is the whole "experience" when encountering said person. Appearance, the way you walk, general behaviour etc. is "Charisma". Not just "CH1=Lol u are ugly!!!1111".
Yes, I once wanted to say about the same (when I saw somewhere post like "Let's town partols will shoot at all 1 CH characters!!!"). Charisma = how that person feel himself in society, low charisma means he is shy or don't want to speak with people, may be he have no sence of humour. High charisma means he know how to control people with his voice, he is very chatable and active. Also people will trust tasks to persons with more charisma, because low charsima people look like they can't handle anything. So person with 1 CH isn't ugly mutant. But... look Fallout 1, 2, Tactics, Online description:
"A combination of appearance and charm. A high Charisma is important for characters that want to influence people with words."
So... it's appearance...
In Arcanum there're two stats: Charisma and Beauty.
In Fallout only one. And it works like Beauty only.
And of couse:
NPC's react to low cha with comments on how ugly you are
Also sometimes they call you an animal who escaped zoo or you scaring his customers.