Well yes, this is a bit of a game mechanic flaw, best way to protect yourself against this is to play characters with only 1 Charisma and never use mentat or beer.
If not YOU NEED TO ALWAYS CHECK WHO IS IN YOUR PARTY WHEN GOING ON WORLD MAP, should you lag for any reason, you might not see that someone is following you, so you need a little extra time to make sure your party list is *loaded*.