If someone betrays you, its your fault that you trusted him. If he robs you, its your fault to leave valuables in the public or semi-public area.
If someone follows you on the WM, from one location to the other and you do not notice him, its definitely your fault. You werent paying attention, deal with consequences. (Another thing would be, if the following person was invisible even on the follower list when sneaked. So you would have no chance to find out whether you have been tracked or not.)