I think I found out what is doing it. When I'm standing in the Brahmin gate entrance The guards start yelling at me and I think my NCR hate goes up by 100.