Not Backing Down

If people don't wanna be around someone and that someone shies away, why do they get to attack me?