The desire to annoy no one, to harm no one, can equally well be the sign of a just as of an anxious disposition.
I've found men are less likely to let petty things annoy them.
