rub the wrong way

1. Irritate, annoy, as in His remarks about welfare rubbed a great many people the wrong way. This idiom alludes to rubbing an animal's fur in the wrong direction. [Mid-1800s] Also see ruffle someone's feathers.