eat away at
Destroy gradually, erode; also, worry one constantly. For example, The sea has been eating away at the outer banks for years, or The fact that he failed the test is eating away at him. [Early 1800s]
The American Heritage® Idioms Dictionary Copyright © 2002, 2001, 1995 by Houghton Mifflin Harcourt Publishing Company. Published by Houghton Mifflin Harcourt Publishing Company.