eat away at

1. Destroy gradually, erode; also, worry one constantly. For example, The sea has been eating away at the outer banks for years, or The fact that he failed the test is eating away at him. [Early 1800s]