down on one's luck

1. Afflicted by misfortune, as in They've been down on their luck ever since they moved out West. [Colloquial; second half of 1800s]