Dictionary.com
definitions

count against

verb

(intr, preposition) to have influence to the disadvantage ofyour bad timekeeping will count against you
Collins English Dictionary - Complete & Unabridged 2012 Digital Edition © William Collins Sons & Co. Ltd. 1979, 1986 © HarperCollins Publishers 1998, 2000, 2003, 2005, 2006, 2007, 2009, 2012
Idioms and Phrases with count against

count against

Be disadvantageous to, as in His earnings this year will count against his Social Security benefits. This idiom uses count in the sense of “make a reckoning,” in this case negative. [Early 1900s]

The American Heritage® Idioms Dictionary Copyright © 2002, 2001, 1995 by Houghton Mifflin Harcourt Publishing Company. Published by Houghton Mifflin Harcourt Publishing Company.