Origin of cracker
mid-15c., "hard wafer," but the specific application to a thin, crisp biscuit is 1739; agent noun from crack (v.). Cracker-barrel (adj.) "emblematic of down-home ways and views" is from 1877.
Southern U.S. derogatory term for "poor, white trash" (1766), probably from mid-15c. crack "to boast" (e.g. not what it's cracked up to be), originally a Scottish word. Cf. Latin crepare "to rattle, crack, creak," with a secondary figurative sense of "boast of, prattle, make ado about."
I should explain to your Lordship what is meant by crackers; a name they have got from being great boasters; they are a lawless set of rascalls on the frontiers of Virginia, Maryland, the Carolinas and Georgia, who often change their places of abode. [1766, G. Cochrane]
But DARE compares corn-cracker "poor white farmer" (1835, U.S. Midwest colloquial). Especially of Georgians by 1808, though often extended to residents of northern Florida. Another name in mid-19c. use was sand-hiller "poor white in Georgia or South Carolina."
Not very essentially different is the condition of a class of people living in the pine-barrens nearest the coast [of South Carolina], as described to me by a rice-planter. They seldom have any meat, he said, except they steal hogs, which belong to the planters, or their negroes, and their chief diet is rice and milk. "They are small, gaunt, and cadaverous, and their skin is just the color of the sand-hills they live on. They are quite incapable of applying themselves steadily to any labor, and their habits are very much like those of the old Indians." [Frederick Law Olmsted, "A Journey in the Seaboard Slave States," 1856]