Why is it ok to insult and crack jokes about some minority groups, but not others?

I noticed that when someone mentioned being annoyed by a dwarf in a prior post, that many answerers thought it would be funny to crack jokes about little people, but if the same question had been asked regarding a person of color or maybe even a person in a wheelchair, the answerers would gotten all over the asker for being a bigot. Why is it considered acceptable by some to make fun of a disability such as dwarfism, but skin color or other types of disabilities are sacred? All are cirstances of birth.