1550s, "an abnormality of growth," from Late Latin monstrositas "strangeness," from Latin monstrosus, a collateral form of monstruosus (source of French monstruosité), from monstrum "divine omen, portent, sign; abnormal shape; monster, monstrosity," figuratively "repulsive character, object of dread, awful deed, abomination," from root of monere "to admonish, warn, advise," from PIE *moneie-"to make think of, remind," suffixed (causative) form of root *men- (1) "to think."
Earlier form was monstruosity (c. 1400). Sense of "state or quality of being monstrous" is first recorded 1650s. Meaning "a monster" is attested from 1640s.