"dominant female entity, mistress," attested since 1560s, but rare until late 20c., and not in quite the usual modern sense:
[A]s religion, the supreme dominatrix, exercises her salutary influence on every department of individual and social life, there is no branch of our culture in which Christianity has not effected important meliorations. [Brownson's Quarterly Review, October 1850]
See domination + -trix. Modern BDSM sense "woman, who takes the sadistic role in sadomasochistic sexual activities" is attested by 1970.