fly-by-night (n.)

1796, slang, said by Grose to be an old term of reproach to a woman signifying that she was a witch; used from 1823 in reference to anyone who departs hastily from a recent activity, especially while owing money. The different senses involve the two verbs fly.