1835 as an adverb, "into the town," from the prepositional phrase; see down (adv.) + town. The notion is originally literal, of suburbs built on heights around a city. From 1836 as an adjective; by 1851 as a noun.
the heart of Birmingham's downtown
downtown Manhattan
delinquents roaming the downtown streets
downsize
downspout
downstairs
downstream
downtime
downtown
downtrodden
downturn
downward
downy
dowry