Downman was the son of Francis Downman, attorney, of St Neots, and Charlotte Goodsens, daughter of Francisco Goodsens, a musician of the Chapel Royal from 1711 to 1741; his grandfather, Hugh Downman (1672–1729), had been the master of the House of Ordnance at SHeerness. The Downman family is usually known as a Devonshire one, but the exact connexion between the artist and the Devonshire branch has not been traced. Earlier works assumed Downman was born in 1750 near Ruabon in Denbighshire where he first went to school; however it is now known he was baptised on 12 September 1749 at Eynesbury in Huntingdonshire, adjacent to St Neots. After his early schooling in Ruabon, and then briefly in Liverpool, he attended finally the Royal Academy Schools, and was for a while in the studio of Benjamin West.

Downman set off in 1773 with Joseph Wright of Derby, a pregnant Ann Wright and Richard Hurleston for Italy. Their ship took shelter for three weeks in Nice before they completed their outward voyage in Livorno in Italy in February 1774. Downman returned to Britain in 1775.

He settled down for a while in Cambridge (1777), eventually coming to London, where he contributed to various art exhibitions. In 1804 he moved to the village of West Malling in Kent In 1806, Downman visited Plymouth in the West Country, between 1807 and 1808 he practised at Exeter, afterwards working in London for some years, settling at  Chester in 1818-19, then finally moving to Wrexham, where his only daughter married and where he died on 24 December 1824. He left a large collection of his paintings and drawings to his daughter. He was also the father of Sir Edwin Downman.

He exhibited 148 works in the Royal Academy between 1769 and 1819, mainly portraits, but often fancy subjects, such as 'Rosalind,' painted for the Shakespeare Gallery; 'The Death of Lucretia; The Priestess of Bacchus;' 'Tobias;' 'Fair Rosamond;' 'The Return of Orestes;' 'Duke Robert,' etc. In 1795 he was elected an associate of the Royal Academy.

His first work at the Royal Academy (1769) was A small portrait in oil,' and the last (1819), A late Princess personifying Peace crowning the glory of England reflected on Europe, 1815. In 1884 the trustees of the British Museum acquired, by purchase, a volume containing numerous coloured drawings by Downman, among which were several portraits, now separately mounted. He also left several volumes of exquisite drawings, executed in red and black chalk, of which Ralph Neville Grenville published a catalogue (Taunton, 1865). He also painted some miniature portraits. Engravings were also made by Bartolozzi and others. Many of his portraits have attached to them remarks of considerable importance respecting the persons represented.

The eminent art historian Ellis Waterhouse said this of Downman: "manly distinguished as the prettiest and most elegant of portrait painters of his age in chalk and watercolours" .. to which i would add = and also in pencil.