The building was built in 1774 for John Julius Angerstein, a Lloyds under-writer whose art collection was bought in 1824 to form the nucleus of the National Gallery, London. Acquired by the London Borough of Greenwich in 1967, the house opened as a Local History Library and Art Gallery in 1972.
In October 2003, the local history archives were set to move to a new site on the Woolwich Arsenal site in Woolwich.