The house was built from 1759 for the Lascelles family who had bought the easte after making their fortune in haberdashery and customs work in the West Indies. John Carr was one of the architects; he married one of the Lascelles daughters. The other architect was Robert Adam. There are wall panels by Angelica Kauffmann and furniture by Thomas Chippendale of Otley.
Lancelot Brown ("Capability" Brown) designed the grounds to which Sir Charles Barry later added a grand terrace.