It stands on the site of a nunnery reputedly founded by Anglo-Saxon King Alfred the Great in the 9th century. Some 300 years later, the nunnery was replaced by a Benedictine abbey. This was then surrendered to King Henry VIII during the Dissolution of the monasteries; the monarch then presented the estates to William Herbert (c.1542) who became Earl of Pembroke in 1551 (Herbert was married to Anne Parr, sister of Catherine Parr, Henry's sixth and last wife).
The house is renowned for its gardens - Isaac de Caus began a project to lanscape them in 1632 - which include a variety of water features and a bridge (c.1737) over the River Nadder.
The house was damaged by a fire in 1647 and was then rebuilt to Palladian-style designs by architect John Webb. Its Single and Double Cube Rooms are regarded as among the finest of the era.