There has been a manor on the site of the present house since before the Norman conquest of England. In the Domesday Book (a survey of England published in 1086) the house was listed as belonging to the Peverell family, who arrived from Normandy with William the Conqueror. It was being managed for them at the time by their tenants the Gresleys.
Having passed by inheritance through two further families it was purchased by Sir John Brockley in 1433 who was Lord Mayor of London at the time, and then sold in 1458 to Sir Ralph Verney who also made it to the position of the Lord Mayor as part of his career. It was Verney who largely remodelled the property into the grand mansion house that stands there today.
A notable member of the Verney family was Sir Edmund Verney, who was chief standard bearer to King Charles I during the English Civil War. Sir Edmund was slaughtered at the Battle of Edgehill on October 23rd 1642 and is buried in the family chapel at Claydon. It is said that at dusk, on the anniversary of his death every year, an apparition of the battle itself appears on the lawns of the great house, and has been reported by many servants from the house through the years since Sir Edmund's death.
In 1661, following the Restoration of the Monarchy, Sir Edmund's son (Sir Ralph Verney II) was awarded a baronetcy by King Charles II for his and his father's loyalty and bravery during the preceding period of unrest. He was later, in 1703, made Viscount of Fermanagh and his grandson was, in 1742, created an Earl. Both titles have since, however, become extinct.
Later the house became the home of the Nightingale family, descendants of the Verneys, one of whom being Florence Nightingale the nursing pioneer. She spent many years in her youth at the house, and there is a large collection of her letters on display at the house today. Later in life, she returned to Buckinghamshire, giving great support to the Royal Buckinghamshire Hospital in nearby Aylesbury.
The house was bequeathed by the Nightingale family to the National Trust.