It was built by the Peverel family in the 12th century and became Crown property in 1155 when the third William Peverel fled into exile. A stone keep was built around 1173, surrounded by a curtain wall which was eventually breached in 1216 during the reign of King John. Surviving fragments of this curtain wall were later incorporated in a wall walk that can be seen in the castle garden.
Much of what can be seen today dates back to the 17th century. The castle is now the subject of a development project.