The castle was founded by the Normans shortly after the Norman Conquest of England, but enjoyed its heyday under Bernard de Bailleul during the latter half of the 12th century. The castle passed into the hands of the Balliol family (of which the Scottish king, John Balliol, was the most important member), and then into the possession of Richard Neville, Earl of Warwick. King Richard III of England inherited it through his wife, Anne Neville, but it fell into ruins in the century after his death. The remains are now in the care of English Heritage.