Wallingford itself sits on the western side of the Thames; across the river is the town of Crowmarsh Gifford. The two are linked with a notable 900 ft long stone bridge across the Thames and adjacent floodlands.
Given the town's strategic importance, it has been fortified since at least Saxon times, when it was an important part of the kingdom of Wessex and allowed to mint its own coins. Since William and his Norman army were permitted to cross the river unopposed, the town received special favor from the Norman conquerors.
The Treaty of Wallingford was signed there in 1153, ending the Civil War that had begun after Henry I's death. The town was granted a Royal Charter in 1154 by the new King Henry II, being one of the first towns to receive one (before London, for example).
The Castle was a regular royal residence until the Plague hit the town badly in 1349. The castle declined subsequently (much stone being removed to renovate Windsor Castle) but it became a Royalist stronghold during the English Civil War. It was the last holdout of the Royalists in the region, and the castle withstood a 65 day siege. Oliver Cromwell subsequently ordered the destruction of what was left of the castle and little now remains.