At its height, the palace extended over much of the area currently bounded by Northumberland Avenue in the north; to Downing Street and nearly to Derby Gate in the south; and from roughly the elevations of the current buildings facing Horse Guards Road in the west, to the then banks of the river Thames in the east (the construction of Victoria Embankment has since reclaimed more land from the Thames).
By the 13th century, the Palace of Westminster had become the centre of government in England, and had been the main London residence of the king since 1049. The surrounding area became a very popular — and expensive — location. Walter de Grey, the Archbishop of York bought a property in the area soon after 1240, calling it York Place, founding what would later become the Palace of Whitehall.
Edward I of England stayed in the property on several occassions while work was carried out at Westminster, and enlarged the building to accommodate his entourage. York Place was rebuilt and greatly extended during the 15th century and expanded so much by Thomas Cardinal Wolsey that it it was only rivaled by Lambeth Palace as the greatest house in London, the King's London palaces included. Consequently when King Henry VIII removed the cardinal from power in 1530, he acquired York Place to replace Westminster as his main London residence.
Henry VIII spent the winter of 1529 redesigning York Place, and further extended and rebuilt the palace over the following years. Insired by Richmond Palace, he also included a recreation centre with a bowling green, tennis courts, a pit for cock fighting and a tiltyard for jousting. It's estimated that over 30,000 pounds were spent during the 1540s, 50% more than the construction of the entire Bridewell Palace. By 1650 the Palace was the largest complex of secular buildings in England, with over 1,500 rooms.
James I made a few significant changes to the buildings, notably the construction in 1622 of a new Banqueting House built to a design by Inigo Jones to replace a series of previous banqueting houses dating from the time of Elizabeth I. Its decoration was finished in 1634 with the completion a ceiling by Sir Peter Paul Rubens, commissioned by Charles I (who was to be executed in front of the building in 1649).
In 1691, when the palace was the largest palace complex in Europe — and a jumble of buildings — a fire destoyed much of the older palace structures. This actually gave a greater cohesiveness to the complex. However a further fire on January 4, 1698 destroyed most of the other residential and government buildings. Despite some rebuilding, financial constraints prevented large scale rebuilding. In the second half of the eighteenth century, much of the site was leased for the construction of town houses. Beginning in 1938, the east side of the site was redeveloped with the building now housing the Ministry of Defence.
The Palace Today
The 1622 Banqueting House is the only building now remaining, although it has been somewhat modified. Various remains survive including a former covered tennis court from the time of Henry VIII in the Cabinet Office at 70, Whitehall. An undercroft from Wolsey's Great Chamber, now known as Henry VIII's Wine Cellar also survives in the basement of the Ministry of Defence. (It was moved there on rollers in the (1940s?).)