The college was founded in 1800 under the will of Sir George Downing (also famous for the street of the same name in London - a door formerly from Number 10 is in use in the college). He died in 1749 and the delay in getting the college going was due to his relatives' legal wrangling over the will. So much legal wrangling, in fact, that there wasn't much money left for the college itself.
Hence the grandiose plans for a square of buildings based on the Classical Greek style and of a size to overtake the Great Court at Trinity College had to be curtailed.
The third side of the square was only completed in the 1950s with the building of the college chapel. Where the fourth side would have been is now a large paddock ("The Paddock") with many trees.