Born in Detroit, Brundage studied civil engineering at the University of Illinois, graduating in 1909. A few years later, he founded his own company, the Avery Brundage Company, which was active in the building business around Chicago until 1947.
Brundage was an all-round athlete, competing in the 1912 Summer Olympics in Stockholm in the pentathlon and decathlon events, finishing 6th and 16th, respectively. He also won the US national all-round title on three occasions (1914, 1916 and 1918).
In 1928, he became president of the Amateur Athletic Union (AAU), followed by the presidency of the United States Olympic Committee (USOC) in 1929 and the vice-presidency of the International Amateur Athletic Federation (IAAF) in 1930.
He became a member of the International Olympic Committee (IOC) following the Berlin Olympics in 1936. After the death of IOC president Henri de Baillet-Latour during World War II, Brundage became vice-president of the IOC in 1945. When Sigfrid Edström retired in 1952, Brundage was appointed as his successor.
During his presidency, Brundage strongly opposed any form of professionalism in the Olympic Games. Gradually, this view became less accepted by the sports world and other IOC members. It lead to some embarrassing incidents, such as the exclusion of Austrian skier Karl Schranz, who was accused of being a professional, from the 1972 Winter Olympics.
Brundage may be best remembered for his decision during the 1972 Summer Olympics in Munich, Germany, after which he planned to retire as IOC President. But on September 5, Palestinian terrorists killed 11 Israeli athletes. Many critised Brundage's decision to continue the Games, although few athletes withdrew from the Games.
In addition to his role in sports, Brundage was a noted collector of Asian art. During his lifetime, and by bequest on his death, he gave a massive collection of works of art to the city of San Francisco, California. This collection formed the nucleus (and, as of 2003, still accounts for over half the contents) of the Asian Art Museum of San Francisco, initially founded to house and display his donation.