The building was built in 1861 during the reign of Napoleon III. It originally housed tennis courts; the name is from the precursor of tennis, the jeu de paume.
It was used from 1940 to 1944 to store Jewish cultural property looted by the Nazi regime in France.
Before 1986, it contained the Musée du Jeu de Paume, which held many important impressionist works now in the Musée d'Orsay.